Skip to content

Implementing Reinforcement Learning, namely Q-learning and Sarsa algorithms, for global path planning of mobile robot in unknown environment with obstacles. Comparison analysis of Q-learning and Sarsa

License

Notifications You must be signed in to change notification settings

sichkar-valentyn/Reinforcement_Learning_in_Python

Repository files navigation

Reinforcement Learning in Python

Implementing Reinforcement Learning (RL) Algorithms for global path planning in tasks of mobile robot navigation. Comparison analysis of Q-learning and Sarsa algorithms fo the environment with cliff, mouse and cheese.
DOI

Related works:

Description

RL Algorithms implemented in Python for the task of global path planning for mobile robot. Such system is said to have feedback. The agent acts on the environment, and the environment acts on the agent. At each step the agent:

  • Executes action.
  • Receives observation (new state).
  • Receives reward.

The environment:

  • Receives action.
  • Emits observation (new state).
  • Emits reward.

Goal is to learn how to take actions in order to maximize the reward. The objective function is as following:

Q_[s_, a_] = Q[s, a] + λ * (r + γ * max(Q_[s_, a_]) – Q[s, a]),

where,
Q_[s_, a_] - value of the objective function on the next step,
Q[s, a] - value of the objective function on the current position,
max(Q_[s_, a_]) – Q[s, a]) - choosing maximum value from the possible next steps,
s – current position of the agent,
a – current action,
λ – learning rate,
r – reward that is got in the current position,
γ – gamma (reward decay, discount factor),
s_ - next chosen position according to the next chosen action,
a_ - next chosen action.

The major component of the RL method is the table of weights - Q-table of the system state. Matrix Q is a set of all possible states of the system and the system response weights to different actions. During trying to go through the given environment, mobile robot learns how to avoid obstacles and find the path to the destination point. As a result, the Q-table is built. Looking at the values of the table it is possible to see the decision for the next action made by agent (mobile robot).


Experimental results with different Environments sre shown and described below.
Code is supported with a lot of comments. It will guide you step by step through entire idea of implementation.

Each example consists of three files:

  • env.py - building an environment with obstacles.
  • agent_brain.py - implementation of algorithm itself.
  • run_agent.py - running the experiments.

Content

Codes (it'll send you to appropriate folder):


Experimental results (figures and tables on this page):

Environment-1 with mobile robot, goal and obstacles

RL_Q-Learning_E-1 RL_Q-Learning_E-1


Represents number of episodes via number of steps and number of episodes via cost for each episode

RL_Q-Learning_C-1


RL_Q-Learning_T-1
Looking at the values of the table we can see the decision for the next action made by agent (mobile robot). The sequence of final actions to reach the goal after the Q-table is filled with knowledge is the following: down-right-down-down-down-right-down-right-down-right-down-down-right-right-up-up.
During the experiment with Q-learning algorithm the found shortest route to reach the goal for the environment-1 consist of 16 steps and the found longest rout to reach the goal consists of 185 steps.


Bigger environment-2 with more obstacles

RL_Q-Learning_E-2


Represents number of episodes via number of steps and number of episodes via cost for each episode

RL_Q-Learning_C-2


RL_Q-Learning_T-2


Super complex environment-3 with a lot of obstacles

RL_Q-Learning_E-3


RQ-learning_via_Sarsa


MIT License

Copyright (c) 2018 Valentyn N Sichkar

github.com/sichkar-valentyn

Reference to:

Valentyn N Sichkar. Reinforcement Learning Algorithms for global path planning // GitHub platform. DOI: 10.5281/zenodo.1317898