A simple Q-learning-based agent for the classic Snake game.
The goal is to implement a QAgent class from scratch and train it to make better decisions during gameplay.
The snake uses a Q-table to learn which moves lead to the highest rewards in different game situations. This is done using reinforcement learning, specifically the Q-learning algorithm.
- The snake is controlled by a Q-learning agent.
- The board contains green apples (good) and red apples (bad).
- The agent learns by trial and error: it tries actions, gets rewards, and updates its Q-table.
This function converts the current game situation into a discrete state representation. The snake "looks" in 4 directions (up, down, left, right) from its head, and encodes the first thing it sees in each direction (wall, apple, snake body, etc.).
Example output:
("W", "Z", "G", "S")
Where:
W= wallZ= nothing (empty cell)G= green appleR= red appleS= snake body
This encoded state is used as a key in the Q-table.
The Q-table is a Python dictionary that maps each state to a list of 4 values — one for each possible action:
Q[state] = [q_up, q_down, q_left, q_right]
Action order: UP, DOWN, LEFT, RIGHT.
Episode: 42
Current State: ('Z', 'Z', 'G', 'W') # nothing ahead, nothing left, green apple right, wall behind
Q-values: [0.12, -0.05, 0.89, -1.00] # [up, down, left, right]
Decision:
- The highest Q-value is 0.89 for action LEFT
- Snake moves LEFT
Epsilon controls whether the agent:
- Explores (random move to discover new strategies)
- Exploits (chooses the best known move from the Q-table)
At the beginning of training, exploration is high (more randomness), but over time epsilon decreases, so the agent relies more on what it has learned.
"Greedy" means the agent always chooses the action with the highest Q-value for the current state:
best_action = argmax(Q[state])
This balance between exploration (random) and exploitation (greedy) is essential for learning effective strategies.
The agent receives a numeric reward after each action. These rewards drive the Q-value updates.
| Event | Reward |
|---|---|
| Step (survive / just move) | -0.001 |
| Eat green apple | +1.0 |
| Eat red apple | -1.0 |
| Death (wall/self-collision/len≤0) | -10.0 |
Q-learning update (one step):
Q[s,a] = Q[s,a] + α * ( r + γ * max_{a'} Q[s',a'] - Q[s,a] )
These values encourage survival and collecting green apples, while penalizing risky or terminal actions.
python3 src/main.py --episodes 10000 --save models/qtable_10x10.pkl
python3 src/main.py --visual --dontlearn --load models/qtable_10x10.pkl
Run all commands from the project root (
Learn2Slither/).
--episodes Nnumber of training episodes--visualenable pygame view--dontlearndisable learning (evaluation mode)--save PATHsave Q-table at the end--load PATHload an existing Q-table--board Nboard size (default 10)