ForagerRL_step15

15. Testing & Tutorial Summary

By Killian Trouillet

---

Step 15: Testing & Tutorial Summary

Testing the Trained Multi-Agent Policy

After training, we load the shared PPO model and run both foragers in the GAMA GUI.

Starting GAMA GUI

Open GAMA normally. The GUI server runs on port 1000 by default.

Loading the Shared Model

from train_forager_petz import PPOAgent

agent = PPOAgent(state_dim=15, action_dim=2)
agent.load("saved_models/ppo_forager.pth")

One model, shared by both foragers — just like training.

Running a Cooperative Episode

obs, _ = env.reset()
done = False
while not done:
    actions = {}
    for agent_id in env.agents:
        action, _, _ = agent.select_action(
            np.array(obs[agent_id], dtype=np.float32),
            test=True   # ← deterministic: use mean action
        )
        actions[agent_id] = action

    obs, rewards, terminations, truncations, _ = env.step(actions)
    done = not env.agents or all(terminations.values()) or all(truncations.values())
    time.sleep(0.1)

What You Should See

In the GAMA GUI:

• Blue forager_0 navigates with its LIDAR cone around the left obstacles.
• Teal forager_1 follows a slightly different path due to its starting position.
• The first to arrive at the green food turns orange and freezes.
• When both are orange, the episode ends: cooperative success.

---

Running the Test Script

cd models/petz
python test_forager_petz.py

Expected Console Output

=======================================================
  Smart Forager — MARL Test (gama-pettingzoo GUI)
=======================================================
Model loaded (shared by both foragers)

Running 1 cooperative test episodes...

  Episode 1/1: ✓ COOPERATIVE SUCCESS! | Steps: 61
    forager_0: reward = 89.4
    forager_1: reward = 87.1

=======================================================
  Test Results Summary
=======================================================
  Episodes    : 1
  Success Rate: 100%
  Avg Steps   : 61
  forager_0 avg reward: 89.4
  forager_1 avg reward: 87.1
=======================================================

---

Full Tutorial Summary

Part 1 – Internal RL (GAML only)

Step	Concept introduced
1	Grid world with grid species
2	Forager agent with random movement
3	Reward function and episodes
4	Q-Table as map<string, float>
5	Q-Learning / Bellman equation
6	Charts, heatmap, test mode

Part 2 – Deep RL with Gymnasium

Step	Concept introduced
7	Continuous world, architecture overview
8	GymAgent bridge species
9	LIDAR ray-cast sensors, movement, reward shaping
10	Headless training with custom PyTorch PPO
11	GUI testing, deterministic evaluation

Part 3 – Multi-Agent RL with PettingZoo

Step	Concept introduced
12	PettingZoo Parallel API, PetzAgent bridge, cooperative rewards
13	Multi-agent GAML model, as_map, team obs, episode-end signal
14	Parameter-Shared PPO, batch inference, GamaParallelEnv directly
15	GUI testing, series recap

---

Key Concepts Across All 3 Parts

Concept	Part 1	Part 2	Part 3
World	10×10 grid	100×100 continuous	Same
Agents	1	1	2
Actions	4 discrete	2D continuous [dx, dy]	Same
Sensors	Grid position	8 LIDAR rays	8 LIDAR + teammate pos
Algorithm	Q-Learning	PPO	Parameter-Shared PPO
Bridge	None	GymAgent	PetzAgent
Library	None	gama-gymnasium	gama-pettingzoo
RL Framework	None	PyTorch (custom PPO)	Same
Task	Solo food	Solo food	Cooperative food

---

Key GAML Concepts Introduced in Part 3

PetzAgent, agents, possible_agents, observations, rewards, terminations, truncations, actions, update_data, as_map, all_match, contains_key, episode-end via agents <- []

Key Python Concepts Introduced in Part 3

GamaParallelEnv, env.agents, env.observation_space(agent_id), env.action_space(agent_id), env.reset() → dict, env.step(actions_dict) → dict, parameter sharing, batch inference, select_actions_batch(), per-agent RolloutBuffer

---

Key Files

File	Description
models/petz/forager_petz.gaml	GAMA model with PetzAgent bridge
models/petz/train_forager_petz.py	MARL training script (headless)
models/petz/test_forager_petz.py	Testing script (GUI visualization)