ForagerRL_step1

1. The Grid World

By Killian Trouillet

This is Part 1 of the internal Reinforcement Learning tutorial in GAMA.

Tutorial Overview

Step	Title	Concepts
1	The Grid World	grid, global, experiment, display
2	The Forager Agent	species, reflex, aspect, switch/match
3	Rewards and Episodes	Reward function, episode management, monitor
4	The Q-Table	map<string, float>, state representation
5	Q-Learning Algorithm	Epsilon-greedy policy, Bellman equation
6	Charts, Test Mode and Visualization	chart, heatmap, training stop, automatic test

---

Step 1: The Grid World

Content

This first step illustrates how to create a grid environment in GAMA. We define a 10×10 grid with round food and rectangular obstacles.

Formulation

• Definition of global parameters for grid size, food position, and obstacle positions
• Definition of a world_cell grid with is_food and is_obstacle attributes
• Display of the grid with colored cells: dark gray for obstacles, white for empty, and a green circle for the food

Model Definition

Global section

The global section defines the world parameters. We store the grid size, the food coordinates, and a list of obstacle positions as point values.

global {
    int grid_size <- 10;
    int food_x <- 9;
    int food_y <- 9;
    list<point> obstacle_positions <- [{2,2}, {3,2}, {2,3}, {6,4}, {7,4}, {7,5}];

• int grid_size <- 10: Defines the size of our grid (10 columns × 10 rows).
• list<point>: A list of coordinate pairs. Each {x, y} represents the position of an obstacle.

Initialization

In the init block, we configure the grid cells by setting their is_food and is_obstacle attributes. The operator grid_at allows us to access a specific cell by its coordinates.

    init {
        ask world_cell grid_at {food_x, food_y} {
            is_food <- true;
        }
        loop pos over: obstacle_positions {
            ask world_cell grid_at pos {
                is_obstacle <- true;
            }
        }
    }
}

• ask world_cell grid_at {x, y} { ... }: Selects the grid cell at coordinates (x,y) and executes the block on it.
• loop pos over: list { ... }: Iterates over each element of the list.

Grid definition

A grid in GAMA is a special type of species where each cell is an agent. We define two boolean attributes (is_food, is_obstacle) and a dynamic color attribute that uses the update facet to reflect the cell state at every simulation step.

grid world_cell width: 10 height: 10 neighbors: 4 {
    bool is_food <- false;
    bool is_obstacle <- false;
    rgb color <- #white update: is_obstacle ? rgb(60,60,60) : #white;
}

• width: 10 height: 10: Creates a 10×10 grid of cells.
• neighbors: 4: Uses Von Neumann neighborhood (up, down, left, right only — no diagonals).
• update: facet: Used here to color obstacles only. The food will be drawn separately.

Experiment

The experiment block defines how to run and visualize the simulation.

experiment smart_forager type: gui {
    output {
        display Grid_World title: "Grid World" {
            grid world_cell border: #lightgray;
            graphics "food" {
                ask world_cell where each.is_food {
                    draw circle(5) color: rgb(50, 180, 50);
                }
            }
        }
    }
}

• type: gui: A graphical experiment with a user interface.
• grid ... border: #lightgray: Displays the grid cells.
• graphics "food" { ... } : A graphics layer to draw objects manually. We ask every food cell to draw a circle at its location.

Complete Model

/**
* Name: SmartForager - Step 1: The Grid World
* Author: Killian Trouillet
* Description: This first step creates a simple 10x10 grid environment 
*              with food (green cell) and obstacles (dark gray cells).
* Tags: reinforcement-learning, grid, tutorial
*/

model SmartForager

global {
	int grid_size <- 10;
	int food_x <- 9;
	int food_y <- 9;
	list<point> obstacle_positions <- [{2,2}, {3,2}, {2,3}, {6,4}, {7,4}, {7,5}];
	
	init {
		// Place the food on the grid
		ask world_cell grid_at {food_x, food_y} {
			is_food <- true;
		}
		// Place the obstacles on the grid
		loop pos over: obstacle_positions {
			ask world_cell grid_at pos {
				is_obstacle <- true;
			}
		}
		write "Welcome to the Smart Forager!";
		write "Grid size: " + grid_size + "x" + grid_size;
		write "Food location: (" + food_x + ", " + food_y + ")";
		write "Number of obstacles: " + length(obstacle_positions);
	}
}

grid world_cell width: 10 height: 10 neighbors: 4 {
	bool is_food <- false;
	bool is_obstacle <- false;
	rgb color <- #white update: is_obstacle ? rgb(60, 60, 60) : #white;
}

experiment smart_forager type: gui {
	output {
		display Grid_World title: "Grid World" {
			grid world_cell border: #lightgray;
			graphics "food" {
				ask world_cell where each.is_food {
					draw circle(5) color: rgb(50, 180, 50);
				}
			}
		}
	}
}

---