- Table of Contents
- Architecture Overview
- Protobuf
- Conventions
- Fullsystem
- AI
- Thunderscope
- Simulator
- E-Stop
- Design Patterns
At a high-level, our system is split into several independent processes that communicate with each other. Our architecture is designed in this manner to promote decoupling of different features, making our software easier to expand, maintain, and test.
-
Fullsystem is the program that processes data and makes decisions for a team of robots. It manages Sensor Fusion, which is responsible for processing and filtering raw data, and the AI that makes gameplay decisions.
-
Thunderscope is an application that provides a GUI for visualizing and interacting with our software.
-
The Simulator provides a physics simulation of the world (robots, ball, and field), enabling testing of our gameplay when we don't have access to a real field. This process is optional and used only for development and testing purposes; in a real match, our system will receive data from SSL-Vision.
-
Thunderloop is the software that runs onboard our robots. It is responsible for coordinating communication between our AI computer and the motor and power boards in our robots. It is part our robot software architecture, which is documented here.
Our software is designed to interact with the following software developed and maintained by the RoboCup Small Size League:
- This is the shared vision system used by the Small Size League. It is what connects to the cameras above the field, does the vision processing, and transmits the positional data of everything on the field to our AI computers.
- The GitHub repository can be found here
- Sometimes referred to as the "Referee", this is another shared piece of Small Size League software that is used to send gamecontroller and referee commands to the teams. A human and/or computer auto-referee controls this application during the games to send the appropriate commands to the robots. For example, some of these commands are what stage the gameplay is in, such as
HALT,STOP,READY, orPLAY. - The GitHub repository can be found here
Protobufs or protocol buffers are used to pass messages between components in our system.
After building using Bazel, the .proto files are generated into .pb.h and .pb.cc files, which are found in bazel-out/k8-fastbuild/bin/proto.
To include these files in our code, we simply include proto/<protobuf_filename>.pb.h.
These are protobuf messages that we define and that are important for understanding how the AI works.
Primitives represent the low-level actions that a robot can execute. They are sent to the robots and can be "blindly" executed without knowledge of the World and/or the high-level gameplay strategy. Primitives are understood directly by our robot software, which will translate the primitive into motor and power board inputs to make the robot move, dribble, kick, and/or chip as instructed.
Each Primitive is a C++ class that generates an associated TbotsProto::Primitive protobuf message that can be sent to the robots.
Primitives act as the abstraction between our AI and our robot software. It splits the responsibility such that the AI is responsible for sending a Primitive to a robot telling it what it wants it to do, and the robot is responsible for making sure it does what it's told.
The TbotsProto::RobotStatus protobuf message contains information about the status of a single robot. Examples of the information they include are:
- Robot battery voltage
- Whether or not the robot senses the ball in the breakbeam
- The capacitor charge on the robot
- The temperature of the dribbler motor
Information about the robot status is communicated and stored as RobotStatus protobuf messages. Thunderscope displays warnings from incoming RobotStatus messages so we can take appropriate action. For example, during a game we may get a "low battery warning" for a certain robot, so we know to substitute it and replace the battery before it dies on the field.
Below documents various conventions we use and follow in our software.
We use a slightly custom coordinate convention to make it easier to write our code in a consistent and understandable way. This is particularly important for any code handling gameplay logic and positions on the field.
The coordinate system is a simple 2D x-y plane. The x-dimension runs between the friendly and enemy goals, along the longer dimension of the field. The y-dimension runs perpendicular to the x-dimension, along the short dimension of the field.
Because we have to be able to play on either side of a field during a game, this means the "friendly half of the field" will not always be in the positive or negative x part of the coordinate plane. This inconsistency is a problem when we want to specify points like "the friendly net", or "the enemy corner". We can't simple say the friendly net is (-4.5, 0) all the time, because this would not be the case if we were defending the other side of the field where the friendly net would be (4.5, 0).
In order to overcome this, our convention is that:
- The friendly half of the field is always negative x, and the enemy half of the field is always positive x
yis positive to the "left" of someone looking at the enemy goal from the friendly goal- The center of the field (inside the center-circle) is the origin /
(0, 0)
This is easiest to understand in the diagram below.
Based on what side we are defending, Sensor Fusion will transform all the coordinates of incoming data so that it will match our convention. This means that from the perspective of the rest of the system, the friendly half of the field is always negative x and the enemy half is always positive x. Now when we want to tell a robot to move to the friendly goal, we can simply tell it so move to (-4.5, 0) and we know this will always be the friendly side. All of our code is written with the assumption in mind.
Going along with our coordinate convention, we have a convention for angles as well. An Angle of 0 is along the positive x-axis (facing the enemy goal), and positive rotation is counter-clockwise (from a perspective above the field, looking at it like a regular x-y plane where +y is "up"). See the diagram below.
Because of our Coordinate Conventions, this means that an angle of 0 will always face the enemy net regardless of which side of the field we are actually defending.
Fullsystem processes data and makes decisions for a team of robots. It manages Sensor Fusion, which is responsible for processing and filtering raw data, and the AI that makes gameplay decisions.
Data within Fullsystem is shared between components using the observer pattern; for instance, Sensor Fusion and the Backend are Subjects that the AI observes.
Fullsystem contains a Backend responsible for all communication with the "outside world". The responsibilities of the Backend can be broken down into communication using SensorProto and Primitives messages:
-
Upon receiving the following messages from the network, the
Backendwill store it in aSensorProtomessage and send it to Sensor Fusion:- Robot status messages
- Vision data about where the robots and ball are (typically from SSL-Vision)
- Referee commands (typically from the SSL-Gamecontroller
-
Upon receiving Primitives from the AI,
Backendwill send the primitives to the robots or the Simulator.
The Backend was designed to be a simple interface that handles all communication with the "outside world", allowing for different implementations that can be swapped out in order to communicate with different hardware/ protocols/programs.
Sensor Fusion is responsible for processing the raw data contained in SensorProto into a coherent snapshot of the World that the AI can use. It invokes filters to update components of World, and then combines the components to send out the most up-to-date version.
The World class is what we use to represent the state of the world at any given time. In this context, the world includes the positions and orientations of all robots on the field, the position and velocity of the ball, the dimensions of the field being played on, and the current referee commands. Altogether, it's the information we have at any given time that we can use to make decisions.
A World is composed of the following classes:
A Team class represents a collection of Robots.
A Robot class represents the state of a single robot on the field. This includes its position, orientation, velocity, angular velocity, and any other information about its current state.
The Ball class represents the state of the ball. This includes its position and velocity, and any other information about its current state.
The Field class represents the state of the physical field being played on, which is primarily its physical dimensions. The Field class provides many functions that make it easy to get points of interest on the field, such as the enemy net, friendly corner, or center circle. Also see the coordinate convention we use for the field (and all things on it).
The GameState class represents the current state of the game as dictated by the Gamecontroller. These provide functions like isPlaying(), isHalted() which tell the rest of the system what game state we are in, and make decisions accordingly. We need to obey the rules!
Filters take the raw data from SensorProto and returns an updated version of a component of the World. For example, the BallFilter takes BallDetections and returns an updated Ball.
Why we need to do this: Programs that provide data like SSL-Vision only provide raw data. This means that if there are several orange blobs on the field, SSL-Vision will tell us the ball is in several different locations. It is up to us to filter this data to determine the "correct" position of the ball. The same idea applies to robot positions and other data we receive.
Filters provide a flexible way to modularize the processing of raw data, making it easy to update filters and add new ones. Filters are sometimes stateful. For example, the BallFilter "remembers" previous locations of the ball in order to estimate the ball's current velocity.
The AI is the part of the Fullsystem where all of our gameplay logic takes place, and it is the main "brain" of our system. It uses the information received from Sensor Fusion to make decisions, and then sends Primitives to the Backend for the robots to execute. Altogether, this feedback loop is what allows us to react to what's happening on the field and play soccer in real-time.
The two main components of the AI are strategy and motion planning.
We use a framework called STP (Skills, Tactics, Plays) to organize and implement our gameplay strategy. The STP framework was originally proposed by Carnegie Mellon University back in 2004. The original paper can be found here.
STP is a way of breaking down roles and responsibilities into a simple hierarchy, making it easier to build up more complex strategies from simpler pieces. This is the core of where our strategy is implemented.
The STP diagram shows how this works. Functions to assign tactics to robots and build motion constraints are passed into a Play's get function, which the Play uses to generate tactics with assigned robots and with updated motion constraints.
The S in STP stands for Skills. A Skill represents a lower-level behaviour that a robot can execute. Examples include:
- Moving to a position (without colliding with anything)
- Shooting the ball at a target
- Chipping the ball towards a target
When we refer to Skills in our AI, we typically are talking about Primitives, which are messages representing low-level actions that can be sent to and interpreted by our robots directly. We may also sometimes use the term Skill to describe simpler, skill-like Tactics that other Tactics are composed from (see Hierarchical Tactic FSMs).
The T in STP stands for Tactics. A Tactic represents a single robot's role on a team. Examples include:
- Being a goalie
- Being an attacker
- Being a pass receiver
- Being a defender that shadows enemy robots
- Being a defender that tries to steal the ball from enemies
Tactics return Primitives describing the low-level actions the robot should take as the Tactic. Every Tactic has an associated Tactic FSM (see Finite State Machines) that it uses to decide what Primitive to execute based on the state of the game; the FSM is updated with the current World and returns a Primitive to execute.
A Tactic by itself is not directly assigned to any single robot. Instead, Tactic::get returns a map associating each and every friendly robot with the Primitive they would execute if that robot was assigned that Tactic. We then say that every Primitive has some cost quantifying how easy it is for the robot to perform the action. For example, a MovePrimitive may incur a larger cost depending on the distance between the robot and the MovePrimitive's target position, reason being that it is in some sense "harder" for a robot to reach a faraway target successfully. With all this information, the Play assigns one robot to every Tactic based on the Primitives that would be executed, in such a way that the total cost incurred is minimized; this is a form of the classic assignment problem which we solve using the Hungarian algorithm.
In order to produce the map returned by Tactic::get, every Tactic must have one Tactic FSM for every robot. Each FSM tracks the state of the Tactic for one robot and produces the Primitive that the robot would execute.
Tactics FSMs may make use of other simpler Tactic FSMs that implement common behaviours. These simpler Tactics can be thought of as Skills since they do not represent any strategic role, but instead execute a well-defined action. For example:
MoveTacticsimply returns aMovePrimitiveconstructed using theMoveTactic's control parameters (which matchMovePrimitive's constructor parameters).PivotKickTacticapproaches the ball, pivots around the ball to face towards a specified target, and then kicks the ball in that direction.
The FSMs of these simpler Tactics are used as "sub-FSMs" and are treated as a state in more complex Tactic FSMs (read about hierarchical FSMs in the FSM section). When a Tactic FSM is in one of its sub-FSM states, it should update the sub-FSM with the World and return the Primitive returned by the sub-FSM. This means that the Tactic FSM will effectively "run" the sub-FSM and do its bidding while in the sub-FSM state, and it will leave the state once the sub-FSM terminates (e.g. MoveTacticFSM terminates once the robot has successfully reached the target destination).
This pattern promotes code reuse, allowing us to compose complex Tactic FSMs from smaller, skill-like Tactic FSMs that can be reused in different contexts.
Every Tactic defines its own custom ControlParams struct representing the specific settings and parameters we can set to influence the Tactic's behavior. At every timestep, alongside the current World, we update the Tactic's FSM with ControlParams carrying information about what want we want the Tactic to do (e.g. defend against a certain enemy robot). Tactics making use of other Tactics (as sub-FSMs) can use control parameters to control the behaviour of the sub-FSM, and Plays can set each of its Tactics' control params to coordinate responsibilities (e.g. tell each defensive Tactic to defend against a different enemy robot)
The P in STP stands for Plays. A Play represents a "team-wide goal" for the robots. They can be thought of much like plays in real-life soccer. Examples include:
- A play for taking friendly free kicks
- A play for defending enemy kickoffs
- A general defense play
- A passing-based offense play
- A dribbling-based offense play
Plays are made up of Tactics. A Play chooses which Tactics it wants to assign based on the state of the game. This logic is implemented in an associated Play FSM (see Finite State Machines) that gets updated with the current World and returns a list of Tactics to assign. The Play then attempts to assign robots to these desired Tactics in an optimal manner (see Tactic Assignment).
Plays return PrimitiveSets that map each robot to the Primitive they should execute. These Primitives are produced by the Tactics assigned by the Play to each robot.
The AI chooses which Play to run using PlaySelectionFSM. PlaySelectionFSM is updated with the current World and returns the Play to run based on the current game state.
A finite state machine (FSM) is a system with a finite number of states with defined transitions and outputs based on the inputs to the system. A FSM is in exactly one of its states at any time, and it can change (transition) from its current state to another in response to some input. In particular, we are interested in defining when a FSM should transition to another state (guards) and what should happen when transitions occur (actions), given a specific input (event).
Hierarchical state machines are state machines that are composed of one or more FSMs, which we call sub-FSMs. The parent FSM can treat a sub-FSM as a state with guards and actions when transitioning to and from the sub-FSM. When the sub-FSM enters a terminal state, the parent FSM is able to automatically transition to another state.
We use Boost-ext SML, short for State Machine Library, to create our finite state machines. This library defines state machines through a transition table, where a row indicates the transition from one state to another state using guards, actions and events. The syntax of a row of the transition table looks like this:
src_state + event [guard] / action = dest_state
where the src_state transitions to the dest_state, while performing the action, only if the event is received and the guard is true. Events are structs of new information that FSMs receive, and guards and actions are functions that take events as arguments. Guards must return a boolean and actions must return void. An asterix (*) at the start of a row indicates that the state is an initial state. The rows of the transition table are processed in order from top to bottom and the first row to match is executed.
The library also supports hierarchical FSMs. Sub-FSMs are treated as states where an unconditional transition occurs when the sub-FSM is in the terminal state, X.
const auto SubFSM_S = boost::sml::state<SubFSM>;
// ...in the transition table...
SubFSM_S = next_state // Transitions to next_state only when the SubFSM is in the terminal state, XIn order to update a sub-FSM with an event, we need to do the following:
const auto update_sub_fsm_action =
[](auto event, boost::sml::back::process<TypeOfSubFSMEvent> processEvent) {
TypeOfSubFSMEvent sub_fsm_event = // initialize the sub-FSM event
processEvent(sub_fsm_event);
};
// ...in the transition table...
SubFSM_S + event / update_sub_fsm_action
// When the parent FSM is updated with an event and it is
// in the SubFSM_S state, the update_sub_fsm_action will be
// invoked in the parent FSM. In turn, the update_sub_fsm_action
// will update the SubFSM with an event by calling processEventThe convenience of this syntax comes at the cost of hard to read error messages due to the functor and templating system.
We use SML to implement our Plays and Tactics. Each state represents a stage in the play/tactic where the team/robot should be doing a particular thing or looking for certain conditions to be true. A simple example of this is the MoveFSM. While the robot is not at the destination and oriented correctly, the FSM is in the move state. Once the robot reaches its destination, it enters the terminal state, X, to indicate that it's done. SML also allows us to easily reuse FSMs in other tactics. For example, if a shadowing tactic needs to move to a particular destination with a certain orientation, then it can use the MoveFSM as a sub-FSM state (see Hierarchical Tactic FSMs).
If you're having trouble understanding the SML syntax, take a look at docs/fsm-diagrams.md. It contains automatically generated diagrams of the FSMs in our codebase, which can help with visualizing the FSM control flow.
Boost-ext SML is a library that supports complex functionality with similarly complex syntax and semantics. If complex syntax is misused, the complicated error messages can make development difficult. Thus, we have carefully chosen a standardized subset of the library's syntax to implement our functionality while maintaining high readability.
-
Define guards and actions outside of the transition table: The names of guards and actions should be succinct so that transition tables rows fit on one line and readers can easily understand the FSM from the transition table. In other words, do not insert lambdas/anonymous functions directly in transition tables.
To aid with this, we have macros in
software/util/sml_fsm/sml_fsm.hthat generate named lambda wrappers around guard and action methods you define in your FSM struct. There are also macros for generating state and events that will be compatible with the SML library.struct KickFSM { // Define states inside the FSM struct class KickState; // Define an event for this FSM class Update; // Define your guards and actions as member functions bool isBallInDribbler(Update event); void kickBall(Update event); auto operator()() { using namespace boost::sml; DEFINE_SML_STATE(KickState) // Equivalent to: // const auto KickState_S = boost::sml::state<KickState>; DEFINE_SML_EVENT(Update) // Equivalent to: // const auto Update_E = boost::sml::event<Update>; DEFINE_SML_GUARD(isBallInDribbler) // Equivalent to: // const auto isBallInDribbler_G = [this](auto event) { return isBallInDribbler(event); }; DEFINE_SML_ACTION(kickBall) // Equivalent to: // const auto kickBall_A = [this](auto event) { kickBall(event); }; // You can use the macro-generated states, events, guards, and actions // in the transition table return make_transition_table( *KickState_S + Update_E[isBallInDribbler_G] / kickBall_A, ); } }; // We can instantiate this FSM and update it with an event FSM<KickFSM> fsm(KickFSM()); fsm.process_event(KickFSM::Update());
We also have macros for working with sub-FSMs:
struct KickFSM { class Update; // Define an action that will update the sub-FSM void moveToKickOrigin(Update event, boost::sml::back::process<MoveFSM::Update> processEvent) { // Make sure to call processEvent to update the sub-FSM! processEvent(MoveFSM::Update()); } auto operator()() { using namespace boost::sml; // Declare the sub-FSM as a state DEFINE_SML_STATE(MoveFSM) DEFINE_SML_EVENT(Update) // Similar to DEFINE_SML_ACTION -- generates moveToKickOrigin_A DEFINE_SML_SUB_FSM_UPDATE_ACTION(moveToKickOrigin) return make_transition_table( *MoveFSM_S + Update_E / moveToKickOrigin_A ); } }; // When instantiating an FSM with sub-FSMs, you need to // pass in instances of all the sub-FSMs structs along with the // parent FSM struct into the FSM constructor FSM<KickFSM> fsm(KickFSM(), MoveFSM());
-
Avoid entry and exit actions: these are actions that are always executed when entering or exiting a state. Everything that can be implemented with entry and exit actions can easily be implemented as actions, so this rule reduces source of confusion for the reader.
-
Avoid self transitions, i.e.
src_state + event [guard] / action = src_state: self transitions call entry and exit actions, which complicates the FSM. If we want a state to stay in the same state while performing an action, then we should use an internal transition, i.e.src_state + event [guard] / action. -
Avoid orthogonal regions: Multiple FSMs running in parallel is hard to reason about and isn't necessary for implementing single robot behaviour. Thus, only prefix one state with an asterix (*) so that there is only one initial state.
-
Use callbacks in events to return information from the FSM: Since the SML library cannot directly return information, we need to return information through callbacks. For example, if we want to return a double from an FSM, we can pass in
std::function<void(double)> callbackas part of the event and then make the action call that function with the value we want returned. -
When a variable needs to be shared between multiple states or can be initialized upon construction of the FSM, then define a private member and constructor in the FSM struct, and pass that in when constructing the FSM. Here's a code snippet:
struct DriveForwardFSM { public: DriveForwardFSM(double max_speed): max_speed(max_speed){} // ... private: double max_speed; } FSM<DriveForwardFSM> fsm(DriveForwardFSM(10.0));
Our motion planning (navigation) system is responsible for trajectory planning and obstacle avoidance.
A trajectory (or motion profile) describes the ideal motion that the robot should follow to get from one point to another point. It is a function of time that returns the target (a.k.a. reference) position, velocity, and acceleration for the given timestep. For every MovePrimitive (see Primitives) indicating a target destination, our trajectory planner generates and stores with the MovePrimitive a corresponding trajectory that the robot can take to reach that destination.
Our trajectory planner is heavily based off of TIGERs Mannheim's trajectory planner (read their 2019 TDP). At the most basic level, our trajector planner generates trapezoidal "bang-bang" trajectories. The name comes from the shape of the profile's speed vs. time graph (which is trapezoidal), and "bang-bang" refers to how the acceleration switches abruptly between zero and a fixed limit.
We create 2D trajectories by generating two 1D trajectories to describe the motion along each Cartesian coordinate axis over time. The two 1D trajectories are synchronized such that they complete in approximately the same time. We also generate a separate trajectory for the angular motion of the robot, describing how the robot's angular position, velocity, and acceleration should change over time.
The purpose of using trapezoidal trajectories is to decrease the amount of error that the PID motor controller has to correct for by making the target velocity closer to the motor's current velocity. If we instead had a constant velocity trajectory, the distance at the start between the motor's current velocity (at standstill) and the target velocity would be large, so the PID controller would issue a large motor input in response to the large error. Large motor inputs cause slip, leading to loss in traction and unpredictable movement.
Given a destination, the trajectory planner will generate a number of potential trajectories that will reach the destination. Each trajectory is scored based on its duration and whether it collides with any obstacles. The trajectory planner will return the trajectory with the best score.
We check whether a trajectory collides with any obstacles on the field by stepping over the trajectory in fixed steps (e.g. 100 ms) and checking whether the trajectory's position at any step lies inside an obstacle. We penalize a trajectory's score based on the amount of time until the first collision. This is because a collision that occurs earlier in a trajectory is more likely to happen (for real) compared to a collision occurring later in a trajectory, since obstacles may move and the state of the World may change by the time that later collision would have happened. That is, we cannot reliably predict where obstacles will be in the faraway future, so we are less harsh in penalizing trajectories with late collisions.
Trajectories are generated by sampling intermediate sub-destinations around the current robot position. One trajectory is generated from the current robot position and velocity to the sub-destination, and then additional trajectories to the destination are generated starting from sampled locations along the trajectory to the sub-destination. We can then form a trajectory path by connecting a portion of the trajectory to the sub-destination end-to-end with the trajectory to the destination. This approach enables generation of many possible trajectory paths, with the hope being that some will avoid obstacles that would otherwise be encountered in a direct trajectory to the destination.
Thunderscope refers to the GUI application we use to visualize and interact with our robots and software. Some non-UI related functionality (namely, Robot Communication) is implemented as part of Thunderscope since it acts as a central point in our architecture, making it convenient for coordinating activities between different modules.
thunderscope_main.py serves as the main entry point ("launcher") for our entire system. You can run the script to start up the Thunderscope GUI and a number of other optional processes, such as a Fullsystem for each AI team, the Simulator, and SSL Gamecontroller.
Thunderscope is our main visualizer of our AI. It provides a GUI that shows us the state of the World, and it is also able to display extra information that the AI would like to show. For example, it can show the planned paths of each friendly robot on the field, or highlight which enemy robots it thinks are a threat. Furthermore, it displays any warnings or status messages from the robots, such as if a robot is low on battery.
Thunderscope also lets us control the AI by setting Dynamic Parameters. The GUI lets us choose what strategy the AI should use, what colour we are playing as (yellow or blue), and tune more granular behaviour such as how close an enemy must be to the ball before we consider them a threat.
Thunderscope is implemented using PyQtGraph, a Python graphics and GUI library. PyQtGraph is built upon PyQt, which provides Python bindings for Qt, a C++ library for creating cross-platform GUIs. The general documentation for Qt can be found here. The most important parts of Qt to understand are:
Thunderscope has a field visualizer that uses PyQtGraph's 3D graphics system to render 3D graphics with OpenGL. PyQtGraph handles all the necessary calls to OpenGL for us, and as an abstraction, provides a scenegraph to organize and manipulate entities/objects within the 3D environment (the scene).
software/thunderscope/gl is the main directory for the 3D visualizer. The "GL" prefix lets us identify 3D graphics related code and keeps namings consistent with pyqtgraph.opengl class names. Inside this directory:
GLWidgetis the widget that displays our 3D visualization. It wraps a PyQtGraphGLViewWidgetthat renders all theGLGraphicsItems that have been added to its scenegraph./graphicscontains custom "graphics items". Graphics items (or just "graphics" for short) are objects that can be added to the 3D scenegraph. Graphics should inherit fromGLGraphicsItemand represent 3D objects that can be visualized in the scene (e.g. a robot, a sphere, a circle, etc.)./layerscontains all the layers we use to organize and group together graphics.
We organize our graphics into "layers" so that we can toggle the visibility of different parts of our visualization. Each layer is responsible for visualizing a specific portion of our AI (e.g. vision data, path planning, passing, etc.). A layer can also handle layer-specific functionality; for instance, GLWorldLayer lets the user place or kick the ball using the mouse. The base class for a layer is GLLayer.
A GLLayer is in fact a GLGraphicsItem that is added to the scenegraph. When we add or remove GLGraphicsItems to a GLLayer, we're actually setting the GLLayer as the parent of the GLGraphicsItem; this is because the scenegraph has a hierarchical tree-like structure. GLLayers can also be nested within one another, i.e. a GLLayer can be added as a child of another GLLayer.
Since Thunderscope runs in a separate process from Fullsystem, we use Unix domain sockets to facilitate communication between Fullsystem and Thunderscope. Unix sockets have high throughput and are very performant; we simply bind the unix socket to a file path and pass data between processes, instead of having to deal with TCP/IP overhead just to send and receive data on the same computer.
The data sent between Fullsystem and Thunderscope is serialized using protobufs. Some data, such as data that goes through our Backend (vision data, game controller commands, Worlds from Sensor Fusion, etc.), is sent using unix senders owned by those parts of the Fullsystem directly. In other higher level components of the Fullsystem (such as FSMs, pass generator, navigator, etc.), we want to delegate away the responsibility of managing unix senders directly and have a lightweight way of sending protobufs to Thunderscope. To avoid needing to dependency inject a "communication" object in places we have visualizable data to send to Thunderscope, we take advantage of the g3log logger already used throughout the codebase to log and send visualizable data.
g3log is a fast and thread-safe way to log data with custom handlers called “sinks". Importantly, it gives us a static singleton that can be called anywhere. Logging a protobuf will send it to our custom protobuf g3log sink, which lazily initializes unix senders based on the type of protobuf that is logged. The sink then sends the protobuf over the socket to listeners.
Aside: calling g3log to log protobuf data
Logging protobufs is done at the VISUALIZE level (e.g. LOG(VISUALIZE) << some_random_proto;). Protobufs need to be converted to strings in order to log them with g3log. We've overloaded the stream (<<) operator to automatically pack protobufs into a google::protobuf::Any and serialize them to a string, so you don't need to do the conversion yourself.
In Thunderscope, the ProtoUnixIO is responsible for communicating protobufs over unix sockets. ProtoUnixIO utilizes a variation of the publisher-subscriber ("pub-sub") messaging pattern. Through ProtoUnixIO, clients can register as a subscriber by providing a type of protobuf to receive and a ThreadSafeBuffer to place incoming protobuf messages. The ProtoUnixIO can then be configured with a unix receiver to receive protobufs over a unix socket and place those messages onto the ThreadSafeBuffers of that proto's subscribers. Classes can also publish protobufs (for other classes to receive or to send messages back to Fullsystem) via ProtoUnixIO by configuring it with a unix sender.
Fullsystem has a ProtoLogger that serializes and writes all incoming and outgoing protobufs to a folder. Thunderscope can be launched in a "replay mode" that will load and play back a proto log folder, allowing us to replay old matches and simulated tests.
Dynamic Parameters are the system we use to change values in our code at runtime through Thunderscope. The reason we want to change values at runtime is primarily because we may want to tweak our strategy or aspects of our gameplay very quickly. During games we are only allowed to touch our computers and make changes during halftime or a timeout, so every second counts! Using Dynamic Parameters saves us from having to stop the AI, change a constant, recompile the code, and restart the AI.
Dynamic Parameters are stored in one large protobuf message that is communicated between Thunderscope and Fullsystem. Thunderscope has a widget that displays all our Dynamic Parameters and lets the user change their values. Whenever a Dynamic Parameter value is changed in the widget, Fullsystem and AI will be updated with the new value. For example, we can define a Dynamic Parameter called run_ai that is a boolean value. The run_ai param will show up in the Parameters widget in Thunderscope with a checkbox that sets the value of run_ai. In the "main loop" for the AI, it will check if the value of run_ai is true before running its logic.
Here's a slightly more relevant example of how we used Dynamic Parameters during a game in RoboCup 2019. We had a parameter called enemy_team_can_pass, which indicates whether or not we think the enemy team can pass. This parameter was used in several places in our defensive logic, and specifically affected how we would shadow enemy robots when we were defending them. If we assumed the enemy team could pass, we would shadow between the robots and the ball to block any passes, otherwise we would shadow between the enemy robot and our net to block shots. During the start of a game, we had enemy_team_can_pass set to false but the enemy did start to attempt some passes during the game. However, we didn't want to use one of our timeouts to change the value. Luckily later during the half, the enemy team took a time out. Because Dynamic Parameters can be changed quick without stopping AI, we were quickly able to change enemy_team_can_pass to true while the enemy team took their timeout. This made our defence much better against that team and didn't take so much time that we had to burn our own timeout. Altogether this is an example of how we use Dynamic Parameters to control our AI and other parts of the code.
It is worth noting that constants are still useful, and should still be used whenever possible. If a value realistically doesn't need to be changed, it should be a constant (with a nice descriptive name) rather than a Dynamic Parameter. Having too many Dynamic Parameters is overwhelming because there are too many values to understand and change, and this can make it hard to tune values to get the desired behaviour while under pressure during a game.
Our simulator is what we use for physics simulation to do testing when we don't have access to real field. The simulator is a standalone application that simulates the following components' functionalities:
- SSL-Vision by publishing new vision data
- the robots by accepting new Primitives
Using the current state of the simulated world, the simulator simulates the new Primitives over some time step and publishes new SSL vision data based on the updated simulated world. Since the simulator interfaces with the our software over the network, it is essentially indistinguishable from robots receiving Primitives and an SSL-Vision client publishing data over the network.
The simulation can also be configured with "realism" parameters that control the amount of noise added to vision detections, simulate missed vision detections and packet loss, and add artificial vision/processing delay, all with some degree of randomness.
The code for our simulator is adapted from the ER Force Simulator built by another team in our league. The simulator uses the real-time Bullet physics engine, which simulates collision detection and soft/rigid-body dynamics in 3D.
When it comes to gameplay logic, it is very difficult if not impossible to unit test anything higher-level than a Tactic (and even those can be a bit of a challenge). Therefore if we want to test Plays we need a higher-level integration test that can account for all the independent events, sequences of actions, and timings that are not possible to adequately cover in a unit test. For example, testing that a passing play works is effectively impossible to unit test because the logic needed to coordinate a passer and receiver relies on more time-based information like the movement of the ball and robots. We can only validate that decisions at a single point in time are correct, not that the overall objective is achieved successfully.
Ultimately, we want a test suite that validates our Plays are generally doing the right thing. We might not care exactly where a robot receives the ball during a passing play, as long as the pass was successful overall. The solution to this problem is to use simulation to allow us to deterministically run our entire AI pipeline and validate behaviour.
The primary design goals of this test system are:
- Determinism: We need tests to pass or fail consistently
- Test "ideal" behaviour: We want to test the logic in a "perfect world", where we don't care about all the exact limitations of our system in the real world with real physics. Eg. we don't care about modelling robot wheels slipping on the ground as we accelerate.
- Ease of use: It should be as easy and intuitive as possible to write tests, and to understand what they are testing.
The SimulatedTestFixture consists of three components:
- A Simulator that does physics simulations based on the Primitives it receives
- A Sensor Fusion that processed raw data for the AI
- The AI under test
The components of this system are run in a big loop. The Simulator publishes new vision data with a fixed increment and that data is passed through Sensor Fusion to produced an updated World for the AI. The SimulatedTestFixture will wait to receive Primitives before triggering simulation and publishing the next World. This means that no matter how much faster or slower the simulation runs than the rest of the system, everything will always happen "at the same speed" from the POV of the rest of the system, since each newly published World will be a fixed amount of time newer than the last. See the section on Component Connections and Determinism for why this is important.
Validation Functions are the way we check that the behaviour of the AI is as we expect. They are essentially functions that contain Google Test ASSERT statements, and use Coroutines to maintain state. We can create a list of Validation Functions that are run every time SimulatedTestFixture produces an updated World, so that we can continuously check that the behaviour of the AI is as we expected. See the section on Component Connections and Determinism for how this is used and why this is important.
A Validation Function comes in two types:
- Terminating: Some sequence of states or actions that must occur in a given order and then terminate. (Eg. A robot moves to point A, then point B, and then finally kicks the ball)
- Non-terminating: Some condition that must be met for the entire duration of the test, i.e. "for all time". (Eg. The ball never leaves the field or all friendly robots follows SSL rules).
Benefits of Validation Functions:
- They are reusable. We can write a function that validates the ball never leaves the field, and use it for multiple tests.
- They can represent assertions for more complex behaviour. For example, we can build up simpler
Validation Functionsuntil we have a singleValidationfunction for a specific Tactic. - They let us validate independent sequences of behaviour. For example, we create a different
ValidationFunctionfor each Robot in the test. This makes it easy to validate each Robot is doing the right thing, regardless if they are dependent or independent actions.
When testing, determinism is extremely important. With large integration tests with many components, there are all kinds of timings and execution speed differences that can change the behaviour and results. We make use of a few assumptions and connect our components in such a way that prevents these timing issues.
The Validation Functions are run before updating AI with a new World and getting new primitives so that we stop as soon as we know that there's incorrect behaviour. See the diagram.
Now we have a nice loop from the Simulator -> Sensor Fusion -> Validation Functions -> AI -> Simulator .... As mentioned in their own sections, the Simulator waits to receive Primitives from the AI before publishing a new World, and the Validation Functions wait to receive and validate a World allowing the new world to be published. The final assumption we make to complete this loop is that the AI waits to receive a new World before publishing new Primitives.
What this means is that each component in the loop waits for the previous one to finish its task and publish new data before executing. As a result, no matter how fast each component is able to run, we will not have any issues related to speed or timing because each component is blocked by the previous one. As a result, we can have deterministic behaviour because every component is running at the same speed relative to one another.
Notice this is very similar to the Architecture Overview Diagram, with the Backend replaced by the Simulator and with Validation Functions in the loop between Sensor Fusion and AI.
Thunderscope and connections to it are marked with dashed lines, since they are optional and are not run during the tests unless we are debugging.
The E-Stop allows us to quickly and manually command physical robots to stop what they are doing. It is a physical push button that is connected to the computer via a USB cable. We also have a --keyboard_estop flag you can use when running Thunderscope that lets you use the spacebar as the E-Stop. When Thunderscope is launched, a ThreadedEstopReader is initialized (within RobotCommunication) that is responsible for communicating and reading values from the E-Stop via UART. While running, it will poll the status of the E-Stop to determine whether it is in the STOP or PLAY state:
- If the E-Stop is in the
STOPstate, it overrides the Primitives sent to the robots withStopprimitives. On the robot, Thunderloop is responsible for handling the primitive message and ensuring that the power & motor boards receive the correct inputs for the robot to stop. - If the E-Stop is in the
PLAYstate, primitives are communicated as normal.
Below are the main design patterns we use in our code, and what they are used for.
While not a "design pattern" per se, inheritance and related OOP paradigms are prevalent throughout of our code base. Abstract classes let us define interfaces for various components of our code. Then we can implement different objects that obey the interface, and use them interchangeably, with the guarantee that as long as they follow the same interface we can use them in the same way.
To learn more about inheritance in C++, read the following learncpp.com articles:
Examples of this can be found in many places, including:
The Singleton pattern is useful for having a single, global instance of an object that can be accessed from anywhere. Though it's generally considered an anti-pattern (aka bad), it is useful in specific scenarios. Read the Refactoring Guru article on the Singleton pattern for more information.
We use the Singleton pattern for our logger. This allows us to create a single logger for the entire system, and code can make calls to the logger from anywhere, rather than us having to pass a logger object literally everywhere.
The Factory pattern is useful for hiding or abstracting how certain objects are created. Read the Refactoring Guru articles on the Factory Method pattern and the Abstract Factory pattern for more information.
Because the Factory needs to know about what objects are available to be created, it can be taken one step further to auto-register these object types. Rather than a developer having to remember to add code to the Factory every time they create a new class, this can be done "automatically" with some clever C++ code. This helps reduce mistakes and saves developers work. Read this post about auto-registering factories for more information.
The auto-registering factory is particularly useful for our PlayFactory, which is responsible for creating Plays. Every time we run our AI we want to know what Plays are available to choose from. The Factory pattern makes this really easy, and saves us having to remember to update some list of "available Plays" each time we add or remove one.
The Visitor pattern is useful when we need to perform different operations on a group of "similar" objects, like objects that inherit from the same parent class (e.g. Tactic). We might only know all these objects are a Tactic, but we don't know specifically which type each one is (eg. AttackerTactic vs ReceiverTactic). The Visitor Pattern helps us "recover" that type information so we can perform different operations on the different types of objects. It is generally preferred to a big if-block with a case for each type, because the compiler can help warn you when you've forgotten to handle a certain type, and therefore helps prevent mistakes.
Read the Refactoring Guru article on the Visitor pattern for more information.
An example of where we use the Visitor pattern is in our MotionConstraintVisitor. This visitor allows us to update the current set of motion constraints based on the types of tactics that are currently assigned.
The Observer pattern is useful for letting components of a system "notify" each other when something happens. Read the Refactoring Guru article on the Observer pattern for a general introduction to the pattern.
Our implementation of this pattern consists of two classes, Observer and Subject. Observers can be registered with a Subject, after which new values will be sent from each Subject to all of it's registered Observers. Please see the headers of both classes for details. Note that a class can extend both Observer and Subject, thus receiving and sending out data. In this way we can "chain" multiple classes.
In our system, we need to be able to do multiple things (receive camera data, run the AI, send commands to the robots) at the same time. In order to facilitate this, we extend the Observer to the ThreadedObserver class. The ThreadedObserver starts a thread with an infinite loop that waits for new data from Subject and performs some operation with it.
Warning
If a class extends multiple ThreadedObservers (for example, AI could extend ThreadedObserver<World> and ThreadedObserver<RobotStatus>), then there will be two threads running, one for each observer. We do not check for data race conditions between observers, so it's entirely possible that one ThreadedObserver thread could read/write from data at the same time as the other ThreadedObserver is reading/writing the same data. Please make sure any data read/written to/from multiple ThreadedObservers is thread-safe.
One example of this is SensorFusion, which extends Subject<World> and the AI, which extends ThreadedObserver<World>. SensorFusion runs in one thread and sends data to the AI, which receives and processes it another thread.
The publisher-subscriber pattern ("pub-sub") is a messaging pattern for facilitating communication between different components. It is closely related to the message queue design pattern.
In this pattern, Publishers send messages without knowing who the recipients (Subscribers) are. Subscribers express interest in specific types of messages by subscribing to relevant topics; when a Publisher sends a message of a topic, the messaging system ensures that all interested subscribers receive the message.
Read the Wikipedia article on pub-sub for an introduction to the pub-sub pattern.
We use the pub-sub pattern to facilitate inter-process communication in our system. Through a class called ProtoUnixIO, components can subscribe to receive certain Protobuf message types sent out by other processes or system components.
While debatably not a design pattern depending on who you ask, templating in C++ is a powerful tool that is very useful to understand. We use templating throughout around our codebase. learncpp.com has a good explanation on function templates and class templates.