Skip to content

Commit bdf9a08

Browse files
hillol-nexthopclaude
hillol-nexthop
and
claude
committed
Add fboss-sim: a minimal fake-SAI FBOSS runtime container
fboss-sim packages the fake-SAI FBOSS agents and the fboss2 CLI into a small (~1.2 GB) CentOS Stream 9 Docker image, so FBOSS can be run and exercised end-to-end without any switching hardware. - docker/Dockerfile.runtime builds the runtime image from the CentOS Stream 9 base plus the existing FBOSS centos-09.0 rootfs overlay (systemd agent units, login env hook, eth0 IPv6 sysctl) and the collected binaries + shared libraries. - scripts/fboss-sim-docker-package.py collects the fake-SAI binaries and their ldd-resolved libraries and builds the image. It has two phases: --collect-only (run where ldd resolves the image-base libraries, i.e. inside the CentOS build image) and --build-only (run on the host that owns the Docker daemon); a flagless run does both. - scripts/fboss-sim-docker-run.py creates an IPv6 network and starts the runtime container in split or monolithic agent mode. A new GitHub Actions workflow (.github/workflows/fboss-sim.yml) builds the fake-SAI agent + fboss2 targets via docker-build.py, packages and starts the runtime image, and runs fboss2_integration_test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent fed7e21 commit bdf9a08

9 files changed

Lines changed: 1461 additions & 0 deletions

File tree

‎.github/workflows/fboss-sim.yml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
name: Build & Test fboss-sim
2+
run-name: Build & Test fboss-sim
3+
on:
4+
pull_request:
5+
paths:
6+
- 'fboss-sim/**'
7+
- 'fboss/cli/**'
8+
- '.github/workflows/fboss-sim.yml'
9+
# Allow manually triggering the workflow
10+
workflow_dispatch:
11+
12+
# Cancel superseded runs on the same ref.
13+
concurrency:
14+
group: ${{ github.workflow }}-${{ github.ref }}
15+
cancel-in-progress: true
16+
17+
permissions:
18+
contents: read
19+
20+
jobs:
21+
fboss-sim:
22+
if: github.repository_owner == 'facebook'
23+
runs-on: 32-core-ubuntu
24+
steps:
25+
- name: Check out repository code
26+
uses: actions/checkout@v6
27+
28+
# Build the fake-SAI agents and the fboss2 CLI / integration-test targets
29+
# into <workspace>/.build_dir (the getdeps scratch dir that
30+
# fboss-sim-docker-package.py reads). This also produces the CentOS
31+
# fboss_image:latest used for the collect step below. --target may be
32+
# repeated to build several targets sequentially into one scratch dir.
33+
- name: Build fake-SAI agent + fboss2 targets
34+
run: >
35+
sudo
36+
./fboss/oss/scripts/docker-build.py
37+
--scratch-path ${{ github.workspace }}/.build_dir
38+
--target fboss_fake_agent_targets
39+
--target fboss2_targets
40+
--no-docker-output
41+
--local
42+
--env-var BUILD_SAI_FAKE
43+
--num-jobs 18
44+
45+
- name: Restore workspace ownership
46+
run: sudo chown -R runner:runner ${{ github.workspace }}
47+
48+
# Phase 1: collect binaries + libs inside the CentOS fboss_image, so ldd
49+
# resolves the same system libraries the runtime image base (CentOS
50+
# Stream 9) provides. Writes <workspace>/tmp_build_dir.
51+
- name: Collect fboss-sim artifacts (inside CentOS build image)
52+
run: >
53+
sudo
54+
docker run --rm
55+
-v ${{ github.workspace }}:/var/FBOSS/fboss:z
56+
--cap-add=CAP_AUDIT_WRITE
57+
fboss_image:latest
58+
python3 /var/FBOSS/fboss/fboss-sim/scripts/fboss-sim-docker-package.py --collect-only
59+
60+
- name: Restore workspace ownership
61+
run: sudo chown -R runner:runner ${{ github.workspace }}
62+
63+
# Phase 2: build the runtime image on the host (where the Docker daemon is).
64+
- name: Build fboss-sim runtime image
65+
run: python3 fboss-sim/scripts/fboss-sim-docker-package.py --build-only
66+
67+
- name: Start fboss-sim runtime container
68+
run: python3 fboss-sim/scripts/fboss-sim-docker-run.py
69+
70+
- name: Wait for agent services to be ready
71+
run: |
72+
for i in $(seq 1 30); do
73+
if docker exec fboss_sim_runtime_${USER} /opt/fboss/bin/fboss2 show interface >/dev/null 2>&1; then
74+
echo "✓ Agent ready after $(( (i - 1) * 2 ))s"
75+
exit 0
76+
fi
77+
echo " [$(( (i - 1) * 2 ))s] not ready yet, retrying..."
78+
sleep 2
79+
done
80+
echo "❌ Agent not ready"
81+
docker exec fboss_sim_runtime_${USER} tail -n 50 /var/facebook/logs/fboss/wedge_agent.log || true
82+
exit 1
83+
84+
- name: Run fboss2 integration test
85+
run: docker exec fboss_sim_runtime_${USER} /opt/fboss/bin/fboss2_integration_test

‎fboss-sim/README.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# fboss-sim - FBOSS Simulator Runtime
2+
3+
Minimal Docker runtime (~1.2 GB) for FBOSS with fake SAI — no hardware required.
4+
5+
## Quick Start
6+
7+
```bash
8+
# 1. Build FBOSS with fake SAI (binaries land in .build_dir/build/fboss/).
9+
# Use the standard getdeps OSS build with the fake-SAI target.
10+
sudo ./build/fbcode_builder/getdeps.py build --allow-system-packages fboss
11+
12+
# 2. Build the runtime image.
13+
# The packager has two phases:
14+
# --collect-only : resolve binaries+libs into tmp_build_dir/ (ldd must run
15+
# in an OS matching the image base — CentOS Stream 9)
16+
# --build-only : docker build the image from tmp_build_dir/
17+
# If you built on a CentOS Stream 9 host, run both at once with no flag:
18+
python3 fboss-sim/scripts/fboss-sim-docker-package.py
19+
20+
# 3. Start the container
21+
python3 fboss-sim/scripts/fboss-sim-docker-run.py
22+
23+
# 4. Run integration test
24+
docker exec fboss_sim_runtime_${USER} /opt/fboss/bin/fboss2_integration_test
25+
```
26+
27+
### Split build/host environments
28+
29+
When FBOSS is built inside a CentOS build container on a different-distro host
30+
(e.g. Ubuntu), `ldd` must resolve libraries inside the container, but
31+
`docker build` needs the host's Docker daemon. Run the two phases separately:
32+
33+
```bash
34+
# collect inside the build container (CentOS libs resolve correctly)
35+
docker exec <build-container> python3 \
36+
fboss-sim/scripts/fboss-sim-docker-package.py --collect-only
37+
38+
# build the image on the host
39+
python3 fboss-sim/scripts/fboss-sim-docker-package.py --build-only
40+
```
41+
42+
## Agent Modes
43+
44+
**Split mode** (default): `fboss_sw_agent` + `fboss_hw_agent@0` run as separate systemd services.
45+
The hw_agent connects to sw_agent over IPv6 (`::1`). This requires the container to have a
46+
non-loopback IPv6 address — `fboss-sim-docker-run.py` handles this automatically by creating
47+
a user-defined IPv6-enabled Docker network (`fboss_sim_net_${USER}`, subnet `fd00:fb05:5::/64`).
48+
49+
**Monolithic mode**: single `wedge_agent` process.
50+
```bash
51+
docker exec fboss_sim_runtime_${USER} switch-agent-mode.sh mono
52+
```
53+
54+
## Useful Commands
55+
56+
```bash
57+
# Check agent status
58+
docker exec fboss_sim_runtime_${USER} systemctl status fboss_sw_agent
59+
docker exec fboss_sim_runtime_${USER} systemctl status fboss_hw_agent@0
60+
61+
# View agent logs
62+
docker exec fboss_sim_runtime_${USER} tail -f /var/facebook/logs/fboss/wedge_agent.log
63+
64+
# Run FBOSS CLI
65+
docker exec fboss_sim_runtime_${USER} /opt/fboss/bin/fboss2 show interface
66+
67+
# Open a shell
68+
docker exec -it fboss_sim_runtime_${USER} bash
69+
```
70+
71+
## Architecture
72+
73+
```
74+
fboss-sim/
75+
├── docker/
76+
│ ├── Dockerfile.runtime # CentOS Stream 9 minimal runtime image
77+
│ └── runtime/
78+
│ ├── mono.conf # Monolithic agent config
79+
│ ├── fruid.json # Platform identification (virtual env)
80+
│ ├── setup-container.sh # Runs at image build time: symlinks, jemalloc, services
81+
│ └── switch-agent-mode.sh # Toggle between split and monolithic mode
82+
└── scripts/
83+
├── fboss-sim-docker-package.py # Collect binaries+libs (via ldd), build runtime image
84+
└── fboss-sim-docker-run.py # Create IPv6 network, start runtime container
85+
```
86+
87+
`Dockerfile.runtime` copies the FBOSS `centos-09.0` rootfs overlay
88+
(`fboss-image/image_builder/templates/centos-09.0/root_files/`) for the systemd
89+
layout — the agent units the simulator runs (`fboss_sw_agent`,
90+
`fboss_hw_agent@`, the hw-agents target), the login-shell env hook, and the
91+
eth0 IPv6 sysctl. That rootfs is part of the FBOSS tree, so this directory
92+
has no external build dependencies.
93+
94+
### How `fboss-sim-docker-package.py` works
95+
96+
1. Verifies required binaries exist in `.build_dir/build/fboss/`
97+
2. **Collect phase** (`--collect-only`, or the first half of a flagless run):
98+
resolves each binary's shared libraries via `ldd` and copies binaries,
99+
libraries, and config files into `tmp_build_dir/`. This must run where the
100+
resolved libs match the image base OS (CentOS Stream 9).
101+
3. **Build phase** (`--build-only`, or the second half of a flagless run):
102+
builds the Docker image from `Dockerfile.runtime` using `tmp_build_dir/` as
103+
context, then cleans up `tmp_build_dir/`.
104+
105+
### Why IPv6 matters for split mode
106+
107+
`folly::SocketAddress::setFromLocalPort()` calls `getaddrinfo(nullptr, port, AI_ADDRCONFIG)`.
108+
`AI_ADDRCONFIG` only returns IPv6 results if the host has at least one non-loopback IPv6 address.
109+
Docker's default bridge sets `net.ipv6.conf.eth0.disable_ipv6=1` per-interface, so containers
110+
get no non-loopback IPv6 → Thrift servers bind to `0.0.0.0` only → hw_agent's connection to
111+
`::1` is refused. The user-defined network with `--ipv6` prevents this.
112+
113+
## Binaries Included
114+
115+
| Binary | Purpose |
116+
|--------|---------|
117+
| `wedge_agent-sai_impl` | Monolithic agent (fake SAI) |
118+
| `fboss_sw_agent` | Split mode: SW/control plane |
119+
| `fboss_hw_agent-sai_impl` | Split mode: HW/forwarding plane (fake SAI) |
120+
| `fboss2` | FBOSS CLI |
121+
| `fboss2-dev` | FBOSS dev CLI |
122+
| `fboss2_integration_test` | CLI integration test suite |
123+
| `setup_fboss_env` | Environment setup helper |
124+
125+
## Container Capabilities
126+
127+
- `CAP_NET_ADMIN`: TUN interface creation (TunManager)
128+
- `CAP_SYS_ADMIN`: systemd cgroup management
129+
- `/dev/net/tun`: virtual network interfaces
130+
- `--shm-size=512m`: shared memory for FBOSS IPC

‎fboss-sim/docker/Dockerfile.runtime‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Minimal FBOSS Simulator Runtime Image
2+
# This image contains only the essential runtime components (~1.2 GB vs 34 GB)
3+
# Build artifacts are copied from tmp_build_dir (extracted by fboss-sim-docker-package.py)
4+
5+
FROM quay.io/centos/centos:stream9
6+
7+
# Install all runtime dependencies in a single layer
8+
# - epel-release: provides EPEL repository for jemalloc
9+
# - systemd: provides systemctl for service management
10+
# - iproute: provides /sbin/ip for network interface management (used by wedge_agent)
11+
# - procps-ng: provides pkill for process management (used by AgentPreStartExec)
12+
# - jemalloc: alternative malloc implementation that's more robust against corruption
13+
# - jq: JSON processor for config manipulation
14+
# - vim, git: development/debugging tools
15+
RUN dnf install -y \
16+
epel-release \
17+
&& dnf install -y \
18+
systemd \
19+
vim \
20+
git \
21+
iproute \
22+
procps-ng \
23+
jemalloc \
24+
jq \
25+
&& dnf clean all && rm -rf /var/cache/dnf
26+
27+
# Create FBOSS directories
28+
RUN mkdir -p \
29+
/opt/fboss/bin \
30+
/opt/fboss/lib \
31+
/opt/fboss/share \
32+
/etc/coop \
33+
/root/config \
34+
/var/facebook/fboss \
35+
/var/facebook/logs/fboss \
36+
/dev/shm/fboss
37+
38+
# Copy FBOSS production rootfs overlay (systemd services, helper scripts, configs)
39+
COPY fboss-image/image_builder/templates/centos-09.0/root_files/ /
40+
41+
# Copy mono agent configuration (base config)
42+
COPY fboss-sim/docker/runtime/mono.conf /root/config/mono.conf
43+
44+
# Generate split.conf from mono.conf by setting multi_switch to true
45+
# Using jq for robust JSON manipulation (handles formatting, validates syntax)
46+
RUN jq '.defaultCommandLineArgs.multi_switch = "true"' \
47+
/root/config/mono.conf > /root/config/split.conf
48+
49+
# Copy fruid.json (platform identification for virtual environment)
50+
COPY fboss-sim/docker/runtime/fruid.json /var/facebook/fboss/fruid.json
51+
52+
# Copy agent mode switching script
53+
COPY fboss-sim/docker/runtime/switch-agent-mode.sh /usr/local/bin/switch-agent-mode.sh
54+
55+
# Copy container setup script
56+
COPY fboss-sim/docker/runtime/setup-container.sh /tmp/setup-container.sh
57+
58+
# Copy FBOSS runtime binaries and libraries from tmp_build_dir
59+
# The fboss-sim-docker-package.py script extracts fboss_bins.tar.zst to tmp_build_dir
60+
# Optimized layer ordering: rarely-changing content first, frequently-changing last
61+
62+
# Layer 1: Shared libraries (rarely change)
63+
COPY tmp_build_dir/lib/ /opt/fboss/lib/
64+
65+
# Register /opt/fboss/lib with the dynamic linker so binaries resolve their deps
66+
# without needing LD_LIBRARY_PATH to be set in the calling shell.
67+
RUN echo "/opt/fboss/lib" > /etc/ld.so.conf.d/fboss.conf && ldconfig
68+
69+
# Layer 2: Share configs and platform data (rarely change)
70+
COPY tmp_build_dir/share/ /opt/fboss/share/
71+
72+
# Layer 3: All FBOSS binaries (7 required binaries pre-filtered by fboss-sim-docker-package.py)
73+
# Includes: wedge_agent-sai_impl, fboss_sw_agent, fboss_hw_agent-sai_impl, fboss2, fboss2-dev, fboss2_integration_test, setup_fboss_env
74+
COPY tmp_build_dir/bin/ /opt/fboss/bin/
75+
76+
# Run container setup script
77+
# This configures: symlinks, jemalloc, multi-switch mode, service masking, etc.
78+
RUN chmod +x /tmp/setup-container.sh && \
79+
/tmp/setup-container.sh && \
80+
rm /tmp/setup-container.sh
81+
82+
ENV PATH="/opt/fboss/bin:/usr/local/bin:${PATH}" \
83+
LD_LIBRARY_PATH="/opt/fboss/lib" \
84+
FBOSS2_DEV_PATH="/opt/fboss/bin/fboss2-dev" \
85+
MALLOC_ARENA_MAX=1 \
86+
MALLOC_MMAP_THRESHOLD_=0 \
87+
MALLOC_CHECK_=0
88+
89+
STOPSIGNAL SIGRTMIN+3
90+
CMD ["/sbin/init"]

‎fboss-sim/docker/runtime/fruid.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"Actions": [],
3+
"Resources": [],
4+
"Information": {
5+
"Product Sub-Version": "1",
6+
"Facebook PCBA Part Number": "",
7+
"Product Version": "1",
8+
"Product Part Number": "VIRTUAL-001",
9+
"Extended MAC Address Size": "139",
10+
"Facebook PCB Part Number": "",
11+
"Product Name": "MONTBLANC",
12+
"Local MAC": "02:00:00:00:00:01",
13+
"PCB Manufacturer": "Virtual",
14+
"CRC8": "0x0",
15+
"System Assembly Part Number": "VIRT001",
16+
"ODM PCBA Serial Number": "",
17+
"Product Serial Number": "VIRTUAL-MONTBLANC-001",
18+
"ODM PCBA Part Number": "",
19+
"System Manufacturing Date": "2026031300",
20+
"Version": "0",
21+
"Location on Fabric": "Virtual",
22+
"Assembled At": "Container",
23+
"Product Production State": "0",
24+
"Product Asset Tag": "fboss_base_virtual",
25+
"Extended MAC Base": "02:00:00:00:00:00",
26+
"System Manufacturer": "FBOSS Virtual Environment"
27+
}
28+
}

0 commit comments

Comments
 (0)