Scripted tests that run a real client against a local vanilla server (26.2, the native version, unless you pick others), for behaviour unit tests can't reach: items arriving with their components, inventory clicks, entity data, advancements, and what ends up on screen.
Unit tests check pomme's pieces in isolation, such as a decoder against bytes built by hand. They can't show that a real server's packets end up as the right item in the right slot, drawn correctly. That used to need someone to check in game. These scripts do it against vanilla's own server, so the whole path is tested, and the screenshots catch what the checks miss.
A full run takes about half a minute, server start included, and needs no one at the keyboard, so it's cheap to rerun on every change.
just ingame-test # every script
just ingame-test entities # every script in a folder
just ingame-test player/armor_equip # one script
just ingame-test --jobs 2 # two at once (default 6)
just ingame-test --version 1.21.1 # against another version (repeatable)
just ingame-test --versions all world # every tested version, one folder--versions all is one version per joinable protocol, its newest patch
(pomme_protocol::version::tested_versions, 26.3 down to 1.20.1). A full
sweep takes a while: each version downloads its server jar once, and the client
installs that version's assets on its first join. Not every version passes yet:
versions below 1.21.5 don't load chunks, and no script is gated by version so
far, so a sweep shows what's still to fix rather than a green run.
Scripts live in a folder per area (blocks, entities, items, player,
ui, world); a new one goes in the folder it tests. A script's name is its
path without .toml.
The runner (tools/ingame-test):
- Looks each version up in Mojang's version manifest and downloads its
server jar into
target/ingame/servers/<version>/once, checking its sha1. - Starts it on a fresh flat world in offline mode, with the test player
Testeropped. Each version needs the Java itsversion.jsonnames (17 for 1.20.x, 21 for 1.21.x, 25 for 26.x):JAVA_HOME_<major>if set, elseJAVA_HOME, elsejavaonPATH. A newer Java runs older servers. - Runs a debug client per script, joining as that version, then resets the player for the next one, and the world to noon in clear weather with time ticking.
With --jobs N, N workers per version each start their own server and world
under target/ingame/workers/<version>/<n>/ and take scripts from a shared
queue. Their clients use a game dir there too, so a run never touches your own
options.
Each script writes report.json, its screenshots and the client's output
(client.log) to target/ingame/<version>/<script>/, e.g.
target/ingame/26.2/entities/data/. The run fails if any script does.
A script whose commands don't exist on older servers (item component syntax
like stone[custom_name=...] is 1.20.5+) declares the versions it's valid on,
above its first step, with a comment saying why:
# Item components in commands arrived in 1.20.5.
min_version = "1.20.5"max_version works the same way. Scripts outside the range print SKIP for
that version. Gate only for what the server can't do; a script failing because
of a client bug stays a failure.
A script about a Pomme extension, which vanilla has no screen or option for,
sets pomme_only = true the same way. The vanilla harness prints SKIP for
it instead of failing.
just wire-fixtures <version>... runs world/join on each version with
--capture: the client records every frame the server sends, before
translation, and the runner keeps the login and config frames plus the first
few of each game packet in pomme-client/src/net/wire-fixtures/<version>/.
net::wire_corpus replays them through the translation and the client's
readers in cargo test, so each version is checked against what its real
server sends. Recapture a version when its join traffic changes, such as after
a protogen or reference update for it. Packets a version is known to misread
are listed in KNOWN_GAPS there; once one reads, the test asks for its entry
to go.
The client side is the test-harness cargo feature: debug tooling that's off
by default and never in release builds. It adds two launch flags:
--harness-script <file>runs a script, then exits 0 on pass and 1 on fail.--harness-port <port>serves the same commands on127.0.0.1only, for driving a client by hand.
Commands go through the same paths as the UI. Chat goes through the chat sender, and clicks go through the inventory's click prediction and packets, so a test exercises the real code.
A script is a list of [[step]] tables, run in order once the client is in
game. After it leaves the game (the pause menu's Disconnect), only steps that
work on the title screens may follow: waits, screenshots, input, cursor_to_button
and screen checks.
Scripts start under the deterministic step, so both clients' shots hold
still and compare alike. A script about that noise itself (animated textures,
particles) sets deterministic = false above its first step, and can turn it
on later with the step.
Each step has exactly one action:
[[step]]
chat = "/give @s minecraft:iron_chestplate" # chat, or a command with `/`
[[step]]
open_inventory = true # or close_menu = true
[[step]]
click = { slot = 36, op = "quick_move" } # pickup_left, pickup_right,
# quick_move, { swap = { hotbar = 0 } }
[[step]]
creative_set_slot = { slot = 37, from = 36 } # creative mode: copy a stack, as
# the creative screen places one
[[step]]
look_at = [0.5, -58.5, 4.0] # turn to face a point
[[step]]
modifiers = { shift = true } # held for the clicks and presses
# that follow; {} releases
[[step]]
mouse_down = "left" # or mouse_up; left, right, middle,
# back, forward
[[step]]
hover = { slot = 36 } # creative item tab: hotbar slot
# 36-44, or { catalog = 0 };
# any tab: { tab = "inventory" };
# inventory tab: any slot 5-45,
# or { trash = true };
# offset = [x, y] GUI px from
# the target's centre
[[step]]
cursor = [2, 2] # cursor to a point, GUI px
[[step]]
cursor_to_button = "Disconnect" # cursor to the middle of the
# button with this label
[[step]]
scroll = -2 # mouse wheel over the open
# screen, lines (up is positive)
[[step]]
text = "stone" # types into the focused field
[[step]]
press = { action = "key.drop", ctrl = true } # a key binding's default key, or
# key = "key.keyboard.j" for a
# physical key; ctrl is optional
[[step]]
key_down = { action = "key.forward" } # held until key_up, which
# takes the same key; repeat =
# true is the system repeating
# a key already held
[[step]]
focus = false # the window losing focus,
# letting go of held keys and
# buttons, or gaining it
[[step]]
bind = { action = "key.drop", keys = ["key.keyboard.j"] } # this session only;
# no keys resets the defaults
[[step]]
option = { main_hand = "left" } # sets client options, as the
# options screen does; also
# menu_background_blur,
# brightness, render_distance
# (chunks), distant_terrain
# (chunks, 0 for off),
# show_subtitles, chunk_fade
# (seconds),
# inactivity_fps_limit
# (minimized or afk),
# shader_pack (a name in
# fixtures/shaderpacks, "" for
# none) and resource_pack (a name
# in fixtures/resourcepacks to
# select as the pack screen does,
# "-name" to deselect)
[[step]]
deterministic = true # holds flicker, particles,
# animated textures, clouds, view
# bobbing and the autosave
# indicator still for the rest of
# a script that started without
[[step]]
hold_packets = true # incoming packets wait unhandled
# until hold_packets = false, as
# behind a slow connection
[[step]]
chunk_arenas = { vertices = 65536, max = 1 } # chunk arenas this small, this
# many at most, so a small render
# distance fills them; a new size,
# or a cap under the arenas there
# are, remeshes everything (no-op
# on vanilla)
[[step]]
wait_ms = 1000
[[step]]
wait_ticks = 20 # game ticks rather than time;
# the commands after it run on
# the frame it ends, so each
# reaches the same tick every
# run, as `wait_ms` can't promise
[[step]]
frame_times = { ms = 5000 } # times the frames for 5 s; the
# report line gives the frame
# rate, its 1% low, the render
# encode time, the GPU wait, the
# GPU's own time per pass, and
# the sections and triangles drawn;
# set `deterministic = false` to
# time clouds and particles too
[[step]]
screenshot = "equipped" # target/ingame/<script>/equipped.png
keep_chat = true # chat is cleared before a shot,
# as F3+D clears it, unless kept
[[step]]
match_screenshots = ["first", "second"] # fails unless two shots match
# pixel for pixel, to within a
# shade per channel
max_differing_pixels = 100 # but for this many pixels
# (default 0)
[[step]]
expect = { slot = 6, item = "iron_chestplate" }
timeout_ms = 5000 # how long to retry (default 5 s)An expect is retried every frame until it holds or times out. It can check:
| Check | Fields |
|---|---|
| A menu slot | slot, item (air for empty), optional count, components (types present) and absent (types not present) |
| A living entity | entity (kind), optional variant (asset name, e.g. cold), name, baby, glowing, villager_type and profession |
| A dropped item | item_entity, optional count |
| An advancement | advancement (id) |
| A chat line | chat (text it contains) |
| The recipe book | recipes: at_least (that many recipes known), highlighted (that many still marked new) |
| A block | block (state name, e.g. air), at, optional block_entity (type) |
| A biome | biome (name, e.g. swamp), at |
| Settled terrain | terrain = "settled": every loaded column meshed and uploaded (none waiting for pool space nearer than the eviction frontier, or while another arena fits), and no two of the chunk arenas' ranges overlapping; terrain = "compacted" also waits for no chunk arena left to drain or free, and terrain = "evicted" for far columns waiting for room with the eye's own column drawn |
| Distant terrain | distant_terrain = { at = [x, z] }: the finest distant terrain section holding that block column is meshed and uploaded; gone = true for not (Pomme-only) |
| The local player | player = { ... } with any of health, food, air (the air supply in ticks, 300 when full), level, game_mode, effect, no_effect, max_health, raining, hand, dimension, day_time, position (within 0.01), y_rot (within 0.01), spectating (the kind of entity the camera follows, empty for none), post_effect (the spectator view effect drawn: creeper, spider, invert, or empty) and using_item (an item held in use, such as a raised shield) |
| The screen outside a world | screen (title, server_list, direct_connect, world_list, create_world), optional address (the Direct Connection field) |
| The open menu | menu (a container's menu type, e.g. generic_9x3, or inventory, creative, pause, sign (the sign editor), none), optional title; on the creative screen also tab (its vanilla id, e.g. building_blocks), row (the first catalog row shown) and search |
| A scrolling list | list: where an options screen's or Key Binds' list sits, top, end or middle (other lists don't report it yet) |
| The HUD text | hud = { ... } with any of title, subtitle and sidebar_title ("" when none is shown), boss_bars (every name, top to bottom), sidebar (every row as "name number", top to bottom), tab_list (every listed player as { name, game_mode? }, in tab order), chat_open and action_bar ("" when none shows) |
| A played sound | sound (event name, e.g. block.note_block.harp), among the last 64 played |
Slots are menu slots. In the player inventory, 5-8 are armor (head to feet),
9-35 the main inventory, 36-44 the hotbar and 45 the offhand. /give fills
the first free hotbar slot, 36.
just ingame-server # prints the port
just client-dev --features test-harness -- --version 26.2 --username Tester \
--quick-access-multiplayer 127.0.0.1:<server port> --harness-port 25599Then send one JSON command per line, and read one reply per line:
echo '{"cmd":"inventory"}' | nc 127.0.0.1 25599
# {"ok":true,"data":{"menu":"none","slots":[...],"cursor":null}}The commands are status, chat (text), open_inventory, close_menu,
click (slot, op), creative_set_slot (slot, from), inventory,
entities, advancements, player, hud, sounds, block (at), look_at (at),
mouse_down and mouse_up (button), press (action, a key binding id like key.drop, or key, an input id like key.keyboard.j; optional ctrl), key_down and key_up (the same action or key, held between them; key_down takes an optional repeat), bind (action, optional keys), option (main_hand: left or right; saved as closing its options screen would, so the server is told), modifiers (shift, held until the next modifiers), hover (slot 36-44 or catalog cell in a creative item tab, a tab id, or on the inventory tab a slot 5-45 or trash; optional offset), cursor (at, GUI px), cursor_to_button (label), scroll (y, wheel lines), text (text), chat_log and screenshot (path, optional keep_chat).