Skip to content

field-manual.md

Latest commit

 

History

History
1562 lines (1190 loc) · 84 KB

field-manual.md

File metadata and controls

1562 lines (1190 loc) · 84 KB
Raw

Real-World IoT Firmware Vulnerability Research Field Manual

Date: 2026-05-29

Purpose: a challenge-driven operating manual for turning difficult IoT firmware into a defensible vulnerability finding.

Use this after the quickstart when a case hits extraction, emulation, hardware, or proof-quality friction:

Compact Menu

Operating Principle

Real firmware research is not one tool. It is a loop:

observe -> classify -> extract -> map -> hypothesize -> verify -> reproduce -> explain -> report

Before choosing another tool, write:

What exact question am I trying to answer next?
What artifact would answer it?
What is the smallest tool or experiment that can produce that artifact?

Examples:

Bad question Better question
Can EMBA find a bug? Which network-exposed parser or web handler accepts attacker input?
Can I emulate the firmware? Do I need runtime proof, or is static reachability enough for this finding?
Is this CVE real? Is the exact vulnerable component/version present, built with the vulnerable feature, and reachable?
Can AI analyze this binary? What function, call chain, strings, and xrefs should AI summarize with addresses?

Authorization And Safety

Use this only on firmware and devices you are authorized to analyze.

Default safety posture:

Unknown firmware services: host-only network
Unknown binaries: VM/container/snapshot
Unknown samples: no shared folders unless read-only
Cloud upload: only if scope and confidentiality allow it
AI upload: only selected snippets/logs, never proprietary bulk dumps by default

Safe proof rules:

  • For command injection, prove with id, echo, or sleep, not destructive commands.
  • For path traversal, read harmless files such as /etc/passwd in a lab rootfs, not private host files.
  • For crashes, prove control and root cause; do not claim code execution unless you can support it.
  • For credentials, prove where the credential is consumed before claiming impact.
  • For cloud/mobile coupling, do not attack real vendor infrastructure.

Case Loop

Use this loop for every case:

1. Preserve original
2. Extract what can be extracted
3. Build an attack-surface map
4. Select high-reachability leads
5. Prove one lead deeply
6. Write limitations honestly
7. Archive artifacts so the result is reproducible

The deliverable is a chain:

reachable input
  -> code path
  -> vulnerable operation
  -> controlled trigger
  -> impact
  -> constraints

Treat every other artifact as support for one edge in this chain.

Real-World Example Map

Use public writeups as process traces, not payload libraries:

What was the objective?
What artifact did they collect next?
What uncertainty did that artifact remove?
What proof moved the claim from lead to demonstrated impact?

Study direct IoT/firmware examples first:

Research goal Real-world example Evidence pattern to reuse
Build a WAN-to-LAN IoT exploit chain Claroty Team82, Pwn2Own: WAN-to-LAN Exploit Showcase, Part 1 They treated the router as the first boundary, then reasoned about how a WAN foothold changes the LAN threat model. Copy the sequence: identify exposed router function, prove router impact, then explicitly map the pivot path to the downstream IoT device.
Pivot from router access to an IP camera Claroty Team82, Pivoting from WAN to LAN to Attack a Synology BC500 IP Camera They downloaded vendor firmware, identified architecture, rootfs, web server config, CGI handlers, and the parser reached by an unauthenticated route. Copy the firmware-to-route-to-parser chain before making exploitability claims.
Turn router firmware into a Pwn2Own exploit path Neodyme, Your router might be a security nightmare: Tales from Pwn2Own Toronto 2022 They downloaded official firmware, extracted it, got a lab shell, used runtime process/network data to shrink the surface, then used Ghidra scripting and string search to focus on one service. Copy the "map broadly, then narrow ruthlessly" sequence.
Keep going when firmware is encrypted and emulation fails Neodyme, RCE on the HP M479fdw printer They failed to emulate a hardware-dependent service, then pivoted to static analysis of the update/decryption path and IPC flow. Treat failed emulation as a constraint that selects the next smallest static question.
Analyze a WAN-reachable router daemon Synacktiv, Pwn2Own Austin 2021: Defeating the Netgear R6700v3 They selected a default-started service, understood its update flow, proved the crash condition, then reasoned about mitigations and reliability. Record daemon lifecycle, trigger timing, and restart behavior.
Use focused service emulation for exploit validation Claroty Team82, Hack The Emulated Planet They extracted firmware, selected the Boa management service, mapped dispatcher.cgi as a pre-auth CGI target, and used QEMU/chroot plus QEMU GDB support to validate the vulnerable cookie path. Reuse this when route, auth state, and sink are already known.
Prove reachability through an optional feature NCC Group, BrokenPrint: A Netgear stack overflow They did not stop at "the service exists"; they tied reachability to ReadySHARE being enabled by a connected USB printer. Record feature preconditions and configuration assumptions.
Chain IoT protocol research into LFD/RCE/privilege escalation SpaceRaccoon, Getting a Shell on the Tapo C260 Camera The researcher separated broad hunting from end-to-end feature tracing, started with logging strings and source functions, identified request handlers, then proved concrete file disclosure and execution paths. Reuse the source-to-sink discipline.
Combine hardware access, extraction cleanup, static analysis, and selective emulation SpaceRaccoon, Hacking the Nokia Beacon 1 Router The path went from teardown and UART to NAND dumping, removing out-of-band bytes, extracting UBI, finding web CGI sinks, and using Qiling only for the password-generation routine. Copy the "emulate one function, not the whole device" instinct.
Recover JFFS2 partitions from raw NAND Claroty Team82, Avaya IP Phone Security Research, Part A They found JFFS2 filesystems, reconstructed a partition table, used nandsim/nandwrite/mtdblock to mount partitions, then had to reason about OOB/ECC before writing back. Copy the lesson that OOB/ECC can be the deciding blocker, not a footnote.
Rebuild a UBI volume to enable local access Unauthorized Access, No clouds, just sunshine The Connexoon work reads NAND with SAM-BA, uses nandsim, ubiattach -O 2048, mounts root/rootB, rebuilds with ubinize, and then validates local API behavior. Reuse the rootfs/UBI-volume evidence chain, not the device-modification steps.
Reconstruct raw NAND-backed persistent state Local Gemtek WVRTM-127ACN case, README, set-nandsim.sh The project dumped full NAND, split Config/Env/Kernel/Storage partitions, matched page/erase geometry, attached mtd8 as UBI, and mounted /mnt/jffs2. Copy the decision rule: simulate MTD only when raw flash state is material to the lead.
Modify unsigned rootfs while kernel remains signed Local NB6VAC case, README, root-firmware-NB6VAC, root note The project shows UART-assisted observation, WFI tag extraction/rebuild, JFFS2/UBI splitting, nandsim/UBIFS mounting, rootfs password modification, and an OTP MCU that blocks open firmware. Copy the split finding: rootfs modification is possible, secure-boot bypass remains a separate MCU/root-of-trust question.
Avoid repeated NAND writes during device bring-up lp0, HiBy R2 DOOM The author used boot logs, vendor source/toolchain evidence, and pivot_root to move runtime writes to removable media. Copy the decision: sometimes the right storage experiment is to reduce NAND dependence rather than emulate it.
Treat automotive update packages as layered artifacts psa-nac-firmware-reverse-engineering, Continental OVIP firmware notes The scripts decrypt PSA/Stellantis NAC firmware files and mount UBIFS rootfs images. Copy the layered workflow: archive -> decrypt -> filesystem mount -> component/update-chain review.
Build a compatible analysis OS instead of a faithful device Local emulation folder, ARMX The local Buildroot examples and ARMX import vendor rootfs/firmware into a controlled QEMU guest with compatible kernel/libc/debug/storage tooling. Use this when old ABI/tooling compatibility matters more than exact board fidelity.
Reverse a non-Linux embedded firmware format Quarkslab, Reverse Engineering a VxWorks OS Based Router The work shows how proprietary RTOS firmware breaks the usual Linux-rootfs assumptions. Track file-format recovery, architecture selection, strings, memory layout, and failed extraction theories.
Build a firmware-focused wireless-chipset research loop Quarkslab, Reverse-engineering Broadcom wireless chipsets The path combined driver understanding, firmware analysis, reproducing public bugs, partial emulation, fuzzing, and coordinated disclosure. Copy the loop when your target is a component firmware rather than a whole router filesystem.
Chain cloud connectivity into industrial IoT firmware impact Cisco Talos, A deep dive into WAGO's cloud connectivity They traced intended cloud trust boundaries, web-management configuration, firmware-update behavior, and local service parsing. Copy the end-to-end model: cloud command source, local file placement, service parser, privilege context.
Triage n-days across OT/IoT router firmware Forescout Vedere Labs and Finite State, Rough Around The Edges They used firmware images, component identification, version age, vendor advisory checks, exploit availability, and manual validation to avoid false positives. Copy this for SBOM/CVE triage: reachable and built-in matters more than a scary CVSS score.
Analyze malware/backdoors in an IoT infection campaign Check Point Research, IoTroop Botnet: The Full Investigation They combined sensor data, infrastructure mapping, sample classification by architecture, C2 protocol analysis, and second-stage payload review. Copy the separation between device vulnerability, malware behavior, infrastructure, and attribution confidence.
Investigate compromised routers as malware infrastructure Microsoft Defender for IoT, Uncovering Trickbot's use of IoT devices in command-and-control infrastructure They connected traffic observations, router-specific command behavior, compromised MikroTik devices, and detection logic. Use this pattern when the "firmware case" is an incident-response question around abused network devices.
Analyze an IoT camera supply-chain component Nozomi Networks Labs, ThroughTek P2P Supply Chain Vulnerability They started from network traffic in a device lab, identified a reused P2P SDK, and framed risk across many OEM devices. Copy the vendor-component mindset: sometimes the bug lives in a shared SDK rather than the branded device.
Practice QEMU firmware emulation tradeoffs Applied Security, IoT Firmware Emulation with Qemu They compared manual QEMU setup with automatic emulation frameworks and documented why services did or did not start. Copy the troubleshooting habit: compare images, init behavior, mounts, NVRAM/config expectations, and service startup deltas.

Use CTF and lab writeups for repetition after the real-case examples:

Practice goal Example What to copy into your own workflow
Solve a firmware/hardware CTF by modeling the system, not guessing flags CTFtime, HITCON CTF 2021 chaos-firmware writeup The team first drew the architecture: modified QEMU, virtual PCI device, driver, sandbox, and firmware. Only after that did they attack the correct boundary. Copy the architecture-first writeup style for complex emulation targets.
Solve a hardware CTF by mixing observation, bus isolation, firmware RE, and controlled experiments GitHub, RHME3 writeups The writeups show CAN observation, schematic reading, bus isolation, UDS enumeration, ROM dumping, simulator use, and careful patch/pivot decisions. Write down failed assumptions and why a later path superseded them.
Analyze raw microcontroller flash CTFtime, justCTF 2024 Budget SoC The writeup starts from a raw flash dump, recognizes ESP32 structure, converts it into something decompilers can use, then follows constants and storage artifacts. Copy this for bare-metal or RTOS cases where there is no neat rootfs.
Emulate and reverse a router admin binary CTFtime, Pwn2Win 2016 Suspect Router The writeup identifies OpenWrt/MIPS signals, sets up emulation, creates signatures, models data structures, and debugs behavior. Use this pattern when a single service binary matters more than full-system boot.
Reason about CGI-style embedded web logic CTFtime, BSidesSF Steel Mountain: Sensors The challenge is small, but the workflow transfers: find exposed firmware/binaries, identify CGI behavior, inspect path construction, and reject failed exploit ideas. Preserve why a promising path did not work.

Keep these broader examples as transferable methodology, not as IoT firmware substitutes:

Transferable pattern Example What to copy into IoT firmware work
Find and explain an in-the-wild 0-day Google Project Zero, Bad Binder: Android In-The-Wild Exploit They started from credible external signals, reviewed patches and requirements, narrowed the candidate to one root cause, then explained why the evidence justified urgent treatment. Tie every claim to a specific patch, primitive, and exploit precondition.
Revisit an n-day/1-day after public advisories and improve the chain Source Incite, ZohOwned: A Critical Authentication Bypass on Zoho ManageEngine Desktop Central The writeup starts from public incident/advisory material, reconstructs the known chain, lists its limitations, then finds a better reachable path. Copy the limitation-first mindset before claiming exploitability.
Detect a supply-chain backdoor from weird runtime symptoms Openwall oss-security, backdoor in upstream xz/liblzma leading to ssh server compromise The investigation began with slow SSH logins and valgrind errors, then compared release tarballs against source, traced build-time injection, and used runtime tracing to explain impact. Treat small anomalies as leads that need measurement.

Reading order for examples:

  1. Start with one example that matches your current blocker.
  2. Write a four-line summary: goal, artifact, proof, limitation.
  3. Translate one research move into your case notes.
  4. Do not copy payloads into unauthorized targets.
  5. Add the example to your final report only if it explains your method, not as authority for an unproven claim.

First-Hour Dashboard

Within the first hour, try to fill this table:

Question Command / source Answer
What is the original SHA256? sha256sum firmware.bin
What does file think it is? file firmware.bin
Does binwalk see chunks? binwalk firmware.bin
Does unblob extract more? unblob --report unblob.json firmware.bin
Is entropy high everywhere? binwalk -E firmware.bin or unblob -vvv
Is there a Linux rootfs? find etc/passwd, bin/busybox, www, sbin
Architecture/endian? file bin/busybox, readelf -h
Web/service stack? grep init scripts and web roots
Best first lead? route/service + sink + controlled input
What blocks dynamic proof? kernel, NVRAM, device nodes, missing libs, hardware

If you cannot fill the whole table, that is fine. The blanks become your next questions.

Challenge Index

Use this as the fast "what now?" map.

Symptom Likely meaning Next move
file says data unknown container, encrypted blob, compressed stream, raw flash, signed update run binwalk, unblob report, entropy, xxd, strings, vendor header search
high entropy everywhere encryption, compression, packed image, signature block, random padding search for low-entropy islands, headers, magic, known vendor tools; do not brute-force crypto first
binwalk finds many offsets but extracts little false positives, unsupported format, carved chunks need manual boundaries use unblob, inspect offsets with dd, hexdump, file, compare with known formats
extracted files lack /etc or /bin not Linux rootfs, partial partition, vendor resource bundle, mobile app package find bootloader/kernel/rootfs boundaries, check UBI/JFFS2/SquashFS/cpio, inspect strings
rootfs exists but binaries will not run wrong arch, missing dynamic loader, missing libs, wrong endian file, readelf -h, readelf -l, QEMU -L ROOTFS, copy qemu static only if needed
web root exists but no server config custom init scripts, embedded web server, routes compiled into binary grep init, process names, strings in httpd binaries, xrefs to route strings
service starts but exits missing NVRAM/config/device nodes/env vars trace with strace, stub config files, inspect init scripts, use EMBA/system emulation logs
secrets/config absent from rootfs but boot/init references UBI, JFFS2, /overlay, /mnt/jffs2, or /dev/mtd* persistent state is probably in a raw flash partition identify partition geometry and try read-only MTD/UBI/JFFS2 reconstruction before assuming the secret is absent
UART/RS-232 prompt appears could be log-only, login shell, U-Boot shell, initramfs shell, or root shell classify prompt, auth, UID, boot phase, and writable storage before claiming impact
JTAG/SWD pads or TAP/DAP detected debug may expose memory, flash, or only boundary scan identify adapter/config, target, reset behavior, and lock/fuse state; start with non-writing discovery
SPI/I2C/CAN/USB strings or pads appear protocol surface may be storage, peripheral, debug, control, or update path map bus observation to firmware driver/process and attacker position before prioritizing
OT/industrial protocol or PLC gateway evidence appears safety/process constraints dominate proof choices record asset role, network zone, allowed proof, and lab/simulator state before dynamic testing
EMBA cannot emulate unsupported board/peripheral/kernel/init assumptions use static reachability, QEMU user-mode, or focused binary harness; document blocker
CVE list is huge scanner inventory, not vulnerability proof rank by reachable service + exact version + feature enabled + exploit primitive
command sink exists but no input path weak lead find route, caller, environment variable, config source, or drop it
crash occurs but not reachable local/offline issue prove exposed parser or startup path before claiming device impact
AI finds "vulnerabilities" instantly likely hallucination/overreach require file/function/line/address and a verification command

Firmware Intake

Start with immutable evidence:

CASE=fw_case_01
mkdir -p runs/$CASE/{artifacts,extract,notes,repro,screenshots}
cp /path/to/firmware.bin runs/$CASE/original.bin
sha256sum runs/$CASE/original.bin | tee runs/$CASE/artifacts/sha256.txt
file runs/$CASE/original.bin | tee runs/$CASE/artifacts/file.txt
ls -lh runs/$CASE/original.bin | tee runs/$CASE/artifacts/size.txt

Record scope:

# Scope

- Source of firmware:
- Authorization:
- Device/model/version:
- Firmware filename:
- SHA256:
- Time budget:
- Allowed actions:
- Disallowed actions:
- Cloud/AI upload allowed:

Do not edit the original firmware. Every transformation gets its own file and note.

Acquisition ladder

The local IoTFirmwareAnalysisGuide is a beginner tutorial. Its firmware-acquisition section adds one practical reminder: search beyond the obvious vendor download page.

Use this ladder before deciding that a firmware image is unavailable:

Source What it can give you Evidence to record Stop when
vendor support page official latest firmware, release notes, manuals URL, model, region, version, checksum if available exact target version is present
alternate regional support pages older or differently packaged firmware country/region, model suffix, version differences package lineage is unclear or not the same hardware
vendor FTP/archive paths older releases and transition packages directory listing, timestamps, naming pattern source authenticity cannot be established
device update traffic module IDs, product IDs, signed URLs, component packages lab capture, request parameters, auth/cert-pinning constraints real vendor/cloud testing is out of scope
vendor GPL/source drops build config, decryptor/updater source, kernel patches source URL, version mapping, license bundle name bundle cannot be tied to the target image
customer support or device portal non-public firmware tied to serial/model request path and authorization basis access requires misrepresentation or violates scope
read-only hardware extraction full flash state, factory partitions, older plaintext image board revision, chip ID, dump hashes, method hardware access is not authorized or risk is too high

Treat acquisition as evidence, not housekeeping. A firmware from a different region or hardware revision can support diffing or decryptor recovery, but it is not automatically the target firmware.

Extraction And File Format Problems

Normal Extraction Order

Use multiple extractors because they fail differently:

binwalk runs/$CASE/original.bin | tee runs/$CASE/artifacts/binwalk.txt
binwalk -Me runs/$CASE/original.bin -C runs/$CASE/extract/binwalk

If available:

unblob --report runs/$CASE/artifacts/unblob_report.json \
  runs/$CASE/original.bin \
  -e runs/$CASE/extract/unblob

Why both:

  • Binwalk v3 is fast and directly firmware-oriented.
  • unblob is strong at safe recursive extraction, chunk metadata, entropy/randomness, and multi-file formats.
  • EMBA wraps multiple extraction paths and adds reporting/context.

Manual Boundary Recovery

If tools identify offsets but do not extract:

OFFSET=123456
dd if=runs/$CASE/original.bin of=runs/$CASE/extract/chunk_$OFFSET.bin bs=1 skip=$OFFSET status=progress
file runs/$CASE/extract/chunk_$OFFSET.bin
binwalk runs/$CASE/extract/chunk_$OFFSET.bin

Inspect headers:

xxd -g 1 -l 256 runs/$CASE/original.bin
strings -a -n 6 runs/$CASE/original.bin | head -200

Search for common markers:

rg -a -n "hsqs|sqsh|UBI#|UBI!|CrAMFS|uImage|FIT|TRX|HDR0|OpenWrt|Linux version|BusyBox|rootfs" runs/$CASE/original.bin

Filesystem-Specific Handling

Format Sign First move
SquashFS hsqs / sqsh unsquashfs -d out image.squashfs
CPIO/initramfs 070701 / gzip/lzma wrapped decompress, then cpio -idmv
UBI/UBIFS UBI# / UBI! EMBA/unblob first, then ubi tools if needed
JFFS2 magic / NAND-ish layout EMBA/binwalk/unblob; beware endian and cleanmarkers
ext2/3/4 image mountable partition mount read-only loopback
cramfs cramfs magic fsck.cramfs, mount -o loop,ro
tar/zip/tgz archive extract normally, then recurse

Read-only mount rule:

sudo mount -o loop,ro image.ext4 /mnt/fw

Never mount unknown firmware read-write.

When Extraction Fails

Extraction failure is not one thing. Classify it.

Case A: Encrypted Or Compressed?

High entropy can mean encryption or compression. Distinguish with structure:

Observation More likely
high entropy but recognizable header/footer compressed container
high entropy after a clear signed header encrypted payload
high entropy only in regions mixed image with compressed/encrypted chunks
no strings except header/version encrypted/signed update
repeated blocks / padding flash image, alignment, or weak encoding

Do this:

binwalk -E firmware.bin | tee entropy.txt
strings -a -n 8 firmware.bin | tee strings.txt
xxd -g 1 -l 512 firmware.bin

Look for:

  • vendor magic
  • model/version names
  • update scripts
  • public certificate names
  • compression names
  • "AES", "RSA", "SIGN", "FWUP", "U-Boot", "FIT"

Case B: Signed But Not Encrypted

Signed firmware often still extracts. The signature prevents installing modified firmware, not reading it.

How to reason:

Can I see strings and filesystem chunks?
  yes -> probably signed/plain or compressed/signed
  no  -> maybe encrypted or custom packed

For a vulnerability report, insecure update logic is only a finding if you show a bypass, missing validation, downgrade issue, weak hash, or trust-chain flaw. "It has a signature" is not a bug.

Case C: Encrypted Firmware

If encrypted:

  1. Check whether update package includes key material or scripts.
  2. Search vendor GPL/source packages.
  3. Search mobile app resources for decryption keys only if scope allows.
  4. Search bootloader/rootfs from older firmware versions.
  5. Look for same device family images with unencrypted updates.
  6. Consider hardware extraction only if authorized and in scope.

The IoTFirmwareAnalysisGuide adds a transition-firmware heuristic: search for older or "middle" releases before hardware extraction. Some vendors ship an intermediate plaintext update that contains the decryptor for later encrypted images, or release notes say an older version must be installed before a newer one. That does not prove a decryptor exists, but it is a cheap search before hardware work. Firmware decryption tutorial

Decision rule:

latest image encrypted
  -> search older versions, regional mirrors, FTP archives, and release notes
  -> identify last extractable or transition image
  -> search it for updater/decryptor binaries, key material, and accepted suffixes
  -> run or reverse only the decryptor path needed for the target image

Abandon this branch when no version lineage can be established, the transition image targets different hardware, the decryptor rejects the target format, or the key is hardware-backed and absent from the image.

Do not spend the engagement attacking modern crypto without evidence of an implementation flaw. Report the blocker:

The update payload appears encrypted: high entropy across the payload, no recognizable filesystem markers, and only a small cleartext header. I could analyze the header/update metadata, but not the root filesystem without keys, an older plaintext firmware, or hardware extraction.

Case D: Bare-Metal Firmware

Signs:

  • no Linux strings
  • no /etc, /bin, /sbin
  • vector table near start
  • MCU architecture
  • flat code/data
  • no filesystem

Move to:

  • Ghidra/IDA/Binary Ninja
  • vendor SDK/MCU identification
  • memory map reconstruction
  • peripheral register naming
  • UART/SWD/JTAG if authorized

Do not force Linux tooling onto bare-metal firmware.

RootFS And Architecture Recovery

After extraction, find candidate root filesystems:

find runs/$CASE/extract -path "*/etc/passwd" -o -path "*/bin/busybox" -o -path "*/www/*" | tee runs/$CASE/artifacts/rootfs_markers.txt
find runs/$CASE/extract -maxdepth 5 -type d \( -name etc -o -name bin -o -name sbin -o -name www -o -name htdocs \)

Pick the rootfs by evidence:

best rootfs = has /etc + /bin or /sbin + web/service files + architecture binaries

Architecture:

ROOTFS=/path/to/rootfs
find "$ROOTFS" -type f -perm /111 -exec file {} \; | tee runs/$CASE/artifacts/executables_file.txt
rg -n "ELF.*MIPS|ELF.*ARM|ELF.*AArch64|ELF.*Intel|ELF.*PowerPC" runs/$CASE/artifacts/executables_file.txt
readelf -h "$ROOTFS/bin/busybox" 2>/dev/null | tee runs/$CASE/artifacts/busybox_readelf.txt

Dynamic loader:

readelf -l "$ROOTFS/bin/busybox" 2>/dev/null | rg "interpreter|Requesting"
find "$ROOTFS" -path "*/ld-*so*" -o -path "*/libc.so*"

This tells you which QEMU user-mode binary and -L rootfs prefix to use.

Attack Surface Mapping

You are looking for attacker-controlled input paths.

Startup And Services

find "$ROOTFS/etc" -maxdepth 5 -type f 2>/dev/null | rg "init|rc|service|inetd|xinetd|systemd|cron"
rg -n "httpd|uhttpd|lighttpd|boa|nginx|dropbear|telnetd|inetd|dnsmasq|upnp|miniupnp|ftpd|smbd|mqtt|coap" "$ROOTFS" 2>/dev/null | tee runs/$CASE/artifacts/services.txt

Ask:

  • What starts automatically?
  • What listens on the network?
  • What runs as root?
  • What reads config/NVRAM?
  • What parses unauthenticated input?

High-ROI research shortcuts:

  • Default-started service beats obscure binary. Synacktiv's Netgear R6700v3 target was a daemon started in the default router configuration.
  • Feature-gated reachability still counts if you document the gate. NCC's BrokenPrint finding required ReadySHARE with a connected USB printer.
  • Runtime inventory can shrink a huge rootfs. Neodyme's Netgear RAX30 writeup used process/network state before deep reversing.

Web Roots And Routes

find "$ROOTFS" -type d | rg "/www$|/htdocs$|/web$|/cgi-bin$|/usr/www|/www/cgi|/www/luci"
find "$ROOTFS" -type f \( -name "*.cgi" -o -name "*.sh" -o -name "*.php" -o -name "*.lua" -o -name "*.asp" -o -name "*.js" \) | tee runs/$CASE/artifacts/web_files.txt
rg -n "GET|POST|QUERY_STRING|REQUEST_METHOD|CONTENT_LENGTH|HTTP_|cgi-bin|form action|XMLHttpRequest|fetch\\(|ajax|token|session" "$ROOTFS" 2>/dev/null | tee runs/$CASE/artifacts/web_routes.txt

For compiled web servers:

strings -a "$ROOTFS/path/to/httpd" | rg "/|cgi|admin|login|token|password|upgrade|upload|download|apply|system|reboot"

Dangerous Sinks

rg -n "system\\(|popen\\(|exec|eval|shell_exec|passthru|`.*`|\\$\\(|/bin/sh|sh -c|wget|curl|tftp|nc " "$ROOTFS" 2>/dev/null | tee runs/$CASE/artifacts/command_sinks.txt
rg -n "\\.\\./|realpath|fopen|open\\(|readfile|download|upload|tar |unzip|cpio|firmware|upgrade|update" "$ROOTFS" 2>/dev/null | tee runs/$CASE/artifacts/file_update_sinks.txt
rg -n "strcpy|strcat|sprintf|gets\\(|scanf\\(|memcpy|strncpy|snprintf|malloc|free|recv|read\\(" "$ROOTFS" 2>/dev/null | tee runs/$CASE/artifacts/native_sinks.txt

Sink-only evidence is not enough. You need an input path.

Use the selected research as pattern matchers:

If you see Ask next Example
CGI/web handler plus JSON/XML/custom parser Which route reaches it, and is auth required? Claroty Synology BC500
update client, cloud agent, or downloader Who controls the source, what validates the artifact, where is it written? Cisco Talos WAGO and Synacktiv Netgear
P2P camera SDK or vendor cloud protocol Is this branded-device code or reused supply-chain code? Nozomi ThroughTek
old kernel/library/component Is the vulnerable feature present and reachable in this firmware build? Forescout OT/IoT router n-day triage

Finding Class Playbooks

1. Command Injection

Best target:

web/cgi/api parameter -> shell command string -> system/popen/sh

Evidence checklist:

  • route or script name
  • parameter name
  • code line/function building command
  • missing quoting/allowlist
  • safe trigger

Static path:

rg -n "system\\(|popen\\(|eval|exec|/bin/sh|sh -c|`.*`|\\$\\(" "$ROOTFS/www" "$ROOTFS/usr" "$ROOTFS/etc" 2>/dev/null

Safe proof:

Use: ;id
Use: ;echo FW_TEST
Use: ;sleep 5
Avoid: network callbacks, destructive file writes, persistence

Report only if you can connect input to sink.

2. Path Traversal / Arbitrary File Read

Best target:

download/view/log/backup endpoint -> filename parameter -> open/read without canonicalization

Search:

rg -n "download|readfile|fopen|open\\(|cat |backup|log|config|filename|path|\\.\\./|realpath|basename" "$ROOTFS" 2>/dev/null

Proof:

curl -i "http://host/path?file=../../etc/passwd"

Root cause:

  • user-controlled path
  • no canonicalization
  • no base-directory enforcement
  • symlink handling if relevant

3. Authentication / Session Logic

Best targets:

  • endpoints with missing auth check
  • password comparison bugs
  • predictable session tokens
  • client-side-only auth
  • hardcoded fallback accounts

Search:

rg -n "login|logout|session|token|auth|password|passwd|admin|privilege|cookie|Set-Cookie|Authorization" "$ROOTFS" 2>/dev/null

Proof standard:

Unauthenticated request reaches protected action
or low-privilege session reaches admin action
or token can be predicted/reused

Avoid claiming auth bypass from UI JavaScript alone. Prove server-side behavior.

4. Hardcoded Credentials / Keys

A secret is a lead, not automatically a vulnerability.

Search:

rg -n "password|passwd|pwd|secret|token|apikey|api_key|BEGIN .*PRIVATE KEY|authorized_keys|root:|admin:" "$ROOTFS" 2>/dev/null
find "$ROOTFS" -type f \( -name "*key*" -o -name "*.pem" -o -name "*.crt" -o -name "*.conf" \)

Proof standard:

  • credential is used by reachable service
  • account exists
  • login or trust path works
  • privilege/impact explained

If you cannot test:

The firmware contains a hardcoded credential/key. I found evidence that <service/config> consumes it, but I could not validate login without hardware/emulation. Impact is likely <...>; confidence is <...>.

5. Insecure Firmware Update

High-value in IoT, but easy to overclaim.

Search:

rg -n "upgrade|update|firmware|fw|image|signature|verify|rsa|openssl|md5|sha1|sha256|tar|unzip|dd if|mtd|sysupgrade" "$ROOTFS" 2>/dev/null

Look for:

  • unsigned update accepted
  • hash without signature
  • signature verification result ignored
  • downgrade accepted
  • path traversal during archive extraction
  • command injection in update metadata
  • update fetched over HTTP without signature validation

Proof standard:

modified update package accepted
or verifier can be bypassed
or code path ignores failed verification
or update archive writes outside target directory

If you only see "MD5", do not call it a vuln until you show it gates trust.

6. Native Binary Memory Corruption

Best target:

network service or parser -> controllable input -> crash/control -> root cause

Triage:

file "$BIN"
checksec --file="$BIN" 2>/dev/null || true
strings -a "$BIN" | rg "http|cgi|recv|read|strcpy|sprintf|password|upgrade|xml|soap|upnp"
readelf -s "$BIN" 2>/dev/null | rg "strcpy|sprintf|system|recv|read|memcpy"

Dynamic:

QEMU_STRACE=1 qemu-mipsel-static -L "$ROOTFS" "$BIN" args 2>&1 | tee qemu_strace.txt
qemu-mipsel-static -g 1234 -L "$ROOTFS" "$BIN" args
gdb-multiarch "$BIN"

Proof levels:

Level Meaning
crash only local fault without reachability
controlled input reaches crash medium
root-cause buffer/heap issue identified strong
PC/control-flow influence or write primitive stronger
reliable exploit not required for most firmware assessments, but excellent if scoped

7. NVRAM / UCI / Config Injection

Routers often route web input through config systems.

Search:

rg -n "nvram|getenv|setenv|uci|get_config|config_get|config_set|mib_get|mib_set" "$ROOTFS" 2>/dev/null

Reasoning path:

web/API parameter -> config store -> init/service script -> shell command or unsafe parser

This class is sneaky because the vulnerable sink may execute later, not in the request handler.

8. UPnP / SOAP / XML / JSON Parsers

Search:

rg -n "upnp|miniupnp|soap|xml|json|Content-Type|M-SEARCH|SSDP|TR-064|HNAP" "$ROOTFS" 2>/dev/null

Prioritize unauthenticated LAN-facing parsers. For exploitability, show:

  • service starts
  • port/route exists
  • parser accepts controlled field
  • dangerous sink/crash/root cause

9. Mobile App / Cloud Coupling

Sometimes firmware alone is incomplete.

Look for:

  • API hostnames
  • MQTT topics
  • device IDs
  • certificate pins
  • pairing flows
  • local discovery protocols
  • mobile app route names that match firmware endpoints

Search:

rg -n "https?://|mqtt|coap|websocket|api|cloud|token|pair|provision|serial|deviceid|uuid" "$ROOTFS" 2>/dev/null

Do not test real vendor cloud endpoints unless explicitly authorized. You can still document trust assumptions and local attack paths.

10. Ecosystem Adjacency Pass

Use this as a time-boxed pass when firmware strings point outside the rootfs. Keep the claim tied to firmware evidence even when product context points toward mobile apps, radios, cloud services, or field buses.

rg -n "mqtt|coap|rtsp|onvif|ssdp|mdns|dns-sd|ws-discovery|ble|bluetooth|zigbee|zwave|z-wave|thread|lorawan|rfid|nfc|android|ios|apk|ota|pair|provision|gatt|uuid|fcc|uart|jtag|swd|spi|i2c" "$ROOTFS" 2>/dev/null
If firmware hints at Ask next Keep it useful by
MQTT/CoAP topics, brokers, or device IDs Is auth, topic ACL, or provisioning enforced locally? mapping topic/credential source before touching cloud
RTSP/ONVIF/WS-Discovery/mDNS/SSDP Is discovery or media control unauthenticated on LAN? proving only local lab reachability
BLE/ZigBee/Z-Wave/Thread/LoRaWAN/RFID/NFC Does firmware contain pairing, keys, opcodes, or update paths? recording radio as a hardware-scoped validation path
Android/iOS package names, app routes, or cert pins Does the companion app expose firmware routes, keys, or pairing assumptions? inspecting app artifacts only when in scope
UART/JTAG/SWD/SPI/I2C/FCC/chip markings Does hardware access answer a specific blocker? writing the exact question before opening hardware work

Emulation Ladder

Use the smallest runtime model that answers your question.

Tier Tool Use when Stop when
0 static only route/source/sink proof is enough dynamic proof is not necessary or too costly
1 QEMU user-mode one binary/script needs runtime behavior missing kernel/peripheral blocks answer
2 chroot + QEMU static simple rootfs command/service needs libraries/config init/peripheral/network assumptions dominate
3 custom analysis OS / Buildroot guest old libc/kernel/debug/storage compatibility matters more than exact board fidelity setup no longer answers a named binary/service/update question
4 MTD/UBI/JFFS2 reconstruction raw flash-backed config, env, factory, caldata, or overlay state matters recovered storage does not feed the lead or geometry is unsupported
5 EMBA system emulation need reachable services/ports with automated setup EMBA exposes service or explains failure
6 manual QEMU system you know kernel/rootfs/machine model path board/peripheral work becomes too deep
7 hardware runtime requires real peripherals/secure elements/radio scope/time/hardware limits

Research-derived choice rules:

  • If the question is one function or one generated value, use targeted emulation. SpaceRaccoon's Nokia Beacon 1 used Qiling for a password-generation routine.
  • If the question is one network-facing web service and the route/auth/sink are already mapped, build a focused service harness before full-system emulation. Claroty's Planet switch work used QEMU/chroot and QEMU GDB support around Boa and dispatcher.cgi because that component carried the pre-auth attack path.
  • If old libc, kernel, Buildroot, MTD tooling, or debug-tool compatibility is the blocker, build a compatible analysis OS and import the vendor rootfs or firmware into it. The local emulation folder and ARMX are examples of this analysis-guest pattern.
  • If the question is a persistent secret, account, or config value and the rootfs points into raw flash storage, reconstruct that storage instead of forcing full-system emulation. The local Gemtek case uses nandsim, nandwrite, ubiattach -O 2048 -m 8 -d 8, and UBIFS mounting because /mnt/jffs2 is the runtime writable store, not because every password question needs NAND simulation.
  • If a service does not start, compare your manual image to an automated framework result. Applied Security's QEMU firmware emulation writeup shows this debugging move.
  • If low-level hardware blocks emulation, pivot to static proof and document the blocker. Neodyme's HP M479fdw is the reference pattern.

Raw NAND, UBI, And Secrets

Ask this before spending time on nandsim, mtdram, or ubiattach:

Is the value I need stored in the extracted rootfs, generated by one binary, or
kept in a raw persistent partition?

Use MTD simulation when the third answer has evidence. Good triggers:

  • boot logs show page/erase geometry, partition names, UBI attach, or JFFS2/UBIFS mount lines;
  • init scripts mount /mnt/jffs2, /overlay, /data, or another writable store from /dev/mtd*;
  • /etc/passwd, /etc/shadow, service config, SSH files, or random seeds are symlinked or copied into persistent storage;
  • binaries call fw_printenv, nvram, MTD ioctls, or named factory/config partitions;
  • the image source is a full flash dump, not just a vendor update bundle.

The Gemtek example meets those triggers: the README documents a full EEPROM dump, partition splitting into Config/Env/Kernel/Storage images, NAND geometry (2048 byte pages, 64 byte OOB, 131072 byte eraseblocks), boot output that attaches mtd8 to ubi8, and first-boot/runtime scripts that use /mnt/jffs2. The accompanying script then recreates the MTD partition table, writes the split images, attaches UBI with VID offset 2048, and mounts /dev/ubi8_0 on /mnt/jffs2. README, set-nandsim.sh

Do not use this branch for a rootfs-only question. In the same Gemtek material, the default WiFi password path is documented as /bin/assistant logic using serial and MAC-like inputs; that is a targeted reverse-engineering or QEMU-user/helper question, not automatically an MTD question.

Other examples refine the branch:

Example What changed the decision Research lesson
Avaya 9608G raw flash contained multiple JFFS2 partitions; mounting required the partition table, nandsim, nandwrite, mtdblock, and later OOB/ECC analysis if write-back or faithful mount matters, OOB/ECC and bad-block evidence can dominate the experiment
Somfy Connexoon SAM-BA recovered a UBI volume, nandsim/ubiattach -O 2048 exposed root/rootB volumes, and ubinize rebuilt the volume UBI volume metadata and static/dynamic volume sizing are part of the evidence
NB6VAC firmware is WFI-wrapped, rootfs is UBIFS after a JFFS2 prefix, nandsim uses ID ef f1 00 95 and partitions 1,259,259,496,8, and the rebuild preserves image sequence/tag metadata a writable rootfs does not mean the kernel, bootloader, or MCU root of trust was bypassed
Foscam P20 in local EMBA P20_foscam_decryptor.sh decrypts an OpenSSL-wrapped image, extracts app_ubifs, formats a selected MTD device with ubiformat -O 2048, attaches UBI, and copies mounted files automated extraction is only interpretable with its VID offset, selected MTD device, and module-state assumptions
HiBy R2 boot logs showed NAND reads, but the modification strategy pivoted runtime writes to SD-backed rootfs hardware storage evidence can tell you to avoid NAND writes instead of simulating NAND
Youngrok nandsim/UBIFS notes nandsim can model bad blocks/weak pages, while UBI bad-block reserve and LEB accounting can change mount success mount failure may be geometry or reserve math, not absence of data
eMMC notes eMMC exposes user, boot, and RPMB-like areas differently from raw NAND; boot partitions may be read-only by default do not force MTD mental models onto /dev/mmcblk* storage

Evidence standard:

raw partition image and hash
partition offset/size source
page size, OOB size, eraseblock size, and UBI VID/header offset
attach/mount command and output
files or values recovered
script/binary that consumes those files or values
false-positive risks: guessed geometry, missing OOB/ECC, bad blocks, synthetic nodes

Stop when the partition image is unavailable, the recovered filesystem does not feed the lead, or the missing value is per-device state that was never captured. At that point the next honest branch is a scoped hardware read, not more simulation.

Compatible Analysis OS Pattern

Use this when the firmware binaries need an old or matching userspace, but the finding does not require exact board boot.

Local triggers:

  • emulation/myrootfs: old ARM/XScale binaries needed old glibc/toolchain behavior and a QEMU analysis rootfs.
  • emulation/buildroot-armv7: DVA router binaries needed ARMv7, Linux 3.4.11-rt19, uClibc 0.9.33.2, libgcrypt, debug tools, and imported /dva-root//dva-firm artifacts.
  • emulation/hht: HHT/Gemtek-style MIPS work needed old Buildroot/uClibc, QEMU Malta, and nandsim support.
  • NB6VAC emulation: MIPS/Buildroot/UBIFS work was an early partial emulation path, not proof of a secure-boot bypass.
  • ARMX: per-device config, NVRAM preload, rootfs archives, optional flash memory, NFS-shared /armx, port forwarding, and debug helpers formalize the same idea for ARM/Linux targets.

Evidence standard:

why user-mode/chroot was insufficient
guest CPU, machine, kernel, libc, and toolchain versions
vendor rootfs/firmware import paths
debug tools and compiler/debug settings added
NVRAM, flash, preload, /proc, /sys, and /dev assumptions
original startup evidence versus guest launch command

Stop when the branch becomes general toolchain archaeology. Continue only while the analysis guest answers a named binary, service, update-chain, or storage question better than static analysis or QEMU-user.

EMBA Result Interpretation

Before treating EMBA output as evidence, identify which local module produced it and what conditions EMBA changed to produce it. Use emba-source-capability-map.md for the source walkthrough.

EMBA result Treat as Before escalating
S115 version/help output one-binary behavior in a copied/repaired test root record binary hash, QEMU CPU, arguments, missing files, copied files, stdout/stderr
S130 dependency edge dependency lead tagged by strings, object metadata, QEMU-user trace, or L10 trace check the edge marker and underlying log before calling it a dependency
L10 Booted kernel/userspace startup evidence inspect init candidate, init=/rdinit=, serial log, and filesystem repairs
L10 ICMP ok emulated host network reachability do not infer a service or vulnerability
L10 TCP ok or open port live-surface lead map port to PID, command line, and launch source; exclude EMBA debug listeners
F17/F50 CVE/exploit count prioritization metadata verify component version, feature/app/config, reachability, root cause, and safe impact
S26 or S118 "verified" CVE stronger presence evidence still prove attacker reachability and vulnerable feature use

If EMBA modified a startup script, created a device node, supplied NVRAM values, or launched a service with inferred arguments, report the result as "reachable in the repaired EMBA image" until original-image behavior is separately shown.

User-Mode QEMU Pattern

file "$ROOTFS/bin/busybox"
readelf -h "$ROOTFS/bin/busybox"
qemu-mipsel-static -L "$ROOTFS" "$ROOTFS/bin/busybox"
QEMU_STRACE=1 qemu-mipsel-static -L "$ROOTFS" "$ROOTFS/path/to/binary" args

Chroot Pattern

sudo cp /usr/bin/qemu-mipsel-static "$ROOTFS/usr/bin/"
sudo mount -t proc proc "$ROOTFS/proc"
sudo mount --bind /dev "$ROOTFS/dev"
sudo chroot "$ROOTFS" /usr/bin/qemu-mipsel-static /bin/sh

Clean up:

sudo umount "$ROOTFS/proc" "$ROOTFS/dev"

EMBA System Emulation Pattern

Use EMBA after static triage:

sudo ./emba -l "$LOG/06_emulation" -f "$FW" -p ./scan-profiles/default-scan-emulation.emba -P 1 -T 1

Before this:

VMware network: host-only
NAT/bridged: disconnected unless explicitly needed
snapshot: taken

After this:

rg -n "open port|Nmap scan report|ONLINE|IP address|http|login|qemu|archive|emulation" "$LOG/06_emulation"

If emulation fails, record the blocker and choose the next smallest static or hardware-backed test.

Hardware-Only And Peripheral Blockers

Some firmware cannot be meaningfully exercised without hardware.

Before hardware work, do a non-destructive precheck:

  • Confirm authorization, destructive-test limits, and whether multiple devices exist.
  • Record model, board revision, FCC ID if available, visible chip markings, and connector/header photos.
  • Identify likely UART, SPI flash, JTAG, or SWD targets, but do not connect until voltage and pinout risk are understood.
  • Prefer the least invasive question: boot log, console access, flash dump, bootloader environment, or one peripheral signal.
  • Stop if the hardware path no longer answers the primary firmware finding.

Common blockers:

  • secure element or TPM
  • Wi-Fi/Bluetooth/radio hardware
  • camera/sensor pipeline
  • GPIO reset/config pins
  • NVRAM partition not in update image
  • proprietary kernel drivers
  • watchdogs
  • board-specific init

What to do:

  1. Identify the missing dependency precisely.
  2. Stub only if it answers a narrow question.
  3. Prefer static proof if dynamic proof is too costly.
  4. Report the limitation cleanly.

Hardware paths, only if authorized:

  • UART console
  • SPI flash dump
  • JTAG/SWD
  • bootloader environment
  • recovery mode
  • serial logs

Evidence sentence:

Dynamic reproduction was blocked by <specific hardware/peripheral>. Static evidence shows <route/path/sink>. The next hardware validation step is <UART/SPI/JTAG/device test>.

Level 2 Hardware And Boot Access

Use this section when the research question moves from filesystem and service analysis into root of trust, boot shells, terminal access, debug interfaces, or field buses. The safe default is observation first, writes later only when scope and recovery are explicit.

Interface triage

Lead First classification Evidence to save Stop or downgrade when
UART / RS-232 log-only, login, root shell, initramfs, U-Boot, vendor monitor voltage, baud, boot log, prompt, auth state, UID, mount table, boot phase prompt is unavailable, auth blocks scope, or shell is ephemeral and cannot affect the finding
SPI flash firmware, bootloader env, config, factory data, calibration, logs chip ID, board photo, voltage, two matching dump hashes, partition map, strings/filesystem hits read is unstable, chip role is unknown, or recovered data has no consumer
JTAG / SWD boundary scan, CPU debug, flash access, locked debug adapter, OpenOCD/interface config, target/TAP/DAP, halt/read result, lock/fuse state only scan-chain evidence exists or memory/debug is locked
I2C EEPROM, PMIC, sensor, secure element, display/touch, GPIO expander address, register trace, voltage, driver strings, device-tree node, consumer binary observed bytes cannot be mapped to firmware behavior
CAN bus automotive/industrial field bus, diagnostic path, ECU/PLC messaging bitrate, interface, arbitration IDs, firmware handler, lab state, replay constraints no handler or safety-bounded lab state exists
USB host parser, device/gadget parser, update media, debug serial, storage descriptor, mode, driver, endpoint, mount/update script, auth/feature gate descriptor exists but no reachable parser or write path is shown

Boot-shell classification

Do not collapse every prompt into "root." Classify it:

Prompt type What it can prove What it cannot prove alone
U-Boot/vendor bootloader shell boot environment, memory load paths, flash commands, recovery policy Linux root, secure-boot bypass, or persistence
UEFI shell boot variables, filesystem visibility, capsule/update paths, Secure Boot state checks root-of-trust failure unless policy can be altered or bypassed
initramfs/failsafe shell early userspace, mount/recovery behavior, emergency auth normal runtime reachability or persistent compromise
Linux root shell local administrative access in that boot state remote exploitability, production default, or survival across reboot
service/debug shell command surface and privilege for that service device-wide root unless UID, namespace, and filesystem access support it

Evidence line:

Observed <prompt type> via <interface> at <boot phase>; auth=<state>; uid=<id>;
writable storage=<paths>; persistence=<tested/not tested>; normal boot relation=<...>.

Root-of-trust workflow

For secure boot and update-chain claims, build an edge map:

ROM/fuse/OTP
  -> first mutable loader
  -> bootloader environment
  -> kernel / DTB / initramfs
  -> rootfs / overlay
  -> update or recovery writer

For each edge, record:

artifact:
who selects it:
who verifies it:
where the key/hash/rollback state lives:
what happens on failure:
whether a debug or recovery path skips the same rule:

Claim thresholds:

Claim Evidence required
root of trust exists immutable or protected trust anchor, verified artifact list, failure behavior
root of trust is bypassed modified/unsigned artifact executes or verified result is ignored
UART shell weakens trust shell can alter boot policy, bootargs, env, or persistent state used by verified boot
SPI/env weakness matters attacker-writable storage controls a verified boot decision or post-boot trust decision
update-chain weakness matters untrusted package affects executable content, rollback state, or trust metadata

NB6VAC is the cautionary pattern. The local notes show rootfs modification and root access by rebuilding a WFI-tagged firmware whose UBIFS rootfs can be mounted and repacked. They also state that the kernel is authenticated and that an OTP Silicon Labs C8051T634 MCU controls secure boot/integrity. The MCU notes record C2 read limitations, read/write lock bits, a security byte at 0x1fff, and the conclusion that changing the OS likely requires replacing or reprogramming the MCU. Treat those as two different claims:

rootfs modification accepted -> update/rootfs integrity gap
open firmware blocked by MCU -> root-of-trust enforcement blocker

Do not collapse them into "secure boot bypassed" unless an unsigned kernel, bootloader, or alternate OS actually executes under the stated boot policy.

OT and industrial automation

OT firmware research needs a narrower proof standard. A finding should not imply plant impact from firmware evidence alone.

Before dynamic testing, write:

asset role:
process controlled:
safety relevance:
network zone:
lab/simulator state:
allowed proof:
explicitly disallowed actions:
rollback/recovery plan:

Prioritize non-destructive proof:

  • version/config/reachability evidence from firmware and lab inventory;
  • parser/root-cause proof on captured samples or a simulator;
  • read-only bus observation;
  • vendor protocol state-machine mapping;
  • controlled test bench behavior, not live process actuation.

Stop when proof would require unsafe physical motion, live process control, unapproved bus injection, or changes that cannot be rolled back.

Debugging, Instrumentation, Tracing, And Hooking

Choose the debugging layer by the missing evidence edge. A tool is useful when it can answer one of these questions:

which process handled the input?
which file, NVRAM key, device node, bus frame, or library call changed control flow?
which instruction or source line implements the trust/security decision?
does the same behavior survive on the shipped binary, boot order, or hardware?

Linux firmware and emulated Linux userland

Case First tools What it can prove Evidence that changes the branch False-positive risk / stop rule
One stripped Linux binary strace, QEMU-user -strace, gdb-multiarch, gdbserver, matching sysroot syscalls, signals, crash PC/registers, file/env assumptions, dynamic loader failures syscall trace reaches the suspicious file/socket/ioctl or crashes at the target parser stop when the missing behavior is init order, kernel driver state, NVRAM, or hardware
Dynamic-library or config-dependent code QEMU-user with -L, chroot, copied /etc, LD_PRELOAD wrappers, ltrace where available libc/vendor-library calls, arguments to crypto/config/NVRAM wrappers, missing shared objects a wrapper shows attacker-controlled bytes crossing into the sink static, setuid, incompatible loader, or changed library behavior invalidates the claim
Known service route service harness, strace, tcpdump, gdbserver, HTTP tooling route to process to sink under controlled startup request reaches the same binary/function named by init/config evidence abandon if stubs create a launch path that original init never takes
Full-system emulation QEMU gdbstub (-S -s), serial console, tcpdump, EMBA L10 logs, in-guest strace/gdbserver repaired-image boot order, process list, ports, crash/debug evidence inside the VM port/PID maps to original init and a request reaches the suspected component do not claim board fidelity when EMBA or local scripts inject helpers, links, NVRAM, init entries, or debug listeners
Rebuilt application or libraries unstripped sidecar symbols, same compiler/config where possible, sanitizers or Valgrind when the ABI supports them source-level stack, heap misuse, exact library call sequence, developer-intent clues the same input reproduces on the shipped stripped binary or a byte-equivalent optimized build debug builds can change layout, timing, heap behavior, and exploitability
Rebuilt kernel or matching kernel symbols vmlinux, System.map, QEMU gdbstub, KGDB/KDB, ftrace, tracepoints, kprobes, perf/eBPF syscall, driver, ioctl, module, filesystem, and scheduler evidence trace shows the kernel path or driver state needed by the finding stop if the question is userspace-only or if the kernel/device model no longer matches the target

Published basis: strace records system calls and signals; QEMU's gdbstub lets GDB inspect VM state and use breakpoints/watchpoints under system emulation; GDB remote debugging supports small or kernel-like targets when the host has matching symbols and libraries; Linux tracing exposes ftrace, tracepoints, kprobes, perf events, and related mechanisms. The LD_PRELOAD, wrapper, and service-harness rules above are synthesized researcher heuristics: they are evidence-gathering techniques, not proof of production reachability by themselves.

Practitioner evidence:

Scenario Online research example Tools actually used Decision lesson
Router/industrial-switch web RCE Claroty Team82, Hack The Emulated Planet Binwalk, QEMU user-mode/chroot, Boa/CGI service execution, QEMU GDB Pick the exposed management service first when firmware, web config, pre-auth route, and native CGI sink align. QEMU proves the route/process/crash in the emulated environment; original exposure still needs init/config mapping.
General router emulation choice ZDI MindShaRE, How to "Just Emulate It With QEMU" file, readelf, QEMU-user -L, cross-arch chroot, full QEMU VM, Firmadyne, ARM-X Decide the end goal first: one helper binary, one service, or fuller init/network context. Hardware may be cheaper when exploit reliability depends on real caches, timing, or device state.
Function harness before fuzzing TSMR, Blackbox-Fuzzing of IoT Devices Using the Router TL-WR902AC chroot, QEMU -strace, LD_PRELOAD, cross-compiled MIPS hook library, QEMU -g, gdb-multiarch Use LD_PRELOAD and GDB to build a narrow harness around a named parser. Treat the harness as synthetic until the function call order and initialization map back to the shipped service.
Router JTAG debug and flash evidence River Loop Security, Communicating with JTAG via OpenOCD and ZDI, Belkin Surf N300 hardware reversing JTAG pinout validation, target voltage, OpenOCD, GDB/telnet ports, memory read/write, flash dump, boot-env string analysis JTAG is a lead until voltage, TAP/target config, halt/read behavior, and memory/flash evidence show what access exists on the production board.
Companion-app protocol extraction WithSecure Labs, Digital lockpicking and Liu et al., manual IoT protocol reverse engineering Frida/Xposed hooks, Android dynamic debugging, Ghidra/IDA, packet capture, firmware-side gdbserver in the paper's examples Frida can expose pre-encryption app messages and protocol state. Firmware impact needs a corresponding device handler, accepted message, or root-cause trace.
AP-to-modem/CP boundary nns.ee, Quectel EG25-G AT-command RCE and RG500Q-EA AT-command RCE firmware/update-package extraction, Ghidra, AT commands over serial, modem-side daemon/library analysis The AP sends commands to a separate modem OS. Impact is proven on the modem/CP side only when the modem-side parser accepts the command and the resulting action runs in that domain.
RTOS/VxWorks target Qasem et al., OctopusTaint, plus practitioner ARRIS QB5000 VxWorks UART/string tracing architecture/endianness recovery, load-address recovery, VxWorks symbol-table recovery, Ghidra/vxhunter, UART logs/string tracing Recover the execution model before applying Linux habits. For VxWorks/RTOS, symbols, load address, task/CLI strings, and UART behavior may be stronger first evidence than process-level tracing.

Cheapest next experiment:

Run the smallest trace that crosses the missing edge:
static caller -> QEMU-user/strace -> wrapper/harness -> full-system trace -> hardware trace

Abandon the branch when the next trace would only prove an artifact introduced by the harness, copied sysroot, replacement library, EMBA repair, or debug build.

When source, toolchain, or debug symbols are available

Treat this as a precision upgrade, not as automatic production proof.

Record:

source revision:
compiler/toolchain:
config flags:
optimization level:
library versions:
kernel config:
build-id or hash relationship to shipped binary:
debug symbols sidecar path:

Use rebuilt artifacts this way:

Rebuild asset Best use Required confirmation
unstripped matching application source lines, stack frames, variable names, parser state reproduce the same control-flow decision or crash on the shipped binary
unstripped matching library names and arguments for crypto, config, NVRAM, parser, or protocol calls prove the shipped binary loads the same ABI/soname or statically contains equivalent code
instrumented application sanitizers, assertions, coverage, targeted logs show the instrumented input path maps back to the shipped code path
rebuilt kernel/module driver/ioctl/filesystem/network evidence, ftrace/perf/kprobe visibility confirm target kernel config, module version, device tree, and device nodes
rebuilt bootloader/update verifier signature/hash window, rollback logic, failure branch demonstrate the same package decision on the shipped verifier or boot path

Stop using the debug build as primary evidence when compiler options, libc, heap allocator, board config, endianness, or timing changes the behavior being claimed. Keep it as root-cause support and promote only after a shipped-binary or hardware check agrees.

Hooking and instrumentation choices

Tool family Use when Avoid when
strace / syscall tracing the missing edge is file, socket, process, ioctl, signal, or exec behavior the question is internal parser state or crypto comparison bytes
ltrace / PLT tracing dynamically linked library calls matter and symbols/PLT entries are useful static binaries, stripped/inlined routines, or incompatible libc hide the calls
GDB / gdbserver / QEMU gdbstub breakpoints, watchpoints, crash state, or instruction-level proof matter timing-sensitive bugs or watchdog resets dominate the behavior
LD_PRELOAD / shim libraries you control process launch and need to observe or replace libc/vendor calls setuid/static binaries, loader mismatch, or production launch cannot be mapped
Frida-style dynamic hooking a live Linux/Android-class target can run the agent and function interception is faster than rebuilds tiny systems, unsupported CPU/libc, tight RAM, watchdog/timing-sensitive services
ftrace / tracepoints / kprobes / perf / eBPF kernel, driver, syscall, or scheduler evidence is the missing edge kernel config lacks support, privilege is unavailable, or probe effects alter timing
Valgrind/sanitizers rebuilt or compatible binaries need memory-error evidence architecture/libc support is weak or the production binary cannot be related back
logic analyzer / bus decoder behavior crosses UART, SPI, I2C, CAN, SDIO, USB, PCIe, or GPIO captured bytes cannot be mapped to a firmware consumer

RTOS and bare-metal targets

RTOS firmware usually removes the Linux assumptions: no processes, no /proc, no LD_PRELOAD, often no filesystem, and sometimes no MMU. The first question is not "which Linux debugger?" but "where can the firmware's state be observed without changing the failure?"

RTOS situation First tools What it can prove Stop or downgrade when
Source/symbols available vendor IDE, OpenOCD/J-Link GDB, map file, RTOS-aware debugger, UART/RTT logs, SystemView/Tracealyzer/Zephyr tracing task/ISR/queue/timer state, source-level root cause, timing of protocol handling trace hooks or debug build change timing, memory layout, or scheduling
Binary-only but debug unlocked SWD/JTAG halt/read, OpenOCD/J-Link GDB, hardware breakpoints/watchpoints, memory dumps, UART logs crash PC, registers, RAM state, flash contents, interrupt vector and handler relation halt/debug access changes the bug or memory cannot be legally/readably accessed
Debug locked or fuse-protected UART logs, update package diffing, protocol capture, logic analyzer, power/reset observation external state transitions, accepted commands, bus ownership, update behavior no observable input-output relation maps to code
Peripheral-driven parser logic analyzer, bus decoder, emulator/harness for captured frames, hardware watchpoints if unlocked which frame or register sequence reaches the vulnerable state only generic electrical activity is observed

Zephyr has built-in tracing and compiler-instrumentation options, and SEGGER SystemView records RTOS tasks, interrupts, software timers, and user events over interfaces such as RTT/J-Link/UART/TCP or snapshot modes. Those are published tool capabilities. The decision heuristic is synthesized: use RTOS-aware tracing only when scheduler, interrupt, queue, or timing evidence is what separates a hypothesis from a root cause.

Multi-chip devices: AP, CP, MCU, secure element

Treat each chip as a separate trust and execution domain. AP root is not CP root; a Linux shell on the application processor does not by itself prove code execution on a modem/baseband, radio, PLC, secure element, sensor hub, or power-management MCU.

Build this table before escalating:

chip:
role:
firmware image or blob:
boot authority:
debug interface:
shared buses:
who owns keys/secrets:
who validates updates:
messages accepted from other chips:
reset/power dependency:
observable logs or pins:

Use AP-side Linux tools for AP services. Use JTAG/SWD/UART/RTOS tracing for MCU/CP firmware when debug is authorized and unlocked. Use bus tracing for the boundary: SPI/I2C/UART/RS-232/CAN/USB/SDIO/PCIe frames, reset lines, boot-mode pins, and flash accesses. Escalate to hardware only when the finding depends on which chip owns the trust decision, secret, parser, or actuator.

Evidence thresholds:

Claim Evidence required
AP compromise AP process, privilege, persistence, and shipped startup evidence
CP/MCU influence accepted AP-to-CP/MCU message, command, config write, or firmware-update path
CP/MCU compromise PC/control-flow, firmware modification, debug-shell, or persistent state change inside that chip
secure-element weakness key use, policy bypass, or trusted response misuse, not just I2C/SPI traffic
actuator/OT impact lab-bounded physical or simulator state change with safety constraints

Abandon or narrow the claim when the only evidence is "there is another chip" or "there is traffic on the bus." Promote when a captured message maps to a specific handler, state transition, trust decision, or secret consumer.

SBOM, CVE, And Version Claims

SBOM/CVE output is prioritization, not proof.

For every CVE candidate:

component present?
exact version evidence?
vulnerable feature compiled/enabled?
reachable service/path?
attacker-controlled input?
exploit primitive or realistic impact?

Score each CVE:

Score Meaning
0 name/version only
1 exact component/version found
2 vulnerable feature likely present
3 reachable service/path
4 reproducer or strong behavioral proof
5 root cause and impact demonstrated

Only 3+ is usually worth report space. 4+ is a real finding.

Use EMBA SBOM/default-scan outputs to build the list, then manually verify the top candidates.

High-ROI n-day triage, based on Forescout's OT/IoT router firmware study:

  1. Normalize the component name and exact version from binaries, package metadata, banners, or strings.
  2. Check whether the vulnerable applet, feature, build option, kernel config, or service is actually present.
  3. Check whether vendor advisories or changelogs claim a backport without changing the visible version.
  4. Prefer CVEs with public exploitability writeups only if the same reachable feature exists in this firmware.
  5. Downgrade anything that is name/version-only with no reachable path.

AI-Assisted Research Without Self-Deception

AI is excellent at:

  • summarizing logs
  • turning grep output into lead tables
  • explaining shell scripts
  • drafting Ghidra questions
  • writing parsers for EMBA logs
  • creating report skeletons
  • comparing two snippets
  • suggesting verification steps

AI is not allowed to decide:

  • exploitability
  • reachability
  • whether a CVE applies
  • whether a crash is exploitable
  • whether a credential is impactful

Good prompt:

Here is a firmware evidence bundle: file tree excerpt, init script, route handler, and grep results.
Create a lead table with: source file, route/service, attacker-controlled input, sink, missing evidence, and exact next verification command.
Separate facts from hypotheses. Do not claim a vulnerability unless the evidence supports it.

Bad prompt:

Find all vulnerabilities in this firmware.

Codex app workflow:

1. Ask Codex to read the local docs and current run artifacts.
2. Ask for one narrow script/report/check at a time.
3. Keep commands read-only until you approve otherwise.
4. Save outputs under runs/<case>/.
5. Manually verify every claim before it enters the report.

Use structured files:

runs/<case>/artifacts/file_tree.txt
runs/<case>/artifacts/services.txt
runs/<case>/artifacts/web_routes.txt
runs/<case>/artifacts/sinks.txt
runs/<case>/notes/lead_table.md
runs/<case>/notes/ai_usage_log.md

Local templates:

Evidence Standards

A strong finding includes:

Evidence Example
identity firmware name, version, SHA256
environment VM/tool versions, commands
reachability init script, service port, route, request
root cause file/function/line/address
control parameter/header/file/packet field
trigger curl/QEMU/GDB/log proof
impact command execution, file read, auth bypass, crash, credential exposure
limitations what was not proven
fix concrete remediation

Research-derived evidence habits:

  • Claroty-style chain: firmware identity -> architecture/rootfs -> web config -> route -> CGI/parser -> auth state -> trigger.
  • NCC-style reachability: service exists -> feature gate -> default/optional state -> attacker network position.
  • Neodyme-style surface reduction: firmware extraction -> runtime inventory -> focused binary -> root cause -> reliability constraints.
  • Forescout-style CVE claim: component/version -> feature present -> vendor patch/backport check -> reachable path -> exploitability relevance.

Weak phrases to avoid:

EMBA found a CVE.
AI says this is vulnerable.
This function looks dangerous.
Could possibly be exploited.

Better:

The `/cgi-bin/foo` route accepts `name` from the query string. `foo.cgi` concatenates it into a shell command at line 42 and executes it with `system()`. A request with `name=;sleep 5` delayed the response by five seconds in the emulated environment, demonstrating command injection under the web server user.

Stop Rules

Experienced researchers stop bad paths early.

Stop and pivot when:

  • extraction is blocked by strong encryption and no keys/older firmware/hardware are in scope
  • full emulation consumes more than 25 percent of the assignment with no new evidence
  • a CVE has no reachable feature or service
  • a sink has no input path
  • a crash cannot be tied to attacker-controlled input
  • a secret cannot be tied to a reachable trust path
  • AI cannot cite exact evidence

Research-derived pivots:

  • Encrypted or hardware-coupled firmware: pivot like Neodyme's HP printer work, from failed emulation to static update/decryption path analysis.
  • Huge attack surface: pivot like Neodyme's Netgear work, from "many binaries" to runtime process/network inventory.
  • Scanner-heavy CVE list: pivot like Forescout, from CVSS sorting to component, feature, reachability, and backport validation.
  • Firmware looks clean but network traffic is odd: pivot like Nozomi/Microsoft, from firmware-only analysis to protocol, SDK, infrastructure, and device-configuration evidence.

Use this pivot sentence:

I tested <path>. It is blocked by <specific reason>. The strongest remaining path is <new lead> because <reachability/control/sink evidence>.

Practice Path

Build skill by running this sequence repeatedly:

  1. Extract three known router firmware images.
  2. For each, produce a one-page attack surface map.
  3. For each, choose one reachable lead and reject five weak leads.
  4. Reproduce one web/script issue on a toy firmware or CTF image.
  5. Reproduce one QEMU user-mode binary behavior.
  6. Write one report where you honestly document emulation failure.
  7. Build one parser that converts grep/EMBA outputs into a lead table.
  8. Use AI only on the lead table and compare its ranking to your own.
  9. Repeat with a different firmware family.

The research marker is not that every firmware yields a bug. It is that every case yields a clear, evidence-backed explanation of what was tested, what was proven, what failed, and what should happen next.

External Book References

Older IoT books are background references, not active workflow drivers. Use them only when a case already points to their domain:

Case evidence Consult book material for
UART/JTAG/SWD/SPI/I2C evidence pinout caution, debug-interface classification, and read-only hardware workflow
UPnP/mDNS/DNS-SD/WS-Discovery strings LAN discovery threat modeling and feature-gated reachability
BLE/ZigBee/radio strings or companion-app clues pairing, keys, UUIDs, GATT/opcode concepts, and lab hardware constraints
mobile/cloud endpoint names in firmware product attack-surface mapping across firmware, app, cloud, and local discovery

Do not use old tool commands or chapter scores as evidence. Use the current case docs above for decisions and current tool documentation for commands.

Sources