Unfiltered. If this section reads as if things went smoothly, I have written it wrong.
Each entry is a real failure: what I thought was happening, what was actually happening, and how I caught it. Grouped by the class of failure. The insight that transfers beyond this specific build is what matters.
This happened more than once, in two distinct forms.
The first form: the model produced prose describing an action without emitting the corresponding tool call. The runtime had nothing to execute. The chat turn looked successful because it ended without an error. I ran cat ~/.openclaw/jobs.json expecting to see my new entry. The file was unchanged. The model had predicted a plausible-sounding confirmation without issuing the actual write.
The second form: the model emitted a tool call, the call executed, a file was written -- but the content was the model's paraphrase of my specification, not my specification itself. The file existed. Some sections were reworded. One line was summarised into a different sentence. The bytes were wrong.
Both forms share the same root: an LLM generates text, including text that sounds like confirmation of success. It has no privileged knowledge of whether the action actually happened. There is no syntactic difference between "I have done X" when X happened and "I have done X" when it didn't.
The fix. For any step where the exact bytes matter, bypass the model. Write a shell script, have the agent call the script. The script runs verbatim regardless of how the model phrases the invocation.
The insight. Verification must happen at a layer the model cannot fake: the filesystem, the database, the downstream system. cat, ls, git log. Every time. In any agentic system, agent-reported success is a hypothesis, not a fact. Rule: trust reads, verify writes -- applies to any agent on any framework.
The diagrams showed OS cron firing a shell command at the gateway port to trigger scheduled jobs. That picture is wrong.
OS cron is never involved. The gateway's own internal scheduler reads jobs.json and creates an isolated session directly inside the same process. The scheduler and the executor are the same process. No shell command. No port knock. The diagram most people work from is incorrect, and most people never find out because their jobs happen to work anyway.
This produced two concrete failures. First: a cron job created with the correct sessionTarget: isolated but an incorrect sessionKey. The job would have run in the wrong session type -- silent failure, no Telegram announce, no run record. Second: a writer agent directory existed at the expected path. I almost accepted that as proof the agent was configured. The load-bearing config key in openclaw.json still needed verifying. Directory present and delegation working are two different claims.
Both times: wrong mental model of the chain, verification stopped at the wrong layer.
The insight. With the wrong mental model of a chain, debugging goes in the wrong direction. Job not firing? You'd check crontab. Wrong. Check: is the gateway running, is jobs.json valid, is the sessionKey correct, does the payload type match the session type. The right debugging sequence follows from the right mental model -- and the right mental model only comes from tracing the actual chain, not reading the diagram. Wrong model = wrong debug sequence = invisible failures.
The heartbeat mechanism was disabled at the start. The stated reason: nothing useful to do yet.
The real reason is sharper. Without identity files loaded, the agent has full action permissions and zero constraints. Heartbeat firing into that vacuum isn't unproductive -- it's an unsupervised agent with no rules acting on the infrastructure. The order matters: constraints before capability, not capability then constraints.
This same principle surfaced again when installing the imap-smtp-email community skill. The skill ran with the agent's full credentials from the moment it was installed -- before I fully understood what it did or what it could access. The README had four undocumented gaps (TLS defaults wrong for Gmail, missing package.json, SMTP vars required for read-only setup, env resolution path undocumented). But the more important issue wasn't the documentation. It was that a third-party capability inherited everything the agent could do, immediately on install.
The insight. Capability before constraints is the most common mistake in AI product design. The question is never just "what can this agent do" -- it's "what is the worst case if this agent acts on its own initiative right now, with what it currently knows and doesn't know." Audit before install applies to skills, MCPs, plugins, any third-party agent capability. They all run with the agent's full permissions from the moment they land.
Skills installed in the workspace tier are invisible to isolated cron sessions. The same skill that works perfectly in interactive webchat does not exist in the cron execution context. Skills intended for scheduled jobs have to live at the global path (~/.openclaw/skills/), not the workspace path. The capability surface changes depending on where the agent runs.
This is a silent failure mode. Nothing errors. The cron session simply doesn't have the skill available and proceeds without it.
The insight. "It worked in testing" is often a deployment context mismatch, not a code bug. Before shipping any autonomous feature, the question is: what does this agent actually have access to at runtime -- not what you built, not what you tested, but what the production execution context sees. This is a product spec question, not just an engineering checklist item. Deployment context is a product constraint.
gpt-4o-mini interprets instructions rather than executing them literally. Two silent write failures before upgrading. Upgrading to gpt-4.1-mini narrowed the problem but didn't close it.
The interpretation gap is a property of how LLMs work, not a bug in a specific model tier. Models are trained to be helpful, which means rewording, summarising, and improving -- exactly the wrong behaviour when you need verbatim output.
The insight. For any task requiring deterministic output -- file writes, config updates, structured data -- the right fix is a shell script, not a better model. Scripts execute. Models interpret. The model upgrade buys reliability headroom. The architectural fix buys guarantees. They are not the same thing. Reaching for a better model before asking "should this be a script?" is optimising the wrong layer.
The first attempt at Gmail authentication used OAuth. Created a Google Cloud project, enabled the Gmail API, created credentials, started the consent flow.
The consent flow for a personal-scope Gmail app run from a headless VPS is not a supported path. Google reroutes through a browser consent screen. On a headless server, there is no browser. Workarounds existed but added enough friction to stop and ask whether OAuth was the right choice at all.
Switched to an app password: 2FA enabled, 16-character app password scoped to Mail. Done in ten minutes.
The tradeoff is real -- app passwords don't expire automatically, don't have scoped permissions, and don't revoke cleanly if leaked. For a personal agent on a private tailnet, acceptable. For anything shared or production, OAuth done properly from the start is the right answer.
The insight. Match the authentication method to the threat model, not to the spec. "Correct" and "shippable" are sometimes different answers.
Installation, identity files, and Telegram came up cleanly. Web search, outbound email, and the writer sub-agent applied the patterns above without new failure classes. The self-audit ran clean at the end: 10/10 PASS.
One passive observation from early setup: the install wizard left .clobbered backup files showing openclaw.json was overwritten multiple times during initial config. Nothing broke, but it signals the default setup process is less idempotent than the documentation implies.
Before you build: never enable autonomous triggers before you have defined what the agent is not allowed to do. Capability without constraints is not a feature -- it's an unsupervised agent acting on the infrastructure with no guardrails. And before you install any third-party capability, read what it does -- skills and plugins inherit the agent's full permissions from the moment they're installed, before you've verified what they can access.
When you build: understand the actual execution chain, not the diagram. The diagram is often wrong. Know which component initiates each step, because when something breaks you will debug in the sequence your mental model suggests -- and a wrong model sends you to the wrong place every time. Know that deployment context determines capability surface -- a skill that works in interactive chat may not exist in an autonomous scheduled session, and that is a product spec question you need to answer before you ship, not after.
When you verify: never trust what an agent says it did. The model generates plausible-sounding confirmation text regardless of whether the action happened. Check the filesystem, the database, the downstream system -- the layer that cannot lie. And when something isn't working reliably, reach for a shell script before reaching for a better model. Scripts execute. Models interpret. The architectural fix gives you guarantees. The model upgrade gives you headroom. They are not the same thing.
The sequence is: permission model first, then build with the right mental model of the chain, then verify at ground truth. Same dependency order as the five problems of autonomous systems. The constraints have to come first, and the verification has to be real -- not what the agent said happened, but what the filesystem shows.