-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathverify-proof.py
More file actions
1144 lines (1086 loc) · 57.3 KB
/
Copy pathverify-proof.py
File metadata and controls
1144 lines (1086 loc) · 57.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env python3
"""One-command verifier for every claim in docs/DEVNET-PROOF.md (stdlib only, no install).
A judge should not have to click eight explorer links or trust a screenshot. This queries
Solana devnet directly and checks that each on-chain claim still holds: the programs are
executable, the feed PDA is owned by the oracle program, and every referenced transaction
landed with the exact success/rejection this submission claims.
python3 scripts/verify-proof.py # checks devnet, prints PASS/FAIL per claim
RPC_URL=https://your-rpc python3 scripts/verify-proof.py
Exit 0 = every claim verified. Exit 1 = at least one claim FAILED, meaning a thing this
repo asserts is no longer true. Exit 2 = TRANSPORT, meaning the network would not answer,
so this run has no opinion and a retry is the right response.
That third code is the point, and this docstring claimed only two of them until 2026-08-04
while the code had carried three since 2026-07-26. Folding "the RPC was unreachable" into
exit 1 tells a stranger a claim broke when nothing broke except the connection. The sibling
`feed_heartbeat.py` had the same defect, and it got the fix when this one got only the code
change, which is the instance-fixed-class-missed shape this project keeps re-finding.
"""
import base64
import json
from pathlib import Path
import os
import struct
import sys
import time
import urllib.error
import urllib.request
RPC = os.environ.get("RPC_URL", "https://api.devnet.solana.com")
# (label, address, want_executable) -- accounts that must exist on devnet
ACCOUNTS = [
(
"oracle program zeroclaw_oracle",
"EFCRmE5wFLoo5zJ4cu4J6rbQjmkiok8FmDekTGGXrCKn",
True,
),
(
"consumer program consumer_example",
"B2scuv95pA7yA3Kj36wmfoSVZ94WZfUmtwsfr9Kw39Pt",
True,
),
("SF Allowances (audited)", "De1egAFMkMWZSN5rYXRj9CAdheBamobVNubTsi9avR44", True),
(
"device feed PDA (agent-driven, historical)",
"CfWaZAQ9mG1WbAhNCSQJz284MR1NC8fvfiHRaNvyQ9sU",
False,
),
(
"device feed PDA (deterministic LLM-free, laptop)",
"3aMsPjXuMwRNqW3Yy6aqATp1N8nDXc4ZQMpGEncTVx8K",
False,
),
(
"device feed PDA (ARM node, node-born key)",
"JEtuZkcRzePbbLo8oiM26aqpbt1zJyLP4snvQCjVveg",
False,
),
]
FEED_OWNER = "EFCRmE5wFLoo5zJ4cu4J6rbQjmkiok8FmDekTGGXrCKn" # feed PDA must be owned by the oracle
FEED_PDAS = {
"CfWaZAQ9mG1WbAhNCSQJz284MR1NC8fvfiHRaNvyQ9sU",
"3aMsPjXuMwRNqW3Yy6aqATp1N8nDXc4ZQMpGEncTVx8K",
"JEtuZkcRzePbbLo8oiM26aqpbt1zJyLP4snvQCjVveg",
}
# Ownership alone cannot distinguish a live feed from one that stopped months ago, so the
# always-on claim gets its own check. Only the ARM node is asserted fresh: it is the feed
# that backs "yours, running". The laptop publisher is secondary by design and is allowed
# to go quiet when that machine sleeps.
LIVE_FEED = ("ARM node feed", "JEtuZkcRzePbbLo8oiM26aqpbt1zJyLP4snvQCjVveg")
# Reported but NOT gating, for exactly the reason above: this one runs on a laptop that is
# allowed to sleep, so failing the run on it would train a reader to ignore a red result.
SECONDARY_FEED = (
"laptop deterministic feed",
"3aMsPjXuMwRNqW3Yy6aqATp1N8nDXc4ZQMpGEncTVx8K",
)
# Cadence is 20 minutes. The threshold is deliberately loose so one skipped reading (a
# transient upstream weather-API failure, which the publisher refuses to paper over with a
# fabricated value) does not read as a dead node.
# Overridable so the gate can be demonstrated rather than trusted: run
# MAX_FEED_AGE_MIN=0 python3 scripts/verify-proof.py
# and the live check goes red with exit 1 while every static claim stays green. A liveness
# check nobody has watched fail is indistinguishable from one that cannot fail.
MAX_FEED_AGE_MIN = int(os.environ.get("MAX_FEED_AGE_MIN", "90"))
# The laptop publisher fires on this cadence. Used only to decide whether a publish
# ATTEMPT is recent enough to be meaningful, never to gate anything.
PUBLISH_CADENCE_MIN = 20
# Written by .tools/feed_publish_hidden.vbs on every run, including failed ones.
# Gitignored with the rest of .devnet-proof, so a fresh clone simply has no file here
# and the helper below returns None, which is the correct answer for a machine that
# does not run the publisher at all.
_ATTEMPT_LOG = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
".devnet-proof",
"feed-attempt.log",
)
def attempt_heartbeat():
"""Age in minutes of the last publish attempt, plus how it ended.
This exists because of a real 2026-07-26 failure. The WSL VM wedged, the publisher
stopped running entirely, and NOTHING went red: the launcher was fire-and-forget so
it returned 0, Task Scheduler logged ~20 consecutive successes with zero missed
runs, and this script called a 6.6-hour-dead feed "quiet (allowed)". The publish log
could not help either, because a script that never executes cannot write its own
failure line.
The root problem was that two very different situations produced an identical
signature: a laptop that was switched off, and a publisher that ran and failed, both
just stop appending to the publish log. The launcher now records every attempt before
and after it runs, so the two can be told apart here.
Returns (age_min, outcome) where outcome is "rc=N" or "no result, killed mid-run",
or (None, None) when there is no log to read.
"""
try:
age_min = (time.time() - os.path.getmtime(_ATTEMPT_LOG)) / 60.0
lines = [
ln.strip()
for ln in open(_ATTEMPT_LOG, "r", encoding="utf-8", errors="replace")
if ln.strip()
]
except Exception:
return None, None
if not lines:
return age_min, "empty log"
# A trailing "start" with no "rc=" after it means the run hung and was killed by the
# task's execution time limit before it could record an outcome.
last = lines[-1]
return age_min, last if last.startswith("rc=") else "no result, killed mid-run"
# The shop half of "Both are running". Checked because the node and the shop fail
# independently: the node is Oracle Cloud systemd, the shop is a laptop daemon plus a CDN
# page, so the node can publish happily through a completely dead shop.
SHOP_PAY_URL = os.environ.get("SHOP_PAY_URL", "https://zeroclaw-shop-pay.pages.dev/")
# Asserted inside the page body: HTTP 200 only proves a CDN answered, while the pinned
# merchant address is what makes it this shop's page rather than any page.
MERCHANT_PIN = "C331X4YCHCdcESexRTKSjE5etjsWyWJLK73Z18ZWiLHJ"
# The shop daemon's own liveness, asked of systemd inside the box and served over the
# node's named tunnel. This is the signal the pay-page check structurally cannot carry: the
# page is a CDN asset that answers whether or not the daemon runs, so until this endpoint
# existed a dead shop and a quiet one were the same observation from outside.
SHOP_HEALTH_URL = os.environ.get("SHOP_HEALTH_URL", "https://x402.perfpilot.dev/health")
# DeviceFeed: disc8 + authority32 + device32 + feed_kind1 + value_i64 + scale_i8
# + unit[12] + sequence_u64 + observed_at_i64 + published_at_i64 + bump1
FEED_LEN = 8 + 32 + 32 + 1 + 8 + 1 + 12 + 8 + 8 + 8 + 1
# (label, signature, want_err) want_err=None means success (err:null)
TXS = [
(
"shop Track-A settlement (payment_watch PAID)",
"4kDo6NCcAxSe3BSTtQ4onTASenxRWr2miagweVway3RnDMLG7drv6NkTdV7eRtTSDcNXURy2ESpKcqkk2jG9sYqS",
None,
),
(
"x402 machine-commerce settlement",
"EkBmoDknDryQpDtD6hnLoCdhhRjAo3Vmn15VmkQi7niqYHnK5XYL8FpxLabDiQ2S2QuTdD3vsTXMSra72LXgApE",
None,
),
(
"allowance within-cap transfer (succeeds)",
"5qyr7jJi8zb6SjZjnA2QT5C9nuZYgSw6raAefjmWnDDMf3JRgkQX19zssE57EpFSHVCCPfbj5qyxcYSQcfEq9W3Z",
None,
),
# 300 is AmountExceedsLimit in the solana-foundation program's own errors.rs and IDL;
# the citation is in docs/MAINNET-PROOF.md rather than assumed here.
(
"allowance OVER-cap transfer (rejected 0x12c)",
"3TLSrfWVYdC3hSiAWnyyd7T694bLJQDtdJYQ64EWUsBNDehGc6Kq1veR7xa8Y1BiMdpvfFm3N1dKjDrXF3BEq2ps",
{"InstructionError": [0, {"Custom": 300}]},
),
]
# Three attempts, ~1s then ~2s apart. Deliberately small: enough to ride out a blip,
# short enough that a genuinely unreachable RPC still fails the run promptly.
RPC_ATTEMPTS = 3
# The retry budget is GLOBAL, not per call, and that is the load-bearing part. Per call
# it looks cheap, but this script makes a dozen gating checks, so against an RPC that
# HANGS rather than refuses the worst case is attempts x timeout x checks, which is
# minutes of a reader staring at nothing. One shared budget means a genuinely dead
# endpoint costs the retries once and every later check fails fast.
RETRY_BUDGET_S = 25.0
_retry_spent = 0.0
def rpc(method, params):
"""One JSON-RPC call, retrying only a TRANSPORT failure.
proof-check.yml already retries the whole script on a transport blip, so CI was
resilient to a flaky network and a human running this by hand was not. That is
backwards for the artifact whose entire job is letting a stranger re-verify the
claims: a reader on hotel wifi got a red result while every claim still held.
Retrying HERE is safe, and safe by construction rather than by a heuristic, which
is the part worth keeping. The transport-versus-claim distinction this file is
careful about survives because a claim that stopped holding does not raise. It
arrives as a SUCCESSFUL response carrying a different value, and is judged by the
caller. This function only ever sees the network refusing, so it can never retry a
broken claim into looking healthy.
"""
body = json.dumps(
{"jsonrpc": "2.0", "id": 1, "method": method, "params": params}
).encode()
global _retry_spent
for attempt in range(RPC_ATTEMPTS):
started = time.monotonic()
req = urllib.request.Request(
RPC, data=body, headers={"Content-Type": "application/json"}
)
try:
with urllib.request.urlopen(req, timeout=20) as r:
return json.load(r).get("result")
except Exception as e:
# A non-transport error is the caller's to see immediately, and the last
# attempt re-raises so an unreachable RPC still exits as a transport
# failure with its original exception rather than a synthesised one.
if not is_transport_error(e) or attempt == RPC_ATTEMPTS - 1:
raise
# Charge BOTH the failed attempt and the pause to the shared budget. The
# attempt is the expensive half when the endpoint hangs, so a budget that
# counted only sleeps would not bound anything.
_retry_spent += time.monotonic() - started
if _retry_spent >= RETRY_BUDGET_S:
raise
pause = 1.0 * (attempt + 1)
time.sleep(pause)
_retry_spent += pause
def read_feed(addr):
"""Decode a DeviceFeed account. Returns (reading, seq, age_min) or raises."""
v = rpc("getAccountInfo", [addr, {"encoding": "base64"}])
val = v.get("value") if v else None
raw = base64.b64decode(val["data"][0]) if val else b""
if len(raw) != FEED_LEN:
raise ValueError(f"unexpected account length {len(raw)}")
o = 8 + 32 + 32 + 1
value = struct.unpack_from("<q", raw, o)[0]
o += 8
scale = struct.unpack_from("<b", raw, o)[0]
o += 1
unit = raw[o : o + 12].rstrip(b"\x00").decode("ascii", "ignore")
o += 12
seq = struct.unpack_from("<Q", raw, o)[0]
o += 8 + 8
published = struct.unpack_from("<q", raw, o)[0]
return f"{value * (10**scale):.2f} {unit}", seq, (time.time() - published) / 60
def is_transport_error(e):
"""True when the network refused, rather than a claim failing to hold.
CI needs this distinction and could not previously get it. proof-check retries a
transport blip but must NOT retry a claim that stopped holding, and it decided which
was which by grepping this script's own output for words like "unreachable". That
grep reads the whole log, including the SECONDARY feed line, which is non-gating and
prints exactly that word when its read fails. So a genuine broken claim landing in the
same run as a secondary-read blip was classified as transport, retried three times,
and finally reported as an RPC problem rather than as the claim that actually broke.
An exit code cannot be misread that way, so the script now says which kind of failure
it had and the workflow branches on a number instead of on a sentence.
"""
if isinstance(e, urllib.error.HTTPError):
# 429 and 408 are TRANSPORT, and reading them as claim failures was a real defect
# rather than a theoretical one. A public Solana RPC rate-limits routinely, so a
# reader on a shared endpoint got a red verdict reported as "this claim stopped
# holding" when the chain had not been consulted at all. That is the worst possible
# direction for this script to be wrong in: it is the artifact whose whole job is
# letting a stranger re-verify, and it was telling some of them the proofs broke.
#
# Both are safe to retry for the same reason the docstring above gives: a claim
# that stopped holding arrives as a SUCCESSFUL response carrying a different value,
# so it never reaches this function and can never be retried into looking healthy.
return e.code >= 500 or e.code in (408, 429)
if isinstance(e, urllib.error.URLError):
return True
return isinstance(e, (TimeoutError, ConnectionError))
def locate_commit(sha):
"""Where a commit the gate names sits relative to THIS clone. Never a verdict.
The gate reports the commit its binary was compiled from. That is a claim, and the one
thing a reader holding a clone can check about it is whether the commit is real here at
all, and if so how far back it is. Both answers are informative and neither is a fault:
a clone made before that commit landed, a shallow clone, and a clone of a fork all fail
to find a perfectly good commit, so an absence says something about the clone at least
as often as about the binary.
Returns a short human phrase, always. Every failure path -- no git on PATH, not a
repository, a sha this clone has never seen -- resolves to a sentence rather than to an
exception or to None, because the caller prints it inline and a None there would read as
a defect in the gate.
"""
# A `git-dirty` build reports `<head>-dirty`. The head part is still worth locating, so the
# suffix is stripped rather than making the whole value unlookupable -- but the answer says
# which commit it located, because that commit is NOT what was compiled. The caller's own
# source line carries that warning too; saying it twice is cheaper than a reader taking a
# located commit as confirmation the binary matches it.
note = ""
if isinstance(sha, str) and sha.endswith("-dirty"):
sha, note = sha[: -len("-dirty")], " (the clean commit it was built on top of)"
if (
not isinstance(sha, str)
or not 7 <= len(sha) <= 40
or sha.strip("0123456789abcdef") != ""
):
# Includes the gate's own `unavailable` case, whose commit is the literal "unknown".
return "not a commit id, so nothing to look up here"
root = str(Path(__file__).resolve().parent.parent)
try:
import subprocess
kind = subprocess.run(
["git", "-C", root, "cat-file", "-t", sha],
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
timeout=20,
)
if kind.returncode != 0 or kind.stdout.strip() != "commit":
return f"not a commit this clone holds{note}, which a shallow or older clone also looks like"
anc = subprocess.run(
["git", "-C", root, "merge-base", "--is-ancestor", sha, "HEAD"],
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
timeout=20,
)
if anc.returncode != 0:
return f"a real commit here{note}, but not an ancestor of your HEAD"
behind = subprocess.run(
["git", "-C", root, "rev-list", "--count", f"{sha}..HEAD"],
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
timeout=20,
)
n = behind.stdout.strip()
if behind.returncode != 0 or not n.isdigit():
return f"an ancestor of your HEAD{note}; distance unreadable"
return (
f"your HEAD is that same commit{note}"
if n == "0"
else f"an ancestor of your HEAD, {n} commit(s) back{note}"
)
except Exception as e:
return f"could not be looked up ({type(e).__name__})"
def main():
fails = 0
static_fails = 0
# Counted only for GATING checks. The secondary feed never gates, so its transport
# trouble must not make a real failure elsewhere look retryable, which is the exact
# confusion this replaces.
transport_fails = 0
print(f"verifying docs/DEVNET-PROOF.md against {RPC}\n")
print(
"STATIC claims -- the record. These are immutable devnet history and deployed"
)
print(
"program state; once true they stay true, so they prove the work happened, NOT"
)
print("that anything is running right now.\n")
for label, addr, want_exec in ACCOUNTS:
try:
v = rpc("getAccountInfo", [addr, {"encoding": "base64"}])
val = v.get("value") if v else None
if not val:
print(f"FAIL {label}: account not found")
fails += 1
continue
if want_exec and not val.get("executable"):
print(f"FAIL {label}: not executable")
fails += 1
continue
if addr in FEED_PDAS and val.get("owner") != FEED_OWNER:
print(f"FAIL {label}: wrong owner {val.get('owner')}")
fails += 1
continue
extra = "executable" if want_exec else f"owner={val.get('owner')[:8]}"
print(f"PASS {label} ({extra})")
except Exception as e:
print(f"FAIL {label}: RPC error {e}")
fails += 1
if is_transport_error(e):
transport_fails += 1
# A transaction the RPC will not serve has THREE possible causes and only one of
# them is a broken claim. Public devnet prunes after about four days, so a
# transaction older than that is absent from the endpoint while remaining a real,
# settled transaction. Reporting that as a claim that stopped holding is a false
# red, and a checker that cries wolf trains a reader to ignore it.
# So: when the RPC has nothing, fall back to the captured bundle and verify the
# signature offline. If it verifies there, the claim holds and the endpoint is
# simply the wrong instrument for it.
bundle_txs = {}
bundle_path = (
Path(__file__).resolve().parent.parent
/ "docs"
/ "proof-bundle"
/ "devnet-transactions.json"
)
if bundle_path.exists():
try:
bundle_txs = json.loads(bundle_path.read_text(encoding="utf-8")).get(
"transactions", {}
)
except Exception:
bundle_txs = {}
def verified_offline(sig: str) -> bool:
"""True iff the captured bundle proves this transaction by signature."""
entry = bundle_txs.get(sig)
if not entry or entry.get("status") != "CAPTURED":
return False
try:
import subprocess
r = subprocess.run(
[
sys.executable,
str(Path(__file__).resolve().parent / "verify_proof_offline.py"),
],
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
timeout=180,
)
return r.returncode == 0 and sig[:16] in r.stdout
except Exception:
return False
for label, sig, want_err in TXS:
try:
t = rpc("getTransaction", [sig, {"maxSupportedTransactionVersion": 0}])
if not t:
if verified_offline(sig):
print(
f"PASS {label}: pruned by the endpoint, verified offline from the bundle"
)
else:
print(
f"FAIL {label}: tx not found, and no captured bytes to verify offline"
)
fails += 1
continue
got = t.get("meta", {}).get("err")
if got == want_err:
print(f"PASS {label} (err={json.dumps(got)})")
else:
print(
f"FAIL {label}: err={json.dumps(got)} expected {json.dumps(want_err)}"
)
fails += 1
except Exception as e:
print(f"FAIL {label}: RPC error {e}")
fails += 1
if is_transport_error(e):
transport_fails += 1
static_fails = fails
print(
"\nLIVE claims -- the only checks here that can go red. Everything above stays"
)
print("green whether or not a single machine of ours is switched on.\n")
label, addr = LIVE_FEED
try:
reading, seq, age_min = read_feed(addr)
if age_min > MAX_FEED_AGE_MIN:
print(
f"FAIL {label} freshness: last reading {age_min:.0f} min ago "
f"(> {MAX_FEED_AGE_MIN}); the node is not publishing"
)
fails += 1
else:
print(
f"PASS {label} freshness ({reading}, seq={seq}, {age_min:.0f} min ago)"
)
except Exception as e:
print(f"FAIL {label} freshness: {e}")
fails += 1
if is_transport_error(e):
transport_fails += 1
# Reported, never gating. This publisher runs on a laptop that is allowed to sleep, so
# failing the whole run on it would teach a reader that a red line here means nothing,
# which is how a liveness check stops being one.
label, addr = SECONDARY_FEED
try:
reading, seq, age_min = read_feed(addr)
if age_min <= MAX_FEED_AGE_MIN:
print(
f"INFO {label}: fresh ({reading}, seq={seq}, {age_min:.0f} min ago, "
f"not gating)"
)
else:
# Stale. Two very different causes, and until 2026-07-26 they were
# indistinguishable here, which let a dead publisher read as "allowed" for
# 6.6 hours. The attempt heartbeat separates them.
beat_age, outcome = attempt_heartbeat()
recent_attempt = (
beat_age is not None and beat_age <= 2 * PUBLISH_CADENCE_MIN
)
if recent_attempt:
print(
f"WARN {label}: stale AND the publisher is still firing "
f"({reading}, seq={seq}, {age_min:.0f} min ago; last attempt "
f"{beat_age:.0f} min ago, {outcome}). It is running and not "
f"landing, so this is broken rather than asleep. Still not "
f"gating: the ARM node above carries the claim."
)
else:
detail = (
"no publish attempt logged recently, so this machine was away"
if beat_age is None
else f"last attempt {beat_age:.0f} min ago, so this machine was away"
)
print(
f"INFO {label}: quiet (allowed) ({reading}, seq={seq}, "
f"{age_min:.0f} min ago; {detail}, not gating)"
)
except Exception as e:
print(f"INFO {label}: unreadable ({e}, not gating)")
# The shop is the other headline use case, and until now nothing in this script touched
# it. An audit put the hole precisely: the ARM node runs on Oracle Cloud systemd,
# independent of the shop, so a dead shop plus a publishing node printed a clean bill of
# health.
#
# HONEST SCOPE, because the first version of this comment overclaimed. What follows
# checks a STATIC Cloudflare Pages asset. That page is served by a CDN and answers 200
# whether or not the shop daemon is running, so this does NOT detect a dead daemon and
# does NOT by itself close the false-green. Demonstrated the day it was written: the WSL
# VM hosting the daemon was wedged (marked Running, unresponsive past 45s) while this
# check would still have passed on pin presence alone.
#
# What it DOES prove is narrower and still worth gating on: that the deployed page is
# the pinned build rather than a stale one. That is a real regression class, since the
# merchant pin is the control standing between a swapped recipient and a transfer.
# Detecting a dead daemon needs a signal from the daemon itself (a health endpoint or a
# channel round-trip) and is tracked separately rather than pretended at here.
try:
req = urllib.request.Request(
SHOP_PAY_URL, headers={"User-Agent": "Mozilla/5.0 (verify-proof)"}
)
# Read a generous cap rather than 64 KiB, and DISTINGUISH a truncated read from a
# genuine absence. The old 65536 was a silent time bomb: the pin sat at byte 63,009
# until the page grew by 2,545 bytes, which pushed it to 65,554 and put it EIGHTEEN
# bytes past the cut. The gate then reported "merchant pin MISSING" about a page
# serving the correct pin, which is the worst possible wording -- it names a
# swapped-recipient regression when the real event was the page getting longer.
# A cap is still right (an attacker-controlled body should not be read unbounded),
# so the fix is to notice when the cap was reached instead of reasoning past it.
CAP = 2_000_000
with urllib.request.urlopen(req, timeout=20) as r:
raw = r.read(CAP + 1)
truncated = len(raw) > CAP
body = raw[:CAP].decode("utf-8", "replace")
# 200 alone only proves a CDN answered. The pinned merchant address is what makes
# the page the shop's page rather than any page, so that is what is asserted.
if r.status == 200 and MERCHANT_PIN in body:
print(f"PASS shop pay page reachable and pinned to the shop ({r.status})")
elif truncated:
print(
f"FAIL shop pay page: HTTP {r.status}, body exceeded the {CAP:,}-byte read "
f"cap, so the merchant pin was NOT SEARCHED rather than found absent"
)
fails += 1
else:
print(
f"FAIL shop pay page: HTTP {r.status}, merchant pin "
f"MISSING from {len(body):,} bytes read in full"
)
fails += 1
except Exception as e:
print(f"FAIL shop pay page unreachable: {e}")
fails += 1
if is_transport_error(e):
transport_fails += 1
# The check the two above structurally cannot make. Both of them observe things OUTSIDE
# the shop: a feed the node publishes, and a page a CDN serves. Neither can distinguish a
# shop that is quiet from one that is stopped, which is the whole reason the 2026-07-26
# outage sat unnoticed. This asks systemd, inside the box, and the answer travels out over
# the node's named tunnel.
#
# WHAT IS GATED, and why it is not the obvious field. The gate is the unit's state, not
# how recently it handled traffic. A shop nobody has messaged for six hours is healthy and
# gating on trace age would paint it red, which is how a liveness line stops meaning
# anything to whoever reads it next. The age is printed because it is worth seeing and
# never asserted on.
# Bound before the try so an unreachable endpoint leaves it None rather than undefined.
# The x402 ledger check below reads it, and an unbound name there would raise inside the
# verifier itself, which is the one place a crash reads as "the claim could not be
# checked" when it actually means "the checker is broken".
health = None
try:
req = urllib.request.Request(
SHOP_HEALTH_URL, headers={"User-Agent": "Mozilla/5.0 (verify-proof)"}
)
with urllib.request.urlopen(req, timeout=20) as r:
health = json.loads(r.read(65536).decode("utf-8", "replace"))
shop = health.get("shop") or {}
# .get with no default, so a field the endpoint stops sending reads as None and fails
# rather than defaulting to something that passes.
alive = shop.get("active") is True and shop.get("state") == "active"
age_s = shop.get("trace_age_seconds")
age_note = (
f"last handled traffic {age_s / 60:.0f} min ago"
if isinstance(age_s, (int, float))
else "traffic age not reported"
)
if r.status == 200 and alive:
print(
f"PASS shop agent process alive per systemd "
f"({shop.get('unit')}, {shop.get('state')}; {age_note}, not gating)"
)
else:
print(
f"FAIL shop agent liveness: HTTP {r.status}, unit "
f"{shop.get('unit')!r} state {shop.get('state')!r} "
f"active={shop.get('active')!r}"
)
fails += 1
except Exception as e:
print(f"FAIL shop agent /health unreachable: {e}")
fails += 1
if is_transport_error(e):
transport_fails += 1
# A BODY CAN PARSE AND STILL NOT BE AN OBJECT. `json.loads` happily returns a bare string or a
# list, so `health` survives the block above holding one: the `.get` in there raises, the
# except prints and counts a failure, and the name stays bound to the non-mapping. Every block
# below assumes a mapping and would raise on it OUTSIDE any try, in a `main()` with no
# top-level handler, taking the whole run down and printing no tally at all. Measured on the
# receipts block, which is where it landed first, and reachable in the same way further down.
#
# Normalised once here rather than at each site: the failure has already been reported and
# counted, so None is exactly what the downstream blocks' existing "already failed above"
# branches expect, and no message changes.
if not isinstance(health, dict):
health = None
# CLAIM: a receipt this shop sent actually reached a customer.
#
# Every check above observes a PROCESS or a PAGE. None of them can tell a shop whose
# send path is refused from one that is serving customers, because a live process with
# a dead channel passes all three. The gate now reads the settlement announcer's own
# record of what it delivered and reports it here, which is the first signal in this
# file that is an EFFECT of the channel working rather than a state around it.
#
# WHY THE ANNOUNCER RATHER THAN THE SHOP DAEMON. The announcer is a shell script that
# never consults a model, and its `sent:` and `SEND FAILED` lines are the receipt
# actually landing or not. The shop daemon's log cannot answer this: the announcer is a
# separate unit whose stdout goes to the journal, so a reader pointed at the daemon log
# would scan forever and never see a delivery.
#
# NOT GATED, and that is a decision rather than caution. A gate on live box state can
# turn main red with no repo change, and a channel reconnect is exactly the transient
# that would do it. It is also not gateable on the merits: the announcer sends only when
# a payment settles, so an absence of receipts is what a quiet Tuesday looks like as much
# as what a broken send path looks like, and a red that means either means neither.
#
# SO ONLY `connected` IS EVIDENCE. Everything else prints what is missing and asserts
# nothing, including the empty case, which must not be read as an outage.
rc = (health or {}).get("receipts")
if health is None:
# /health already failed above with its own reason. A second line about a block
# inside a body nobody received would be noise dressed as a finding.
pass
elif rc is None:
print(
"PEND receipt delivery not yet observable: the deployed gate predates the "
"/health receipts block. Not gating. It becomes a live claim on the next deploy."
)
else:
# .get with no default throughout, so a field the endpoint stops sending reads as
# None and is reported as malformed rather than defaulting into something quiet.
d = rc.get("delivery")
scanned = rc.get("lines_scanned")
found = rc.get("records_found")
if (
not isinstance(d, dict)
or not isinstance(scanned, int)
or not isinstance(found, int)
):
print(
f"INFO receipt delivery block malformed: delivery={type(d).__name__}, "
f"lines_scanned={scanned!r}, records_found={found!r}; not gating"
)
elif rc.get("log_readable") is not True:
print(
f"INFO receipt delivery unknown: the announcer's record could not be read "
f"({rc.get('detail')}). Nothing is claimed either way, not gating"
)
else:
status = d.get("status")
age = d.get("last_success_age_seconds")
chan = d.get("channel")
run = rc.get("last_run") or {}
# The denominator travels with the count, always. A zero out of zero lines and
# a zero out of hundreds are different verdicts about this instrument, and only
# one of them is a statement about the shop.
basis = f"{found} delivery record(s) in {scanned} line(s) read from {rc.get('source')}"
run_note = ""
if isinstance(run.get("announced"), int):
committed = (
"committed" if run.get("ledger_committed") else "NOT committed"
)
run_note = (
f"; last run announced {run['announced']}, ledger {committed}"
)
if status == "connected":
mins = (
f"{age / 60:.0f} min ago"
if isinstance(age, (int, float))
else "age unreported"
)
print(
f"INFO receipt delivery: a receipt landed {mins} ({basis}){run_note}, "
f"not gating"
)
elif status == "failing":
where = f" via {chan}" if chan else ""
# The attempt age, NOT the success age, which is null here and must stay
# null: nothing was delivered. Printing it is the whole point of the
# separate field, because a refusal minutes old is a transient the next
# tick may clear and one weeks old is an outage nobody noticed.
att = d.get("last_attempt_age_seconds")
when = (
f" {att / 60:.0f} min ago"
if isinstance(att, (int, float))
else " at an unknown time"
)
print(
f"INFO receipt delivery: the newest send FAILED{where}{when} ({basis})"
f"{run_note}. This is positive evidence of a broken send path, but it is "
f"live box state, so it is reported and not gated"
)
elif status == "stale":
print(
f"INFO receipt delivery: newest receipt is older than the window "
f"({basis}){run_note}. Not evidence of health and not evidence of an "
f"outage either, not gating"
)
else:
print(
f"INFO receipt delivery: no datable delivery found ({basis}){run_note}. "
f"A shop that sold nothing produces exactly this, so it reads as an "
f"absence of sales rather than a fault, not gating"
)
# CLAIM: the x402 daily cap survives a restart.
#
# The gate rebuilds its spend ledger and its redeemed nonces from the earnings log at
# boot, because the unit is Restart=always and a counter living only in process memory
# would hand every payer a fresh full allowance on every restart. That property was
# asserted in the write-up and its only evidence was a line on the node's stderr, which
# is readable by the operator and by nobody else.
#
# THREE OUTCOMES, NOT TWO, following the same reasoning as the RPC corroboration in
# payment-watch. A coherent ledger block passes. An incoherent one fails. A block that is
# ABSENT means the node is running a build older than this check, which is a true and
# useful thing to report and is NOT the same statement as "the cap is broken", so it
# prints PENDING and does not gate. Collapsing "not deployed yet" into a red would make
# the red mean two different things, and a signal that means two things means neither.
ledger_gates = False
led = (health or {}).get("ledger")
if led is None:
print(
"PEND x402 cap-restart not yet observable: the deployed gate predates the "
"/health ledger block. Not gating. It becomes a live claim on the next deploy."
)
else:
# .get with no default throughout, so a field the endpoint stops sending reads as
# None and fails rather than defaulting into something that passes.
restored = led.get("restored_sales_at_startup")
nonces = led.get("redeemed_nonces")
settled = led.get("settled_atomic_units")
cap = led.get("daily_cap_atomic_units")
healthy = led.get("lock_healthy")
skipped = led.get("unparseable_lines_skipped")
shaped = all(
isinstance(x, int) for x in (restored, nonces, settled, cap, skipped)
)
ledger_gates = True
if not shaped or healthy is not True:
print(
f"FAIL x402 ledger block malformed or lock poisoned: restored={restored!r} "
f"nonces={nonces!r} settled={settled!r} cap={cap!r} healthy={healthy!r}"
)
fails += 1
elif settled > 0 and restored == 0:
# The one internally contradictory state: the node has settled sales in memory
# while claiming it restored none. Either the earnings log is not being written
# or it is not being read, and both break the cap across the next restart.
print(
f"FAIL x402 ledger inconsistent: {settled} atomic units settled but 0 sales "
"restored at startup, so the earnings log is not round-tripping"
)
fails += 1
else:
note = (
f"restored {restored} sale(s), {nonces} redeemed nonce(s), "
f"{settled} atomic units against a {cap} cap"
)
skip_note = (
f"; {skipped} unparseable line(s), restored spend is a lower bound"
if skipped
else ""
)
if restored == 0:
# Honest: zero restored is also what a node that has genuinely never sold
# anything reports, so it is not evidence of survival on its own.
print(
f"PASS x402 ledger block coherent ({note}{skip_note}). Zero restored is "
"consistent with a node that has not sold yet, so this is a shape check "
"until a sale exists."
)
else:
print(
f"PASS x402 daily cap survived the last restart ({note}{skip_note})"
)
# THE BOX'S OWN DRIFT VERDICT. deploy/box_selfcheck.py runs on the node and asserts that the
# deployed skills and tools are byte-identical to a named commit, that the network-bearing
# config fields still say mainnet, and that no funds-critical constant has drifted into state.
# That verdict is worth more than anything reachable from outside, because it can see deployed
# bytes and running services that an external prober cannot see at all -- but only if somebody
# retrieves it. This is that somebody.
#
# FOUR OUTCOMES, and the HTTP status carries the distinction rather than the body:
# 404 the deployed gate predates the /selfcheck route -> PENDING, does not gate
# 503 route present, no verdict on disk -> FAIL, the timer is not running
# 200 a verdict -> judged on `ok` and freshness
# anything else / unreachable -> FAIL
# 404 and 503 must stay distinguishable. Collapsing them would make one red mean either "we
# have not shipped this yet" or "the check silently stopped running", and those need opposite
# responses.
selfcheck_gates = False
# Bound before the try for the same reason `health` is, one screen up: the build-provenance
# report below reads it, and an unbound name there would crash the verifier itself.
sc = None
sc_url = os.environ.get(
"SHOP_SELFCHECK_URL", "https://x402.perfpilot.dev/selfcheck"
)
# An hour of slack on top of the hourly timer, so one missed tick is not an alarm while a
# stopped timer still is.
max_age = int(os.environ.get("MAX_SELFCHECK_AGE_S", "7800"))
try:
req = urllib.request.Request(
sc_url, headers={"User-Agent": "Mozilla/5.0 (verify-proof)"}
)
with urllib.request.urlopen(req, timeout=20) as r:
sc = json.loads(r.read(65536).decode("utf-8", "replace"))
selfcheck_gates = True
age = sc.get("age_seconds")
ok = sc.get("ok")
checks = sc.get("checks")
sha = sc.get("deployed_sha")
if not isinstance(age, int) or not isinstance(checks, list) or ok is None:
print(
f"FAIL box self-check malformed: age={age!r} ok={ok!r} "
f"checks={type(checks).__name__}"
)
fails += 1
elif age > max_age:
# A stale verdict is the failure this endpoint exists to make visible: the box would
# otherwise keep serving an old green answer forever with nothing to indicate the
# check had stopped running.
print(
f"FAIL box self-check is {age}s old (limit {max_age}s), so the hourly timer is "
"not running and the served verdict describes the past"
)
fails += 1
elif ok is not True:
bad = [c.get("name") for c in checks if c.get("ok") is not True]
print(f"FAIL box has DRIFTED from {sha}: {', '.join(map(str, bad))}")
fails += 1
else:
print(
f"PASS box matches {sha} on all {len(checks)} invariants "
f"(verdict {age}s old)"
)
except urllib.error.HTTPError as e:
if e.code == 404:
print(
"PEND box self-check not yet observable: the deployed gate predates the "
"/selfcheck route. Not gating. It becomes a live claim on the next deploy."
)
elif e.code == 503:
print(
"FAIL box self-check endpoint is live but has no verdict to serve, so the "
"hourly timer is not installed or not running"
)
selfcheck_gates = True
fails += 1
else:
# ANY OTHER STATUS IS THE SERVER DECLINING TO ANSWER, so it is classified by the
# same predicate every other caller uses rather than by this block's own opinion.
# Until 2026-08-19 it was not, and this was the only gating block in the file that
# never touched `transport_fails`: a 500, 502, 504 or 429 in front of the box
# counted as a claim that stopped holding, so the run exited 1, proof-check.yml
# printed "A published claim stopped holding" and refused to retry, and the alarm
# named a claim that was fine. `is_transport_error` already calls those transport
# and `test_verify_proof_transport.py` pins it; the defect was a caller
# contradicting the predicate its own file defines.
print(f"FAIL box self-check returned HTTP {e.code}")
selfcheck_gates = True
fails += 1
if is_transport_error(e):
transport_fails += 1
# NOTE THE TWO BRANCHES ABOVE THAT DELIBERATELY DO NOT CONSULT THE PREDICATE, because
# this endpoint gives 404 and 503 an application meaning the generic classifier cannot
# know. `is_transport_error` calls 503 transport, and here it is the box saying the
# route is live with no verdict on disk -- the timer stopped, which is a finding and
# must never be retried into looking healthy. 404 is the pre-deploy state and gates
# nothing. Both are intercepted before reaching the `else`, so the predicate's own
# verdict on 503 never applies at this call site. That is intentional; do not "fix" it.
except Exception as e:
# Distinct from the two above: this is transport, not a verdict. It still gates, because a
# node nobody can reach is a real problem, but the message must not read as drift -- and it
# must not be COUNTED as drift either, which is what the classification below records.
print(f"FAIL box self-check unreachable: {e}")
selfcheck_gates = True
fails += 1
if is_transport_error(e):
transport_fails += 1
# WHICH COMMIT THE BINARY ANSWERING YOU WAS BUILT FROM.
#
# The line above reports `deployed_sha`, and a reader comparing that against the repository
# believes they have checked the gate. They have not. `deployed_sha` names the commit the
# WORKSPACE deploy was generated from -- the config, skills and SOPs listed in
# deploy/deploy-targets.json. A compiled binary is deliberately not in that file map, so the
# gate can be rebuilt without moving `deployed_sha` and the workspace can be redeployed
# without rebuilding the gate. Two facts, and only one of them is about this binary.
#
# A DIFFERENCE BETWEEN THEM IS NOT AN ERROR, and this block must never be read as though it
# were. They answer different questions, they move independently by design, and today they
# legitimately differ. Nothing here compares them for agreement, and nothing here gates: the
# value is a property of a remote binary, so a gate on it would turn main red on someone
# else's deploy, which is the same reasoning the receipt and ledger blocks already carry.
#
# WHAT IS CHECKABLE IS DIFFERENT, and there are two of them. Whether /health and /selfcheck
# report the SAME commit, since one binary serves both and a disagreement means two processes
# answered. And where that commit sits in the clone the reader is holding, which is the only
# part of the gate's claim a stranger can test locally.
#
# A NON-OBJECT `gate` IS A REAL DEPLOYED SHAPE, NOT A DEFENSIVE HYPOTHETICAL, and this is the
# reason the type checks below must never be deleted as paranoia. Until the build-provenance
# deploy landed, /health served `"gate": "ok"` -- a STRING, verifiable in this repository's own
# history at the commit before it. Every box that has not been redeployed still serves it, and
# rolling the gate binary back restores it in one command with the backup already on the node.
#
# So the shape a reader is MOST likely to point this verifier at is the one that used to index
# straight into remote JSON and raise inside `main()`, which has no top-level handler. That
# killed the WHOLE run: no tally, and every PASS and FAIL above it lost, over a field that
# gates nothing. Measured four ways before the guard existed, all four fatal.
#
# ONE VERDICT, NOT TWO. An old binary and a binary answering in a shape this cannot parse are
# the same actionable fact for a reader: the provenance is unknown and nothing else is wrong.
# Giving the unparseable case its own alarming branch would make the ordinary pre-deploy state
# look like a defect, so both land on the same PENDING, and the message avoids asserting
# "predates the field", which is true of only one of them.
# Same normalisation as `health` above, for the same reason: the selfcheck block's own `.get`
# raises inside its try on a non-object body, which is reported and counted there, and leaves
# this name bound to a string or a list that nothing below could survive.
if not isinstance(sc, dict):
sc = None
raw_gate = (health or {}).get("gate")
gate_h = raw_gate.get("build_commit") if isinstance(raw_gate, dict) else None
src_h = raw_gate.get("build_commit_source") if isinstance(raw_gate, dict) else None
gate_s = (sc or {}).get("gate_build_commit")
src_s = (sc or {}).get("gate_build_commit_source")
# Only a string is a commit. Anything else present is recorded so the line can say what shape
# arrived, and is then treated exactly as absent.
odd_shapes = [
f"gate={type(raw_gate).__name__}"