Skip to content

feature: golden-dataset COBOL/JS serverless parity harness - #268

Open
hannahhkyme wants to merge 2 commits into
mainfrom
devin/1787681560-golden-parity-harness
Open

hannahhkyme wants to merge 2 commits into
mainfrom
devin/1787681560-golden-parity-harness

Conversation

@hannahhkyme

@hannahhkyme hannahhkyme commented Aug 25, 2026

Copy link
Copy Markdown

Summary

Adds a reproducible parity harness that proves a JS serverless conversion of the portfolio programs behaves like the COBOL, per case, including error paths — plus the reference handlers themselves, since no JS conversion existed on main to compare against (golden/README.md documents the whole pipeline).

49 cases, all passing. 39 expected results are captured from real COBOL runs (GnuCOBOL 3.1.2.0, BDB indexed handler) and committed under golden/cobol-run/; the 10 PORTTRAN cases are derived: true because PORTTRAN.cbl can't compile here (it copies a PORTREC copybook absent from src/copybook and references PORT-TOTAL-UNITS/PORT-TOTAL-COST, defined nowhere in the repo). Each derived golden carries a derivation field quoting the paragraphs it came from, and the report marks them DERIVED EXPECTED.

Flow, wired into npm test so divergence breaks the build:

golden/cases/*.json ──golden:inputs──> golden/inputs/**/{seed,input}.dat
                                            │                    │
                       golden:capture ──────┘                    └──> handlers (decode same bytes)
                                            ▼                                    │
                        golden/cobol-run/**/{stdout,dump,audit}                  │
                                            │ golden:expected                    ▼
                        golden/expected/**.json ────── diffCanonical ──── canonical actual

Both sides reduce to one canonical form (serverless/src/parity/canonical.js): padding trimmed, COMP-3 decoded to exact decimal strings compared by value not bytes (BigInt throughout — no float money anywhere), clock-stamped fields tokenized to @RUNDATE/@RUNSTAMP. The JS side is fed the same fixed-width files the COBOL was fed rather than the case JSON, so the two sides can't differ in what they were asked to process — only in how. Failures name the exact path: finalState[1].createDate: expected(COBOL)="@RUNDATE" actual(JS)="BROKEN".

Layouts are transcribed from PORTFLIO.cpy (148 bytes, 18-byte PORT-KEY) and TRNREC.cpy; the simplified layout in test-data-specs.md is used only for its list of canonical cases. Determinism comes from a fixed randomSeed (123456789) in golden/config/golden-config.json feeding a seeded LCG, mirroring TSTGEN00's RANDSEED.

The goldens encode what the programs do, not what they claim

Several captured behaviors contradict the source comments and the docs; the handlers reproduce them deliberately (all listed in golden/README.md):

  • PORTVALD rejects every portfolio ID, including a well-formed PORT0001: after the PORT prefix check it moves the 4 digits into an X(10) work field and applies the NUMERIC class test, which the 6 trailing spaces can never pass. Implemented as that same two-step so the quirk is visible rather than hardcoded.
  • PORTUPDT action V inherits MOVE <X(50)> TO S9(13)V99 semantics: ...2500075 becomes 2500075.00, not 25000.75 (shared cobol-move.js, which throws on non-digit input rather than guessing GnuCOBOL's junk handling).
  • PORTTRAN never performs 2200-UPDATE-POSITIONS. Buy/sell/transfer/fee, the insufficient-units check and the AUDPROC audit trail are unreachable dead code — the program reads, validates, counts, and never rewrites the file it opens I-O. So the requested "insufficient units on sell" case (TRN-003) passes validation and changes nothing, and the JS handler implements no position keeping: doing so would make the harness green against behavior the COBOL doesn't have.
  • PORTVALD amount validation via the same MOVE can never fall outside the documented range; account validation runs against the whole X(50), so trailing spaces fail where 50-digit zero-padding passes; PORTDEL doesn't validate reason codes and leaves AUD-FILLER at INITIALIZEd low-values (27 NUL bytes in the golden audit record); PORTUPDT doesn't restamp PORT-LAST-MAINT.

Notes for review

  • src/ is untouched. Capture needs three patches (fixed-format comment columns, a duplicate-copybook name clash in PORTADD, an ACCEPT ... FROM TIME STAMP GnuCOBOL rejects) applied to copies under build/src/; see golden/patches/README.md.
  • golden/cobol/GLDLOAD.cbl loads seeds via ACCESS MODE IS RANDOM — with SEQUENTIAL any out-of-key-order seed silently failed with status 21, which is exactly what the RED-001/RED-004 browse-order cases need. The capture script now hard-fails if loaded/dumped counts don't match the manifest.
  • npm run parity and npm test need only Node 20+ and the committed fixtures; GnuCOBOL is required only to re-capture (npm run golden:capture).
  • Report is written to golden/reports/parity-report.txt in TSTVAL00's layout (132-col banner, fixed-width detail lines, total/passed/failed/success-rate); it's gitignored as a build artifact.

Devin-Org: engineering

Link to Devin session: https://app.devin.ai/sessions/434efed784834b52aa12d9d7da4d8e4f
Requested by: @hannahhkyme


Devin Review

Status Commit
⚪ Not started

Run Devin Review

💡 Connect your GitHub account to enable automatic code reviews.

Devin Review (Staging)
Open in Devin Review

Add captured COBOL evidence, fixed-width fixtures, and JavaScript parity handlers with unit tests.

Co-Authored-By: hannah.kyme <hannahhkyme@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: hannah.kyme <hannahhkyme@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant