Synthetic Data Pipeline Toy

This repo is a companion sample for the article LLM Output은 아직 Product Data가 아니다.

It does not use production data. It does not call a live LLM API by default. The goal is to make the pipeline shape executable:

LLM output -> candidate
candidate + source input -> review packet
review packet -> decision data
decision -> repair / reject / pass
pass + gate checks -> product data

What This Demonstrates

• A charge candidate can be schema-valid but product-wrong.
• source_input includes selected and contrastive senses so evaluators do less guessing.
• review_packets.jsonl keeps generated payload, source input, precheck metadata, and empty review fields together.
• review_decisions.jsonl turns comments into routing data: pass / repair / reject.
• publish.mjs blocks anything without an explicit passing decision and compatible checks.
• metrics.mjs separates false_accept_rate from false_publish_rate.

Run

Node 20.11+ is enough. The scripts are dependency-free ESM files and do not call a live API by default.

pnpm generate:candidates
pnpm precheck:candidates
pnpm build:review-packets
pnpm draft:decisions
pnpm repair
pnpm audit:sample
pnpm publish:bundle
pnpm metrics

Or run the whole flow:

pnpm pipeline:run

Expected summary:

Total candidates: 30
Pass: 27
Repair: 2
Reject: 1
Published: 27
False publish sample audit: 0/5

The generated files under artifacts/ are intentionally checked in so readers can inspect the handoff artifacts without running the pipeline first. Re-running pnpm pipeline:run regenerates them.

File Layout

synthetic-data-pipeline-toy/
  AGENTS.md
  README.md
  data/
    fake_dictionary.jsonl
  artifacts/
    candidates.jsonl
    precheck_results.jsonl
    review_packets.jsonl
    review_decisions.jsonl
    repaired_candidates.jsonl
    audit_sample.jsonl
    published_bundle.json
    metrics.json
  src/
    lib.mjs
    generate-candidates.mjs
    precheck-candidates.mjs
    build-review-packets.mjs
    draft-decisions.mjs
    repair.mjs
    audit-sample.mjs
    publish.mjs
    metrics.mjs

Where An LLM API Fits

This toy uses deterministic mocks so the repo is runnable without keys. In a real pipeline, the generation, evaluator, repair, and decision-draft steps can call an LLM API.

The apply gate should remain deterministic. A model can draft a decision, but a missing decision, failed semantic grounding, unresolved human review route, unsafe content, duplicate, or runtime mismatch should block publish.

Where A Coding Agent Fits

A repo-aware coding agent is not the judge. It is closer to an operator.

For example, when sense_grounding.matched_contrastive_sense is added, the agent can update the review packet builder, decision drafter, repair queue, publish gate, metrics output, and README examples together.

Agent-authored changes still need tests, diff review, and sample audit.

Public Data Boundary

This repo contains only fake dictionary inputs and toy artifacts. It is not a trimmed export of a production product and does not include private source data.