Skip to content

README.md

Latest commit

 

History

History
77 lines (56 loc) · 2.45 KB

README.md

File metadata and controls

77 lines (56 loc) · 2.45 KB
Raw

Research Workflow

This directory is the durable experiment ledger.

Structure

  • decision-log.md: adopted, weakened, killed, revived ideas
  • falsified-hypotheses.md: queryable ledger of rejected hypotheses (phase id, magnitude, scope of rejection)
  • prompt_bank: versioned prompt sets and answer keys
  • experiments/<year>/...: one dated file per decision-worthy experiment family

Use the current decision log, experiment registry, and claim register as the live sources of truth. Raw artifact logs may still contain machine-local paths captured at execution time; those are provenance fields, not reproduction instructions.

Some older checked artifacts were path-normalized for OSS release in commit b15213c: absolute local paths and private model roots were replaced by placeholders such as <repo-root> and <model:...>. Treat those fields as privacy-preserving provenance metadata, not byte-for-byte raw terminal logs.

Experiment Note Template

Each experiment note should contain these sections in this order:

  1. Preregistration
  2. Execution
  3. Result
  4. Interpretation
  5. Links

Put the preregistration in the file before the run happens.

Pre-Registration Minimum

Every decision-worthy experiment should state:

  • hypothesis
  • track (A or B)
  • primary metric
  • secondary metrics
  • unit of analysis
  • comparison
  • acceptance band
  • rejection band
  • what counts as inconclusive

When an experiment finishes, explicitly state the preregistration outcome:

  • Accepted
  • Rejected
  • Inconclusive
  • Accepted with caveat only when the note names the exact caveat

If you want to use softer narrative language like partial reproduction, say that in Interpretation, not instead of the preregistration outcome.

Update Rules

  • one file per experiment family, not one file per tiny run
  • append new runs chronologically
  • link raw artifacts instead of pasting giant dumps
  • update decision-log.md when a hypothesis changes status

Long Local Runs

For long semantic runs on this laptop, prefer cooperative termination over force-killing processes:

  • use --checkpoint-path <json> to persist partial results after each chunk
  • use --stop-file <path> to request clean exit after the current item or chunk

This is an operational convenience only. It does not make a run timing-safe for Track B.