[marin] _discover_files treats artifact.json sidecar as JSONL data

[Helw150]

TL;DR

_discover_files matches .json and so sweeps the executor sidecar
artifact.json in as a data file; load_jsonl then parses it as
JSONL. Production symptom today: transforms whose fn returns None
write the literal null to artifact.json, and normalize crashes
with AttributeError: 'NoneType' object has no attribute 'get'. The
deeper bug is type mis-detection — a single-document JSON file is
treated as a JSONL stream — so other sidecar shapes fail differently
(see table below). Regressed by #5732 which renamed .artifact
(dotfile, skipped) → artifact.json (not skipped). Fix: skip marin
sidecars by basename in _discover_files.

Failure modes (all same root cause)

artifact.json payload	Outcome
null — what Artifact.save(None, ...) writes	AttributeError in has_text (verified in production)
Multi-line pretty-printed JSON	msgspec.DecodeError: Input data was truncated (synthetic)
Single-line JSON dict with a text field	Silent phantom record in output (synthetic)

To Reproduce

import os, tempfile, json
import pyarrow as pa, pyarrow.parquet as pq
from marin.datakit.normalize import normalize_to_parquet

with tempfile.TemporaryDirectory() as d:
    pq.write_table(
        pa.table({"id": ["a"], "text": ["hello"]}),
        os.path.join(d, "data-00000-of-00001.parquet"),
    )
    with open(os.path.join(d, "artifact.json"), "w") as f:
        f.write("null")  # or '{\n"x":1\n}' or json.dumps({"text": "phantom"})
    normalize_to_parquet(input_path=d, output_path=os.path.join(d, "out"))

Expected behavior

_discover_files should not surface marin executor sidecars
(artifact.json, provenance.json) as data files. Skipping by exact
basename — the same way dotfiles are skipped today — is the natural
fix.

Additional context

• Discovery / extension filter: lib/marin/src/marin/datakit/normalize.py:198-208
• Artifact filename constant: lib/marin/src/marin/execution/artifact.py:18
• Same class of bug in decon._discover_eval_files
  (lib/marin/src/marin/datakit/decon.py:179-197): docstring claims
  provenance.json is skipped, code does not.
• Broader cleanup option: drop .json from
  zephyr.readers.SUPPORTED_EXTENSIONS so single-document JSON can't
  masquerade as JSONL.

[Helw150]

🤖 Confirmed the reproducer locally:

• With the 4-byte artifact.json (literal null) → AttributeError: 'NoneType' object has no attribute 'get' at normalize.py:349 after 3 shard retries.
• Same script without the sidecar → normalize completes cleanly with 1 partition.

So the parquet data path is fine; artifact.json getting swept into _discover_files is the sole trigger.

[claude]

🤖 # Research

PR #5732 renamed the executor artifact sidecar from .artifact to artifact.json. _discover_files only filters by extension and dotfile prefix, so artifact.json now slips through as input. When the upstream fn returns None, Artifact.save(None, ...) writes literal null, which load_jsonl decodes to None, and has_text(None) crashes. decon._discover_eval_files has the same gap for provenance.json and artifact.json.

Relevant code

• normalize.py#L197-L210 — extension+dotfile filter, missing sidecar filter
• decon.py#L186-L197 — same gap
• artifact.py#L18 — artifact.json filename constant
• validation_utils.py#L49-L53 — writes provenance.json

Proposed Fix

Add a module-level constant listing marin sidecar basenames (artifact.json, provenance.json) in marin.datakit.normalize and skip them in both _discover_files and _discover_eval_files. Add regression tests that drop artifact.json containing null next to real data.

PR: #5865

Plan:

• Confirm issue actionable
• Duplicate-work preflight (no overlapping PRs/issues)
• Post Research + Proposed Fix comment
• Branch agent/20260520-fix-5864
• Add regression tests
• Fix _discover_files and _discover_eval_files
• pre-commit and pytest
• Verify CI on [marin] datakit discovery: skip artifact.json/provenance.json sidecars #5865

[Helw150]

🤖 Silent-injection reproducer (the dangerous case — no error, no warning):

import os, tempfile, json
import pyarrow as pa, pyarrow.parquet as pq
from marin.datakit.normalize import normalize_to_parquet

with tempfile.TemporaryDirectory() as d:
    pq.write_table(
        pa.table({"id": ["real_a", "real_b"], "text": ["hello", "world"]}),
        os.path.join(d, "data-00000-of-00001.parquet"),
    )
    # Sidecar with a `text` field survives has_text and enters the data stream.
    with open(os.path.join(d, "artifact.json"), "w") as f:
        f.write(json.dumps({"id": "sidecar_phantom", "text": "I AM NOT DATA"}))

    normalize_to_parquet(input_path=d, output_path=os.path.join(d, "out"))

    rows = pq.read_table(
        os.path.join(d, "out", "outputs", "main", "part-00000-of-00001.parquet")
    ).to_pylist()
    print(len(rows), "rows")
    for r in rows:
        print(" ", r)

Output (2 input rows → 3 normalized rows):

3 rows
  {'text': 'I AM NOT DATA', 'id': 'a192d2eb5fffd8d12454b075fd34f58e', 'source_id': 'sidecar_phantom', 'partition_id': 0}
  {'text': 'hello',         'id': 'b5e9c1ad071b3e7fc779cfaa5e523818', 'source_id': 'real_a',         'partition_id': 0}
  {'text': 'world',          'id': 'fa0d38a9b38280d0891e4985bdb2583e', 'source_id': 'real_b',         'partition_id': 0}

Coordinator counters confirm: zephyr/records_in: 3, normalize/unique_records_out: 3. The sidecar is admitted as a first-class record.

In production this is what Artifact.save(NormalizedData(...), ...) writes — a real artifact.json payload would inject {"main_output_dir": "...", "num_partitions": N, "counters": {...}} as a tokenized "document" in downstream pretraining.

[ravwojdyla]

That is my bad, I should have caught it during review, we definitely what it to be .artifact.json