[vllm] Repair stale Meta Llama GCS caches flattened into Mistral layout

[ahmeda14960]

🤖

Describe the bug
Some historical cached meta-llama model prefixes in GCS were downloaded with a flattened layout: files that should have lived under original/ were written at the model root. When a cached Llama prefix contains root params.json and root consolidated*.pth, native TPU vLLM can parse the checkpoint as Mistral and resolve MistralForCausalLM instead of LlamaForCausalLM.

To Reproduce

1. Inspect any broken prefix below and confirm it has root params.json and root consolidated*.pth, with no original/params.json.
2. Launch native vllm serve against that GCS prefix on TPU.
3. Check startup logs for Resolved architecture: MistralForCausalLM.
4. Compare with a repaired prefix, where params.json and consolidated*.pth live under original/ and vLLM resolves LlamaForCausalLM.

Expected behavior
Meta Llama caches should preserve original/params.json and original/consolidated*.pth, and native TPU vLLM should resolve LlamaForCausalLM.

Additional context
Current download code in lib/marin/src/marin/datakit/download/huggingface.py:99 preserves nested HF paths, and experiments/models.py:35 writes stable model prefixes. The issue is stale historical caches already written under those stable prefixes.

Why delete and re-download is required:

• re-running the current downloader into the same prefix does not remove stale root params.json or stale root consolidated*.pth
• the stable output path means malformed objects remain mixed with the corrected layout unless the prefix is cleaned first
• deleting the whole prefix is safer than partial cleanup and also clears executor bookkeeping such as .executor_status

Affected broken prefixes found in the accessible Marin buckets:

• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B--d04e592/
• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B--main/
• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-central1/models/meta-llama--Llama-3-2-1B--4e20de3/
• gs://marin-us-central1/models/meta-llama--Llama-3-2-1B--main/
• gs://marin-us-central1/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/
• gs://marin-us-central2/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-central2/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/
• gs://marin-us-east1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-east5/models/meta-llama--Llama-3-2-1B--main/

Already repaired / clean:

• gs://marin-us-east5/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-east5/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/

No meta-llama model prefixes were found in gs://marin-eu-west4/models/ or gs://marin-us-west4/models/.

Validated repair: after deleting and re-downloading gs://marin-us-east5/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/, native TPU vLLM resolved LlamaForCausalLM and a 100-request smoke test succeeded.

[ahmeda14960]

🤖 Scan update from April 1, 2026.

I scanned the accessible Marin model buckets for models/meta-llama* prefixes and marked a prefix as broken if it still has root-level params.json or root-level consolidated*.pth.

Broken prefixes:

• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B--d04e592/
• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B--main/
• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-central1/models/meta-llama--Llama-3-2-1B--4e20de3/
• gs://marin-us-central1/models/meta-llama--Llama-3-2-1B--main/
• gs://marin-us-central1/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/
• gs://marin-us-central2/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-central2/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/
• gs://marin-us-east1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-east5/models/meta-llama--Llama-3-2-1B--main/

Already clean:

• gs://marin-us-east5/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-east5/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/

No meta-llama model prefixes were found in:

• gs://marin-eu-west4/models/
• gs://marin-us-west4/models/

Operational recommendation stays the same: delete each broken prefix completely, then re-download with the current experiments/models.py pipeline. The successful us-east5 8B-Instruct rerun is the validation that this repair works.

[ahmeda14960]

🤖 Addendum (from claude-signal session): Sliding window attention makes this worse than a cosmetic mismatch

How vLLM turns Llama into Mistral

The mismatch happens because of how the pinned vllm-tpu==0.13.2.post6 auto-detects config format. In vllm/transformers_utils/config.py, config_format="auto" checks for params.json before config.json:

1. If root params.json exists → parse as Mistral format
2. Only then fall back to config.json → parse as HF format

The historical Marin download bug (pre-3fdd982b2) flattened original/params.json to root params.json in GCS caches. So cached artifacts have a hybridized layout with both HF files and Meta-original files at the root.

Once vLLM picks Mistral parsing, the Mistral config parser in vllm/transformers_utils/configs/mistral.py unconditionally rewrites the architecture for dense (non-MoE) models:

config_dict["architectures"] = ["MistralForCausalLM"]

This overwrites the original LlamaForCausalLM from config.json, regardless of what the model actually is.

Direct HF repo IDs don't trigger this because the upstream HF repos keep Meta-original files under original/ — there is no root params.json, so vLLM stays on HF parsing and preserves LlamaForCausalLM.

Why this silently corrupts long-context inference

Beyond the architecture class name, MistralForCausalLM and LlamaForCausalLM have a critical behavioral difference in the HuggingFace transformers implementations.

Sliding window attention:

• MistralConfig defaults to sliding_window=4096. MistralModel.forward() selects create_sliding_window_causal_mask when sliding_window is not None, and MistralAttention.forward() passes sliding_window=getattr(self.config, "sliding_window", None) to the attention kernel.
• LlamaConfig has no sliding_window parameter. LlamaModel.forward() always uses create_causal_mask (full causal attention). LlamaAttention.forward() does not pass a sliding_window kwarg.

Impact: When a Llama model is misresolved as Mistral, attention is silently masked beyond 4096 tokens. The model still runs and produces coherent-looking text, but any sequence longer than 4096 tokens gets corrupted attention patterns. This is especially dangerous because:

1. Short evals (e.g. MATH-500, MMLU) will not catch the regression
2. Long-context tasks (summarization, RAG, multi-turn chat) silently degrade
3. There is no error or warning — the sliding window mask is applied as if it were correct

Secondary differences (less likely to bite for standard Llama-3 models):

• Mistral hardcodes bias=False on all attention Q/K/V/O projections; Llama makes this configurable via config.attention_bias
• Mistral hardcodes bias=False on MLP projections; Llama uses config.mlp_bias
• If a model used non-default bias settings, loading as Mistral would silently drop those weight tensors

This further raises the priority of repairing the broken prefixes listed in the issue — any RL or long-context inference pipeline using these caches may be producing silently degraded results.

[ahmeda14960]

🤖 Repair status update (from claude-signal session)

Deleted (all broken prefixes ≤ 8B)

8B models:

• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B--d04e592/
• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B--main/
• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-central2/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/
• gs://marin-us-east1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/

1B models:

• gs://marin-us-central1/models/meta-llama--Llama-3-2-1B--4e20de3/
• gs://marin-us-central1/models/meta-llama--Llama-3-2-1B--main/
• gs://marin-us-east5/models/meta-llama--Llama-3-2-1B--main/

Re-downloaded (8B Instruct via Iris CPU jobs)

All three completed successfully with correct original/ subdirectory layout:

• gs://marin-us-central1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/ — Iris job /ahmed/redownload-llama31-8b-instruct-us-central1 succeeded
• gs://marin-us-central2/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/ — Iris job /ahmed/redownload-llama31-8b-instruct-us-central2 succeeded
• gs://marin-us-east1/models/meta-llama--Llama-3-1-8B-Instruct--0e9e39f/ — Iris job /ahmed/redownload-llama31-8b-instruct-us-east1 succeeded

Confirmed original/consolidated.00.pth and original/params.json are nested correctly (no root-level flattening).

Verified safe before deletion

Downloaded the latest Iris controller state checkpoint and queried all running/pending jobs — none reference any of the affected model prefixes. The meta-llama references in recent logs are all tokenizer metadata mismatch warnings from Levanter training jobs, not vLLM inference.

Still broken (70B, not yet repaired)

• gs://marin-us-central1/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/
• gs://marin-us-central2/models/meta-llama--Llama-3-3-70B-Instruct--6f6073b/

Not re-downloaded yet (base/non-Instruct models)

• meta-llama--Llama-3-1-8B--d04e592 (us-central1)
• meta-llama--Llama-3-1-8B--main (us-central1)
• meta-llama--Llama-3-2-1B--4e20de3 (us-central1)
• meta-llama--Llama-3-2-1B--main (us-central1, us-east5)

These can be re-downloaded on demand using the same pattern:

MARIN_PREFIX=gs://marin-{region} uv run iris --config lib/iris/examples/marin.yaml job run \
  --cpu 2 --memory 8GB --disk 32GB --region {region} --no-wait \
  -e MARIN_PREFIX gs://marin-{region} \
  -- python -m experiments.redownload_llama_8b_instruct

[ahmeda14960]

🤖 Final repair status (from claude-signal session) — all prefixes repaired

All 12 broken prefixes from the original issue have been deleted and re-downloaded with correct original/ subdirectory layout. No root-level params.json flattening in any repaired prefix.

8B Instruct (Llama-3.1-8B-Instruct, 0e9e39f)

Region	Status
us-central1	✅ re-downloaded
us-central2	✅ re-downloaded
us-east1	✅ re-downloaded
us-east5	was never broken

8B Base (Llama-3.1-8B, d04e592)

Region	Status
us-central1	✅ re-downloaded

8B Base (Llama-3.1-8B, main)

Region	Status
us-central1	deleted (not re-downloaded — pinned revision d04e592 is available)

1B (Llama-3.2-1B, 4e20de3)

Region	Status
us-central1	✅ re-downloaded

1B (Llama-3.2-1B, main)

Region	Status
us-central1	deleted (not re-downloaded — pinned revision 4e20de3 is available)
us-east5	deleted (not re-downloaded — pinned revision 4e20de3 is available)

70B Instruct (Llama-3.3-70B-Instruct, 6f6073b)

Region	Status
us-central1	✅ re-downloaded
us-central2	✅ re-downloaded
us-east5	was never broken

All Iris download jobs succeeded. The main revision prefixes were not re-downloaded since the pinned revisions are the ones referenced by experiments/models.py.

[ahmeda14960]

Finished, all models were re-downloaded. Will be future proofed when we move to vLLM fork