Fix duplicate token append in Metal generator

[natan4903]

Metal generate.c appends each sampled token twice

Summary

In the Metal backend, gptoss_context_sample() already appends every sampled token to the Context. The generate example then calls gptoss_context_append_tokens() with the same token. Consequently, each token is printed once but is present twice in the internal sequence used to predict later tokens.

This report applies to the current upstream HEAD verified on 2026-08-27:

7b583341fe16729127f6d5b94a7b09ccae97e1a1

Relevant code

Inside gptoss_context_sample(), the sampled token is written to context->token_buffer, after which the function advances the context:

context->num_tokens += 1;
context->num_kv_tokens = context->num_tokens;

The generated tokens are then copied from that same internal token buffer to tokens_out.

After gptoss_context_sample() returns, gpt_oss/metal/source/generate.c appends predicted_token again:

status = gptoss_context_append_tokens(context, 1, &predicted_token);

Since context->num_tokens == context->num_kv_tokens at this point, gptoss_context_append_tokens() takes its ordinary append path and writes a second copy of the token.

Expected behavior

After generating and printing tokens t1, t2, ..., the internal autoregressive context should be:

prompt, t1, t2, ...

Actual behavior

The internal context becomes:

prompt, t1, t1, t2, t2, ...

while stdout still contains only:

t1, t2, ...

The first generated token is unaffected, but later tokens are conditioned on the duplicated sequence. In longer continuations this can produce unnecessary repetition and confused self-corrections.

Minimal state-based reproduction

The duplication can be verified without interpreting generated text. Around the existing calls in generate.c, record context->num_tokens:

const uint32_t before = context->num_tokens;

gptoss_context_sample(
    context, options.temperature, 0, 1,
    &predicted_token, &num_predicted_tokens);

assert(context->num_tokens == before + 1);

gptoss_context_append_tokens(context, 1, &predicted_token);

assert(context->num_tokens == before + 2);

The first assertion confirms that sampling already advanced the visible context. The second confirms that the example added the same token a second time.

Minimal fix

Remove the explicit gptoss_context_append_tokens() call from the generation loop. gptoss_context_sample() already advances the context and supports generating multiple tokens in one call.

The accompanying patch also clarifies the side effect in the public function comment, because the current wording does not state that sampling mutates the Context.

Verification

We kept the original executable for reproduction and built a second generator that differs only by omitting the extra append. The clean generator:

• preserves the same first-token choices;
• produces ordinary autoregressive context advancement;
• substantially reduces repetitive or confused long continuations;
• does not remove the separate instruction-sensitivity phenomenon that motivated the investigation.

The issue was found while tracing how the first generated token enters the KV-cached incremental path used to predict the second token.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c40b70dd2b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Stop before sampling beyond the context buffer

When generation does not encounter the return token and --max-tokens is omitted or exceeds the remaining context capacity, removing the append also removes the only capacity check: after the last valid slot is filled, this loop calls gptoss_context_sample() again, which writes the sampled token at context->num_tokens without checking context->max_tokens, one element past token_buffer. The removed append previously returned gptoss_status_context_overflow as soon as the context became full; replace that accidental guard with an explicit remaining-capacity check before the next sample.

Useful? React with 👍 / 👎.