Cancellation semantics unclear/inconsistent in callback streaming: late `onMessage` / terminal callbacks after `cancelProcess()` / `close()` + unstable cancellation signaling

[ishizuki-tech]

I’m seeing inconsistent cancellation behavior when using callback streaming via:

• Conversation.sendMessageAsync(text, callback: MessageCallback)

Two problems show up:

1. cancellation is difficult to classify robustly, and
2. callbacks can still arrive after cancel/close, which can corrupt client state across teardown or subsequent runs.

Summary

When I attempt to stop an in-flight streaming request (or clean up aggressively due to lifecycle / watchdog recovery):

• Cancellation may surface through onError(throwable) but message strings vary (casing/wording), making it hard to machine-detect cancellation reliably.

• I sometimes receive callbacks after I believe the request is canceled/closed:

  • onMessage(...) after cancelProcess() and/or close()
  • terminal callbacks that arrive late and race with teardown or the next run

Late events can violate common client invariants (e.g., “no UI/state updates after teardown”, “exactly one terminal outcome per run”, “next run must not be affected by previous callbacks”).

What I observe (intermittent)

• Unstable cancellation signaling: cancellation appears via onError(Throwable), but the message strings vary (casing/wording), making cancellation hard to machine-detect without parsing strings.

• Late callbacks after cancel/close:

  • onMessage(...) arrives after cancelProcess() and/or close()
  • terminal callbacks can arrive late and race with teardown or the next run

• Cross-run interference risk: callbacks from a previous run can arrive after a new run begins, which can corrupt state machines or UI.

Expected behavior (contract)

Either of these would be workable if clearly documented:

Option A (strong contract)

• cancelProcess() results in a deterministic terminal callback (usually onError with a well-defined cancellation type/code), and then:

  • no further callbacks (onMessage, onDone, onError) are delivered for that request.

Option B (explicit weak contract)

• Late callbacks may occur, but the API documents:

  • ordering/ack semantics (“cancel requested” vs “cancel acknowledged”), and
  • a reliable per-request/run mechanism for clients to ignore old callbacks.

In either case, cancellation should be reliably machine-detectable (not dependent on message-string parsing).

High-level correlations

• Cancel/close while onMessage(...) is actively streaming
• Owner teardown is aggressive (lifecycle stop/destroy)
• A watchdog/timeout cancels a run and a new run starts soon after

Requests to maintainers

1. Document cancellation semantics clearly:

   • request-level vs engine-level cancellation
   • best-effort vs guaranteed terminal behavior
   • recommended ordering between cancelProcess() and close()

2. Stabilize cancellation signaling:

   • stable cancellation type/code/exception (vs message strings)
   • or a dedicated cancellation callback / isCancellation flag

3. Provide a robust disambiguation mechanism:

   • per-request/run identifier in callbacks, or
   • cancellation acknowledgement boundary, or
   • a guarantee that no callbacks occur after “cancel acknowledged”

Questions

1. Are callbacks allowed to arrive after cancelProcess() / close()? If yes, what ordering guarantees exist?
2. Is cancellation expected to be reflected as onError(...)? Is onDone() ever valid after cancellation?
3. Is there a recommended pattern for “cancel then close” (or vice-versa) to minimize late callbacks?

Temporary client-side mitigation

Until the contract is clarified/fixed, I’m forced to implement defensive logic:

• Track per-run runId + finalized/canceled flag and ignore callbacks after finalization
• String-parsing workaround to detect cancellation
• Watchdog timeouts to avoid stuck state during lifecycle transitions

Environment

• LiteRT‑LM: com.google.ai.edge.litertlm:litertlm-android:0.9.0-alpha04

  • Version correlation: observed more frequently with com.google.ai.edge.litertlm:litertlm-android:0.8.0.

• Android / device: Google Pixel 9a (API 36)

• ABI: arm64-v8a

• Model / backend / runtime config: same as Issue 1

• Build environment: AGP 9.0.0, Kotlin 2.3.10 (Compose BOM 2026.02.00)

[dpknag]

Hi @ishizuki-tech,

Thanks for taking the time to write up this second issue and explaining the cancellation behavior so clearly.

I have logged this in our internal tracker so the engineering team can investigate the delayed callbacks and the cancellation behavior. If the team has any updates or follow-up questions during their investigation, we will post them right here.

Thanks again for your continued help!

[dpknag]

Hi @ishizuki-tech,

The team is reviewing this issue. Thank you so much for your patience!

Could you please share a concrete code snippet showing exactly how you sequence cancelProcess() / close() and how your state machine processes the MessageCallback?

Also, could you clarify exactly how your "watchdog" mechanism is implemented and how it interacts with the cancellation?

Thanks again for understanding!

[ishizuki-tech]

Issue #448 — Detailed Reply

Thank you for the follow-up, and thank you again for reviewing this issue.
I built a small dedicated app specifically for this bug so I could remove as many unrelated moving parts as possible.

Short answer

• The watchdog is not required for this bug.
• I can reproduce the native crash with a simpler path that only does:
  1. initialize the engine,
  2. start streaming with sendMessageAsync(...),
  3. wait until the first onMessage(...),
  4. schedule a fixed external delay (1216 ms),
  5. call conversation.close() from a scheduler thread while streaming is still active.
• In the crashing runs, watchdog_timeout_ms is null.
• I reproduced this on both GPU and CPU backends.

---

Environment

• LiteRT‑LM: com.google.ai.edge.litertlm:litertlm-android:0.9.0-alpha06
• Android / device: Google Pixel 9a (API 36)
• ABI: arm64-v8a
• Build environment: AGP 9.1.0, Kotlin 2.3.10, Compose BOM 2026.03.00
• Model: gemma3-1b-it-int4.task Even with a smaller model, I see a same issue.
• Model packaging in repro app: bundled in app assets and staged to app-internal storage before initialization

---

What in the new repro app

To make the issue easier to reason about, I removed the download model module and replaced it with a very simple setup:

• the model is bundled directly in the APK under app assets
• the app stages that model into app-internal storage
• the engine is initialized from the staged local file
• the repro app focuses on scenario-driven conversation lifetime experiments
• each run writes:
  • a structured JSON artifact (run_XXXX.json)
  • a compact critical evidence log (run_XXXX.critical.jsonl)

That let me isolate the failure to the conversation close timing during active streaming, instead of any model provisioning behavior.

---

Concrete close() sequencing used in the minimal repro

Below is the simplified logic used by the repro app for the crashing path.

1) Start streaming from the main thread

val promptMessage = Message.of(config.promptBody)
conversation.sendMessageAsync(promptMessage, callback)

2) In onMessage(...), schedule one delayed external close after the first streamed token/chunk

override fun onMessage(message: Message) {
    val now = SystemClock.elapsedRealtime()
    active.firstMessageAtMs.compareAndSet(0L, now)
    active.lastMessageAtMs.set(now)

    val messageIndex = active.messageCount.incrementAndGet()

    if (
        spec.stopOrigin == StopOrigin.EXTERNAL_DELAY_AFTER_FIRST_MESSAGE &&
        spec.externalStopDelayMs != null &&
        active.stopJobScheduled.compareAndSet(false, true)
    ) {
        active.stopJob = scope.launch(schedulerDispatcher) {
            delay(spec.externalStopDelayMs)
            issueStop(active, reason = "external_delay_after_first_message")
        }
    }
}

3) issueStop(...) calls conversation.close() from the scheduler thread

private fun issueStop(
    active: ActiveRun,
    reason: String,
) {
    if (!active.stopIssued.compareAndSet(false, true)) {
        return
    }

    active.stopIssuedAtMs.set(SystemClock.elapsedRealtime())

    when (active.spec.stopAction) {
        StopAction.CLOSE -> {
            active.closeAttempted.set(true)
            runCatching {
                active.conversation.close()
                active.closeSucceeded.set(true)
                active.conversationReleased.set(true)
            }.onFailure { throwable ->
                logger.log(
                    type = "close_failed",
                    fields = mapOf(
                        "run_id" to active.runId,
                        "conversation_seq" to active.conversationSeq,
                        "throwable" to throwable.stackTraceToString(),
                    ),
                )
            }
        }

        StopAction.CANCEL -> {
            runCatching { active.conversation.cancelProcess() }
        }

        StopAction.NONE -> Unit
    }
}

4) Message callback terminal handling

override fun onDone() {
    publishTerminal(
        active = active,
        terminal = TerminalState(
            kind = TerminalKind.DONE,
            throwable = null,
            deliveredAtMs = SystemClock.elapsedRealtime(),
        ),
    )
}

override fun onError(throwable: Throwable) {
    publishTerminal(
        active = active,
        terminal = TerminalState(
            kind = TerminalKind.ERROR,
            throwable = throwable,
            deliveredAtMs = SystemClock.elapsedRealtime(),
        ),
    )
}

5) Minimal repro configuration actually used

RunSpec(
    label = "close-external-single",
    stopAction = StopAction.CLOSE,
    stopOrigin = StopOrigin.EXTERNAL_DELAY_AFTER_FIRST_MESSAGE,
    externalStopDelayMs = 1216L,
    watchdogTimeoutMs = null,
    notes = "single-close-during-streaming-after-first-message",
)

This is the important point: the minimal repro path does not use the watchdog at all.

---

MessageCallback / state-machine behavior

The repro app uses a deliberately small state machine to make the callback lifecycle observable.

Per-run state tracked

Each run tracks:

• messageCount
• firstMessageAtMs
• lastMessageAtMs
• stopIssued
• stopIssuedAtMs
• terminalDelivered
• terminalDeliveredAtMs
• lateMessagesAfterStop
• lateMessagesAfterTerminal
• duplicateTerminalSignals
• closeAttempted
• closeSucceeded
• cleanupCloseIssued

Callback processing rules

• Every onMessage(...) increments messageCount
• The first onMessage(...) records firstMessageAtMs
• Every onMessage(...) records lastMessageAtMs
• After stopIssued == true, additional messages are recorded as lateMessagesAfterStop
• After a terminal callback has already been observed, additional messages are recorded as lateMessagesAfterTerminal
• onDone() and onError(...) both flow through a single publishTerminal(...) gate protected by compareAndSet(false, true) so duplicate terminal delivery can be detected explicitly

This was added specifically so I could tell whether:

• close() returned before any terminal callback
• tokens/chunks still arrived after stop was requested
• the native layer crashed before onDone() / onError(...) reached Java/Kotlin

In the crashing runs, that is exactly what I see: close() returns, at least one late message may still arrive, and then the process dies natively before a terminal callback is delivered back to the app.

---

How the watchdog works in the broader harness

For completeness, the broader harness also contains a watchdog scenario, but again, it is not required for this bug.

The watchdog is implemented as a scheduled coroutine that only fires if a run has not reached a terminal callback by a timeout:

if (spec.stopOrigin == StopOrigin.WATCHDOG && spec.watchdogTimeoutMs != null) {
    active.watchdogJob = scope.launch(schedulerDispatcher) {
        delay(spec.watchdogTimeoutMs)
        if (!active.terminalDelivered.get()) {
            issueStop(active, reason = "watchdog_timeout")
        }
    }
}

So:

• it is just another trigger source for issueStop(...)
• it does not change how close() or cancelProcess() is executed
• it does not participate in the minimal repro path discussed here

For the runs that actually crashed in my evidence below:

• stop_origin = EXTERNAL_DELAY_AFTER_FIRST_MESSAGE
• external_stop_delay_ms = 1216
• watchdog_timeout_ms = null

That is why I believe it is accurate to say:

The watchdog is present in the larger harness, but it is not needed to reproduce this bug.

---

Reproduction steps

These are the concrete steps I used in the dedicated repro app.

GPU repro

1. Build and install the repro app with the bundled model asset
2. Launch the app
3. Select GPU backend
4. Initialize the engine
5. Tap Run Close-external-delay-once
6. The app starts streaming with sendMessageAsync(...)
7. On the first onMessage(...), the app schedules conversation.close() after 1216 ms
8. close() returns successfully
9. The process later crashes with native SIGSEGV inside liblitertlm_jni.so

CPU repro

1. Relaunch the app after the GPU crash
2. Select CPU backend
3. Initialize the engine
4. Tap Run Close-external-delay-once again
5. The same delayed-close sequence is used
6. close() returns successfully
7. The process again crashes with native SIGSEGV inside liblitertlm_jni.so

---

Key evidence from the app-side artifacts

Critical evidence log — GPU run

{"ts_wall":"2026-03-12T10:56:14.225Z","ts_mono_ms":431170383,"type":"run_start","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","stop_action":"CLOSE","stop_origin":"EXTERNAL_DELAY_AFTER_FIRST_MESSAGE","stop_after_messages":2147483647,"external_stop_delay_ms":1216,"watchdog_timeout_ms":null,"notes":"single-close-during-streaming-after-first-message","launch_thread":"main","prompt_chars":193}
{"ts_wall":"2026-03-12T10:56:14.228Z","ts_mono_ms":431170385,"type":"send_async_begin","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","thread":"main","prompt_chars":193}
{"ts_wall":"2026-03-12T10:56:14.230Z","ts_mono_ms":431170387,"type":"send_async_returned","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","thread":"main"}
{"ts_wall":"2026-03-12T10:56:14.579Z","ts_mono_ms":431170737,"type":"first_message_observed","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","message_index":1,"chars":1,"thread":"Thread-5","time_to_first_message_ms":358}
{"ts_wall":"2026-03-12T10:56:15.809Z","ts_mono_ms":431171967,"type":"stop_request","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","stop_action":"CLOSE","stop_origin":"EXTERNAL_DELAY_AFTER_FIRST_MESSAGE","reason":"external_delay_after_first_message","message_count":32,"thread":"scenario-scheduler"}
{"ts_wall":"2026-03-12T10:56:15.814Z","ts_mono_ms":431171972,"type":"close_begin","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","reason":"external_delay_after_first_message","thread":"scenario-scheduler"}
{"ts_wall":"2026-03-12T10:56:15.815Z","ts_mono_ms":431171973,"type":"close_returned","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","reason":"external_delay_after_first_message","thread":"scenario-scheduler"}

Run artifact snapshot — GPU run

Important fields from run_0001.json:

{
  "phase": "close_returned",
  "run_id": 1,
  "engine_session_id": 1,
  "conversation_seq": 1,
  "label": "close-external-single",
  "stop_action": "CLOSE",
  "stop_origin": "EXTERNAL_DELAY_AFTER_FIRST_MESSAGE",
  "external_stop_delay_ms": 1216,
  "message_count": 32,
  "stop_issued": true,
  "terminal_delivered": false,
  "terminal_kind": null,
  "late_messages_after_stop": 1,
  "close_attempted": true,
  "close_succeeded": true,
  "cleanup_close_issued": false,
  "critical_evidence_file": "/data/user/0/com.negi.litelm/files/issue_artifacts/run_0001.critical.jsonl"
}

This is one of the strongest app-side indicators for me:

• close() was entered and returned normally
• at least one message arrived after stop was issued
• no terminal callback was delivered to the app
• the process then died in native code

Critical evidence log — CPU run

{"ts_wall":"2026-03-12T11:15:08.266Z","ts_mono_ms":432304424,"type":"run_start","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","stop_action":"CLOSE","stop_origin":"EXTERNAL_DELAY_AFTER_FIRST_MESSAGE","stop_after_messages":2147483647,"external_stop_delay_ms":1216,"watchdog_timeout_ms":null,"notes":"single-close-during-streaming-after-first-message","launch_thread":"main","prompt_chars":193}
{"ts_wall":"2026-03-12T11:15:08.273Z","ts_mono_ms":432304430,"type":"send_async_begin","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","thread":"main","prompt_chars":193}
{"ts_wall":"2026-03-12T11:15:08.278Z","ts_mono_ms":432304436,"type":"send_async_returned","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","thread":"main"}
{"ts_wall":"2026-03-12T11:15:08.812Z","ts_mono_ms":432304969,"type":"first_message_observed","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","message_index":1,"chars":1,"thread":"Thread-5","time_to_first_message_ms":553}
{"ts_wall":"2026-03-12T11:15:10.039Z","ts_mono_ms":432306197,"type":"stop_request","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","stop_action":"CLOSE","stop_origin":"EXTERNAL_DELAY_AFTER_FIRST_MESSAGE","reason":"external_delay_after_first_message","message_count":30,"thread":"scenario-scheduler"}
{"ts_wall":"2026-03-12T11:15:10.053Z","ts_mono_ms":432306211,"type":"close_begin","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","reason":"external_delay_after_first_message","thread":"scenario-scheduler"}
{"ts_wall":"2026-03-12T11:15:10.058Z","ts_mono_ms":432306215,"type":"close_returned","run_id":1,"engine_session_id":1,"conversation_seq":1,"label":"close-external-single","reason":"external_delay_after_first_message","thread":"scenario-scheduler"}

Run artifact snapshot — CPU run

{
  "phase": "close_returned",
  "run_id": 1,
  "engine_session_id": 1,
  "conversation_seq": 1,
  "label": "close-external-single",
  "stop_action": "CLOSE",
  "stop_origin": "EXTERNAL_DELAY_AFTER_FIRST_MESSAGE",
  "external_stop_delay_ms": 1216,
  "message_count": 30,
  "stop_issued": true,
  "terminal_delivered": false,
  "terminal_kind": null,
  "late_messages_after_stop": 1,
  "close_attempted": true,
  "close_succeeded": true,
  "cleanup_close_issued": false,
  "critical_evidence_file": "/data/user/0/com.negi.litelm/files/issue_artifacts/run_0001.critical.jsonl"
}

So the same basic pattern is visible in both GPU and CPU runs.

---

Native log excerpts

GPU native crash

2026-03-12 10:56:14.290  ... session_basic.cc:466] RunDecodeAsync
2026-03-12 10:56:14.317  ... sampler_factory.cc:335] OpenCL sampler not available, falling back to statically linked C API
2026-03-12 10:56:15.878  ... libc A  Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x760065002200bb in tid 10924 (Thread-37), pid 10686 (com.negi.litelm)
2026-03-12 10:56:16.342  ... crash_dump64 A  #00 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #01 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #02 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #03 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #04 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #05 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #06 pc ... liblitertlm_jni.so
2026-03-12 10:56:16.342  ... crash_dump64 A  #07 pc ... liblitertlm_jni.so

CPU native crash

2026-03-12 11:15:08.771  ... session_basic.cc:466] RunDecodeAsync
2026-03-12 11:15:10.069  ... libc A  Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x73752f617461643f in tid 12680 (Thread-34), pid 12311 (com.negi.litelm)
2026-03-12 11:15:10.378  ... crash_dump64 A  #00 pc ... liblitertlm_jni.so
2026-03-12 11:15:10.378  ... crash_dump64 A  #01 pc ... liblitertlm_jni.so
2026-03-12 11:15:10.378  ... crash_dump64 A  #02 pc ... liblitertlm_jni.so
2026-03-12 11:15:10.378  ... crash_dump64 A  #03 pc ... liblitertlm_jni.so
2026-03-12 11:15:10.378  ... crash_dump64 A  #04 pc ... liblitertlm_jni.so
2026-03-12 11:15:10.378  ... crash_dump64 A  #05 pc ... liblitertlm_jni.so

---

Important behavioral observations

From the dedicated repro app and the captured artifacts, the sequence looks like this:

1. sendMessageAsync(...) returns successfully
2. streaming begins and onMessage(...) is invoked
3. after the first message, the app schedules a delayed external close()
4. conversation.close() is called on a separate scheduler thread
5. close() returns successfully to the app
6. at least one late streamed message may still arrive after stop was issued
7. no onDone() or onError(...) terminal callback reaches the app
8. the process crashes natively inside liblitertlm_jni.so

That is why I believe the issue is not merely “my app closed too early” in a Java/Kotlin sense.

The more specific failure appears to be that:

• close() can return while native decode / callback activity is still live or racing,
• and the native layer later dereferences invalid state and crashes.

I cannot prove the exact native root cause from app-side evidence alone, but the reproduction is consistent with a teardown / callback / decode race around conversation shutdown.

---

Clarification on cancelProcess() vs close()

The larger harness can test both cancelProcess() and close(), but for this issue the important point is:

• I do not need cancelProcess() to reproduce it
• I do not need the watchdog to reproduce it
• the smallest reproducible path is close() during active streaming after first message + fixed delay

So if the team wants the narrowest possible repro to inspect first, I would suggest focusing on this sequence:

1. create conversation
2. sendMessageAsync(...)
3. wait for first onMessage(...)
4. sleep ~1216 ms
5. call conversation.close() from another thread while streaming is active
6. observe native crash in liblitertlm_jni.so

---

Final summary

I hope this helps clarify the behavior:

• I created a new minimal repro app specifically for this issue
• the repro uses a bundled asset model, so download / MediaStore logic is out of the picture
• the crash reproduces without the watchdog
• the crash also does not require cancelProcess()
• the minimal reproducer is external delayed conversation.close() during active streaming
• I reproduced it on both GPU and CPU on Pixel 9a / API 36
• app-side evidence shows close() returns successfully, terminal callback is never delivered, and the process then dies in native liblitertlm_jni.so

If helpful, I can also share a stripped-down version of the repro app containing only:

• asset model staging
• engine init
• one conversation
• one button for close-after-first-message+1216ms
• JSONL evidence logging

so the team can run the exact minimal scenario directly.

Thanks again.

[ishizuki-tech]

Here a link to the small app repo : https://github.com/ishizuki-tech/LiteRT_Crash-Reproducer/tree/main

Logs:
https://github.com/ishizuki-tech/LiteRT_Crash-Reproducer/blob/main/saved.cpu_crash.log for CPU
https://github.com/ishizuki-tech/LiteRT_Crash-Reproducer/blob/main/saved.gpu_crash.log for GPU

Thanks again