iOS error-code → likely-cause classifier (raw codes aren't enough to guess root cause)

[jonathaneoliver]

Why

When iOS playback fails, the AVPlayer numeric error code alone is rarely enough to guess the root cause. The same code is overloaded across domains (e.g. -12642 means kCMFormatDescriptionError_ValueNotAvailable in CoreMedia and "No matching mediaFile from playlist" when AVPlayer raises it from the HLS path), some codes are completely undocumented (-12860, -12888, -12174), and several of our worst incidents were cascades — one server-side defect produced a sequence of codes, with the first symptom carrying the most diagnostic signal but the loudest log line being a downstream stale-playlist complaint.

We test against legal/well-formed streams, but spec-violating streams exist in the wild — we should be better at recognizing the effect of broken/malformed manifests on the player from error-code patterns alone.

What we know — go-live (server) bugs that produced iOS error codes

Issue	Server defect	iOS effect
#109	Loop-boundary playlist explosion (sliding window not capped across EXT-X-DISCONTINUITY — ~620 B → ~4.8 KB)	-12860 decode error → poll stops → -12888 "playlist unchanged for 1.5× target duration" on audio → playback dies
#110	Residual #109 + content-type/codec-string mismatch on master variants	-12860 on 540p → ABR upshift to 2160p → repeated -12642 "No matching mediaFile" → -12888 cascade
#149	Missing EXT-X-DISCONTINUITY-SEQUENCE (RFC 8216 §4.3.3.3 violation) + MAX_LIVE_WINDOW_DURATION too tight (36 s vs 20 s buffer)	-12642 + AVPlayer "Cannot Open" on cross-discontinuity ABR upshift; oldest-edge fall-off
#151	v9 tags on EXT-X-VERSION:7 playlist; or HOLD-BACK < 3 × TARGETDURATION; or EXT-X-START inserted between #EXTM3U and #EXT-X-VERSION	-12646 "playlist parse error" — entire playlist rejected
#161	HOLD-BACK not preserved across stall recovery (client-fix in iOS app)	Recurring stalls — player snapped to oldest-segment edge with zero margin
#94/#95/#96	go-live concurrency (3-lock DASH cache, fresh LLHLSGenerator per 200 ms tick, double-lock duplicate MPD generation)	Lock contention → manifest stall under load → indirectly -12888/stalls
#137	(LocalHTTPProxy bug) chunked TE forced on 206 Partial Content with Content-Length stripped	URLAsset err=-12174 flood on byte-range fetches
#145	Newer iOS sims still emit -12174 after #137 fix; sim-only AVFoundation tightening	Cosmetic only — no real-device impact
#135	Upstream URLSessionDataTask not cancelled when AVPlayer abandoned client	ECANCELED (POSIX 89) flood — wasted upstream traffic

Codes we've actually seen and what we currently know about them

Code	Label / "errorComment" string	Cause we've identified
-12174	URLAsset os_log warning (no AVError mapping)	206 Partial Content returned with Transfer-Encoding: chunked and no Content-Length after a Range: request (#137); newer-sim variant: HTTP/1.1 Connection: close (#145)
-12642	"No matching mediaFile from playlist" (HLS context) — overloaded with kCMFormatDescriptionError_ValueNotAvailable	Missing EXT-X-DISCONTINUITY-SEQUENCE, live window too narrow vs forward buffer, or wrong content served (#149, #110)
-12646	"playlist parse error"	v7-vs-v9 mismatch, HOLD-BACK underflow, or bad tag insertion order (#151)
-12860	CoreMedia decode error (no public symbol)	Oversized/malformed playlist (#109); also genuine bad codec config in master
-12888	"playlist unchanged for 1.5× target duration"	Almost always downstream — player stopped polling after an earlier -12860/-12646 and the audio playlist now appears stale; or genuine go-live worker stall

PlaybackDiagnostics.swift:1150-1262 has the full reverse-engineered switch covering -11800…-11865 (AVError) and -12640…-12900 (CoreMedia families). Note the labels there are header-derived, not behavioral — -12642's real meaning in our incidents has nothing to do with "ValueNotAvailable".

The core problem

A bare numeric code is one signal. To classify cause we need three:

1. Code — AVPlayerItem.error.code / errorLog().events.last.errorStatusCode.
2. Comment — errorComment / localizedDescription. This is what disambiguates the overloaded codes (-12642 → "No matching mediaFile" vs CM-domain).
3. Context fingerprint captured at error time:
   • last playlist response: size, body hash, status, framing (Content-Length vs chunked TE), Content-Type
   • in-flight variant switch? distance (segments / wallclock) since last loop boundary
   • last EXT-X-DISCONTINUITY-SEQUENCE value seen
   • last byte-range request and response
   • server-side go-proxy state (already captured by SSE / 911 HAR (Add '911' / user-marked button to all player apps + auto-HAR-snapshot on server #308) / freeze HAR (Auto-snapshot HAR on player freeze / auto-recovery #273))

With these three, a rule classifier (code, comment_regex, predicates) → (probable_cause, confidence) turns -12642 into one of:

• MISSING_DISCONTINUITY_SEQUENCE (high) — comment matches "No matching mediaFile" + a discontinuity wrap occurred in the last few seconds + variant switch was in flight
• WINDOW_FELL_OFF_OLDEST_EDGE (medium) — comment matches "No matching mediaFile" + player position was within ~1× TARGETDURATION of MEDIA-SEQUENCE head
• CM_FORMAT_VALUE_NOT_AVAILABLE (low) — error came from a non-HLS path
• UNKNOWN (fallback)

Proposed direction (going forwards)

Hybrid client + server classifier:

• Client-side (iOS app) — small, high-confidence rule set in PlaybackDiagnostics for the ~5 codes we've actually triggered. Stamps probable_cause + confidence into the metric + on-device 911 HAR (Add '911' / user-marked button to all player apps + auto-HAR-snapshot on server #308) / freeze HAR (Auto-snapshot HAR on player freeze / auto-recovery #273). Slow to update (ships with the app) but lets the dashboard label incidents in real time.
• Server-side (analytics sidecar Analytics sidecar: ClickHouse + Grafana for cross-session event analysis & historical replay #336) — richer ruleset that runs over the HAR + SSE log retrospectively. Can use cross-session context (e.g. "all sessions on this content in the last 5 minutes hit -12860 → server-side regression"), update freely without app release, and backfill labels onto historical incidents.
• Disambiguation cheat sheet — checked-in doc (e.g. apple/InfiniteStreamPlayer/IOS_ERROR_CODES.md or PRD section) capturing the (code, comment, context) → cause mappings as we learn them, alongside the issue refs that proved each cause. Both classifiers source rules from the same catalogue.

Concrete first steps

1. Add the missing observed codes (-12174, -12888) and HLS-context labels to interpretCoreMediaErrorCode / interpretAVErrorCode in PlaybackDiagnostics.swift. Cheap, immediate dashboard win.
2. Capture the context fingerprint at error-emission time: extend LocalHTTPProxy's existing per-segment hook (Track per-segment identity via LocalHTTPProxy (iOS) #157) to keep a small ring buffer of the last N (~10) responses (URL, status, framing, size, byterange) so an errorLog event can attach the immediately-preceding HTTP context.
3. Define the rule schema (likely YAML/JSON: {code, comment_regex, predicates[], cause, confidence, refs[]}) and seed it with the known cases from this issue.
4. Wire the client-side classifier into the metric payload (player_metrics_probable_cause) and the dashboard error lane.
5. Implement the same rule-engine server-side under Analytics sidecar: ClickHouse + Grafana for cross-session event analysis & historical replay #336 so retrospective labelling is consistent with live labelling.

Open questions

• Should probable_cause be a free-form string or an enum? Enum is cleaner for analytics; free-form is more honest about partial knowledge.
• How much context history is enough? Last 10 HTTP responses is cheap; a full HAR replay is overkill for live labelling.
• Do we want confidence as a score (0-1) or a level (low/medium/high)? Levels are easier to reason about; scores are easier to aggregate.

References

• Server bugs: go-live: HLS playlist explodes to full segment list at loop boundary #109 go-live/go-proxy: AVPlayer -12860/-12642 cascade failure during variant switching #110 Add EXT-X-DISCONTINUITY-SEQUENCE and enlarge HLS live window #149 Declare EXT-X-SERVER-CONTROL:HOLD-BACK on range-HLS playlists #151 Preserve live offset across stall recovery on iOS #161 go-live: DASH cache lock acquired 3x per request — serialize all concurrent DASH requests #94 go-live: LLHLSGenerator allocated fresh every 200ms tick #95 go-live: double-lock DASH cache pattern allows duplicate MPD generation #96
• Client/proxy: iOS LocalHTTPProxy: AVPlayer URLAsset err=-12174 flood — chunked TE on 206 responses #137 iOS LocalHTTPProxy: URLAsset err=-12174 noise on newer simulators (sim-only, real device clean) #145 iOS LocalHTTPProxy: ECANCELED log flood when AVPlayer abandons requests #135 Track per-segment identity via LocalHTTPProxy (iOS) #157
• Adjacent platform work: Add '911' / user-marked button to all player apps + auto-HAR-snapshot on server #308 (911 HAR), HAR: incident context block (device, scenario, timing, recovery chain) #281 (HAR incident context), Auto-snapshot HAR on player freeze / auto-recovery #273 (auto-HAR on freeze), Analytics sidecar: ClickHouse + Grafana for cross-session event analysis & historical replay #336 (analytics sidecar), Player error-characterization test framework #272 (player error-characterization framework)
• Existing reverse-engineered map: apple/InfiniteStreamPlayer/InfiniteStreamPlayer/PlaybackDiagnostics.swift:1150-1262
• Server comments cross-referencing iOS codes: go-live/pkg/generator/range_hls.go:183-200, go-live/pkg/generator/ll_hls.go:20,129, go-live/internal/api/handlers.go:2010, go-proxy/cmd/server/main.go:4765,4802

[jonathaneoliver]

Points: 13 (Fibonacci)
Effort: ~2–3 weeks calendar (spread across Swift, Go, and schema work with review cycles)
Priority: P2 — next sprint; diagnostic quality work, no user-facing breakage, and depends on #336 (analytics sidecar) still landing first
Value: medium — high leverage for future incident triage and the server-side retro-labelling story, but zero impact on current user-facing playback; no immediate shipping urgency

Rationale: The five concrete steps span three distinct surface areas (Swift PlaybackDiagnostics + LocalHTTPProxy ring buffer, rule schema definition, Go analytics sidecar rule engine, and dashboard wiring) with a hard dependency on #336 merging before step 5 is actionable. Step 1 alone (adding -12174/-12888 to the switch) is a 1-pointer, but the issue is scoped as the full classifier, so the aggregate estimate drives the number. 13 is appropriate: no single step is a deep unknown, but the cross-layer integration (client metric payload → dashboard lane → server retro-labelling) adds coordination surface that makes this larger than it first looks. Calendar effort exceeds raw engineering time because of the app-release cycle for any Swift change and the review/iteration loop on the rule schema.

Recommended split if you want to start sooner:

• Step 1 (add missing codes to interpretCoreMediaErrorCode / interpretAVErrorCode) → 1 point, P2, ship independently
• Steps 2–3 (ring buffer + rule schema) → 5 points, can start once Track per-segment identity via LocalHTTPProxy (iOS) #157 hook is stable
• Steps 4–5 (metric wiring + sidecar rule engine) → 8 points, blocked on Analytics sidecar: ClickHouse + Grafana for cross-session event analysis & historical replay #336