TIKA-4838: Expose the MP4 audio and video track codecs as FourCCs (audio:fourcc, video:fourcc) by dschmidt · Pull Request #3062 · apache/tika

dschmidt · 2026-08-24T13:05:07Z

Adds audio:fourcc and video:fourcc, each the track's codec as a four-character code from the MP4/QuickTime sample description (audio: mp4a/alac/ac-3; video: avc1/hev1).

TikaMp4SoundHandler/TikaMp4VideoHandler already read the sample-entry fourcc (audio uses it to detect drms/enca protected streams; video ignored it); this exposes it as metadata. Because it comes from the track, both are set for video/* files, where xmpDM:audioCompressor does not fire and xmpDM:videoCompressor carries the compressor name (encoder string, e.g. "Lavc… libx264"), not the codec fourcc. The stsd walk is shared between the handlers (Mp4SampleEntries), handling size 0 and largesize entries, and a fourcc is only exposed when it is printable ASCII, with QuickTime's trailing padding trimmed.

For protected streams (drms/enca/encv/drmi) the value is the original codec named by the protection scheme info (sinf/frma); a bare entry without sinf reports the protected format itself. Audio additionally sets audio:has-drm. Both are per-stream values: with several tracks or sample entries they reflect the last one, like audio:bitrate/video:bitrate.

Verified against an M4A (mp4a), a video/mp4 with H.264 video + AAC audio (avc1 + mp4a), a DRM-protected M4A without sinf (drms), and synthetic fixtures built per ISO/IEC 14496-12 8.12: a FairPlay drms entry with sinf/frma (mp4a), an encv entry with sinf/frma (avc1), and a track with two sample entries (mp4a then ac-3); metadata schema registries updated.

https://issues.apache.org/jira/browse/TIKA-4838

Follows TIKA-4779, TIKA-4800 and TIKA-4802.

Copilot

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Expose MP4/QuickTime track codec FourCCs as audio:format and video:format, including recovery of original codecs for protected streams via nested frma boxes.

Changes:

Add new Audio.FORMAT and Video.FORMAT metadata properties and register schema keys.
Populate audio:format/video:format from sample description entries, with frma fallback for protected streams.
Add parser tests asserting the new metadata for audio-only and video+audio MP4s.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tika-parsers/.../MP4ParserTest.java	Adds assertions/tests for `audio:format` and `video:format`.
tika-parsers/.../TikaMp4VideoHandler.java	Extracts video codec FourCC and recovers original FourCC for protected video streams.
tika-parsers/.../TikaMp4SoundHandler.java	Extracts audio codec FourCC and recovers original FourCC for protected audio streams.
tika-metadata-schema/.../metadata-keys.json	Registers `audio:format` and `video:format` keys.
tika-metadata-schema/.../metadata-key-fields.json	Maps `Audio.FORMAT`/`Video.FORMAT` fields to the new keys.
tika-core/.../Video.java	Adds `Video.FORMAT` property with Javadoc.
tika-core/.../Audio.java	Adds `Audio.FORMAT` property with Javadoc.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

… video:format) TikaMp4SoundHandler/TikaMp4VideoHandler read the sample-entry format fourcc but only use it for other purposes (audio: DRM detection; video: nothing). Expose it as audio:format and video:format, each the track's four-character codec identifier (e.g. mp4a/alac/ac-3, avc1/hev1). This gives the per-track codec for video/* files, where xmpDM:audioCompressor does not fire and xmpDM:videoCompressor carries the compressor name. For protected streams the value is the protection scheme fourcc (drms/enca/encv/drmi); audio also sets audio:has-drm. Follows TIKA-4779, TIKA-4800 and TIKA-4802.

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

…ormats, not protection schemes

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

tika-core/src/main/java/org/apache/tika/metadata/Audio.java:92

The Javadoc says XMPDM#AUDIO_COMPRESSOR is a “human-readable codec name”, but in MP4 parsing it is set from the file’s major brand (e.g., "M4A"), which is not the track codec. Please reword to avoid misrepresenting what AUDIO_COMPRESSOR contains here.

     * Distinct from {@link XMPDM#AUDIO_COMPRESSOR}, the human-readable codec name.

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated no new comments.

tballison · 2026-08-26T22:04:52Z

From my claude:

Worth acting on
  1. Protected entries report the wrapper fourcc, not the codec (TikaMp4SoundHandler.java:94, same in Video). For drms/enca/encv/drmi, ISO 14496-12 puts the real codec in
     the child sinf/frma box; the handler already walks child boxes for esds/btrt, so frma is reachable. As-is a FairPlay .m4p gives audio:format=drms — and audio:has-drm
     already carries that fact. Once released, the value semantics are frozen. (The testMP4_drm.m4a fixture's drms entry is a bare stub with no sinf, so a real sample is
     needed to test.)
  2. Last-entry/last-track wins, undocumented (TikaMp4SoundHandler.java:94). FORMAT is overwritten per stsd entry and per track, HAS_DRM is set on any entry. [enca, mp4a]
     → has-drm=true, format=mp4a; AAC+E-AC-3 tracks → format=ec-3 while bitrate comes from the AAC track. Pick first-entry or document it; no multi-entry test guards it.
  3. fourcc emitted verbatim with no printability check (TikaMp4VideoHandler.java:108). Previously only compared; now a public metadata value. Corrupt bytes 00 01 02 03
     land as control chars in XML output. Gate on printable ASCII 0x20–0x7E and trim QuickTime trailing spaces (raw , rle ).
  4 -- I don't agree with
  
  Design question — decide before release
  5. Generic audio:format/video:format in tika-core get MP4-only fourcc semantics; no other A/V parser populates them, and audio:encoding +
  xmpDM:audioCompressor/videoCompressor already express codec identity in other vocabularies. Either name it for what it is (e.g. mp4-prefixed / "sample-entry fourcc") or
  define it container-neutrally with MP4 as first producer. Precedent cuts both ways (audio:has-drm is already MP4-only).
  6. New javadoc in Audio.java:14 / Video.java:35 calls xmpDM:audioCompressor "the human-readable codec name" — but MP4Parser sets it from the ftyp brand (M4A, which the
  test asserts) and videoCompressor from the raw encoder string. Reword or drop the cross-ref.

  Minor
  7. Sound and Video handlers now have byte-identical stsd walk loops + private fourCc(); the Sound copy already diverged (pos + 18 <= end guard). Neither handles size==1
  (largesize) or size==0. Worth a shared walker.
  8. Comment churn: 3-line justify-to-reviewer test comment in MP4ParserTest.java:133; enca comment lost its "ISO common encryption" anchor (relevant to #1).

Let me know what you think.

Address review: move the sample entry loop into Mp4SampleEntries, shared by the sound and video handlers, handling size 0 (to end of box) and size 1 (largesize). Only expose a format fourcc when it is printable ASCII, with QuickTime's trailing padding spaces trimmed. Drop the misleading cross-ref to xmpDM:audioCompressor/videoCompressor, restore the ISO common encryption anchor on the enca comment and trim the test comment.

…deo:fourcc The value is the codec's four-character code, not a generic format name, and FourCCs are not MP4-specific (QuickTime, AVI and others use them), so expose them as audio:fourcc / video:fourcc and say so in the javadoc.

…om sinf/frma A protected entry (drms/enca/encv/drmi) keeps its original codec in the ProtectionSchemeInfoBox's OriginalFormatBox (ISO/IEC 14496-12, 8.12), so report that as the fourcc and leave the DRM fact to audio:has-drm; only a bare entry without sinf still reports the protected format itself. Synthetic fixtures built per the spec and FFmpeg's/GPAC's writers: testMP4_drm_frma.m4a (FairPlay drms with esds and sinf/frma mp4a), testMP4_encv.mp4 (testMP4Video.mp4 with an encv entry, sinf before btrt), testMP4_twoSampleEntries.m4a (mp4a then ac-3, guards last-entry-wins).

dschmidt · 2026-08-26T22:56:26Z

Thanks, all addressed:

Protected entries now report the original codec from sinf/frma; a bare entry without sinf still reports the protected format. Since there was no real sample, the fixtures are synthetic, built per ISO/IEC 14496-12 8.12 and cross-checked against FFmpeg's and GPAC's writers: a FairPlay drms (esds + sinf/frma mp4a) and testMP4Video.mp4 with its avc1 turned into encv (sinf before btrt).
Kept last-entry-wins, consistent with audio:bitrate/video:bitrate; it's documented in the javadoc and now guarded by a two-entry fixture (mp4a then ac-3).
FourCCs are only exposed when all bytes are printable ASCII, with QuickTime's trailing padding trimmed (raw , rle ).
Renamed to audio:fourcc / video:fourcc. FourCCs aren't MP4-specific (QuickTime, AVI, ...), so no container prefix, but the name now says what the value is. A human-readable codec name would need the esds objectTypeIndication and a mapping table, so I'd keep that for a follow-up.
Dropped the cross-reference to xmpDM:audioCompressor/videoCompressor.
Both handlers share Mp4SampleEntries for the stsd walk, which now also handles size 0 and largesize entries (unit-tested).
Trimmed the test comment, restored the ISO common encryption anchor.

Not sure which item 4 was, happy to look if you paste it.

Copilot

Pull request overview

Copilot reviewed 9 out of 13 changed files in this pull request and generated 1 comment.

A header-only frma (size 8) passed the entry-bound check and read its format from the following box.

Copilot

Pull request overview

Copilot reviewed 9 out of 13 changed files in this pull request and generated no new comments.

dschmidt force-pushed the TIKA-audio-format branch 2 times, most recently from 5452bf9 to cc7b684 Compare August 24, 2026 13:52

dschmidt changed the title ~~TIKA-4838: Expose the MP4 audio track codec (audio:format)~~ TIKA-4838: Expose the MP4 audio and video track codecs (audio:format, video:format) Aug 24, 2026

dschmidt marked this pull request as ready for review August 24, 2026 14:31

THausherr requested a lite review from Copilot August 24, 2026 15:24

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Copilot started reviewing on behalf of THausherr August 24, 2026 16:20 View session

dschmidt force-pushed the TIKA-audio-format branch from cc7b684 to 957f037 Compare August 24, 2026 16:53

THausherr requested a lite review from Copilot August 24, 2026 17:38

Copilot started reviewing on behalf of THausherr August 24, 2026 17:42 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread tika-core/src/main/java/org/apache/tika/metadata/Audio.java Outdated

Comment thread tika-core/src/main/java/org/apache/tika/metadata/Video.java Outdated

TIKA-4838 - wording: drms/enca/encv/drmi are protected sample entry f…

9ec25dc

…ormats, not protection schemes

THausherr requested a lite review from Copilot August 24, 2026 18:58

Copilot started reviewing on behalf of THausherr August 24, 2026 18:59 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread ...es/tika-parser-audiovideo-module/src/test/java/org/apache/tika/parser/mp4/MP4ParserTest.java Outdated

TIKA-4838 - align the test comment with the sample-entry wording

706ee4c

dschmidt requested a lite review from Copilot August 24, 2026 21:18

Copilot AI reviewed Aug 24, 2026

THausherr requested a lite review from Copilot August 26, 2026 05:12

Copilot started reviewing on behalf of THausherr August 26, 2026 05:12 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

THausherr requested a lite review from Copilot August 26, 2026 05:16

Copilot started reviewing on behalf of THausherr August 26, 2026 05:17 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

dschmidt added 3 commits August 27, 2026 00:36

dschmidt changed the title ~~TIKA-4838: Expose the MP4 audio and video track codecs (audio:format, video:format)~~ TIKA-4838: Expose the MP4 audio and video track codecs as FourCCs (audio:fourcc, video:fourcc) Aug 26, 2026

THausherr requested a lite review from Copilot August 27, 2026 04:07

Copilot started reviewing on behalf of THausherr August 27, 2026 04:08 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Comment thread ...tika-parser-audiovideo-module/src/main/java/org/apache/tika/parser/mp4/Mp4SampleEntries.java

TIKA-4838 - require the frma box to hold its payload

4b3e63c

A header-only frma (size 8) passed the entry-bound check and read its format from the following box.

THausherr requested a lite review from Copilot August 27, 2026 09:01

Copilot started reviewing on behalf of THausherr August 27, 2026 09:08 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

tballison merged commit 9cad39e into apache:main Aug 27, 2026
4 checks passed

Uh oh!

Conversation

dschmidt commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

tballison commented Aug 26, 2026

Uh oh!

dschmidt commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dschmidt commented Aug 24, 2026 •

edited

Loading