TIKA-4794-option-3 (#2964)

Author: tballison

.skills/dev.md

@@ -47,6 +47,11 @@ provide the suggested commit message for the user to execute.
   Run **without** `-Pfast` before final commit to catch formatting
   and style issues.
 
+  **`-Pfast` skips test *execution*** (by design): a green `-Pfast`
+  build — including `-Pfast test` — has run zero tests, and stale
+  `target/surefire-reports/*` will look current. Verify with a plain
+  (non-`-Pfast`) `test` run.
+
 - **Forked JVM tests** — Integration tests in `tika-pipes` fork new
   JVMs that load classes from the local Maven repo, not from
   `target/classes`.  You must `./mvnw clean install -Pfast` the
@@ -93,6 +98,12 @@ provide the suggested commit message for the user to execute.
   legitimate exception: a path that is the data under test (e.g. an expected
   metadata value extracted from a test document) — leave those untouched.
 
+## Metadata Keys & Schema Registry
+
+Adding/renaming a metadata key touches the committed, build-gated registry in
+`tika-metadata-schema` — regeneration has real traps. See
+`.skills/metadata-schema.md`.
+
 ## Testing an End-to-End Change
 
 When a change affects parsing output (e.g., new parser behavior,

.skills/metadata-schema.md

@@ -0,0 +1,77 @@
+# Metadata Key Registry & Schema Skill
+
+Working with `tika-metadata-schema` — the committed, build-gated registry of Tika's metadata keys.
+File structure and the CLOSED/OPEN/TEMPLATE/UNKNOWN classification are in that module's `README.md`;
+this covers conventions, regeneration, and the traps.
+
+## The three registries
+
+Under `tika-metadata-schema/src/main/resources/org/apache/tika/metadata/`:
+
+- `metadata-keys.json` — closed set: every `Property` constant + the synthesized `tk:digest:*` cross-product.
+- `metadata-open-namespaces.json` — `PassthroughPrefix` prefixes for runtime-minted names (`html:`, `message:raw-header:`, `mdb-prop:`).
+- `metadata-key-fields.json` — TIKA-4797 `{class, field, key}` table for field-identity migration.
+
+Committed on purpose: they are the reviewable audit trail of the key space (a rename or a dropped key
+shows up as a diff). Don't switch to build-time-only generation — that loses the review signal.
+
+## Regenerate (after adding/changing a Property or PassthroughPrefix)
+
+```bash
+# if parser Property classes changed, install them first so the scan sees them:
+./mvnw -Pfast -DskipTests -pl tika-metadata-schema -am install -Dmaven.repo.local=$(pwd)/.local_m2_repo
+# regenerate all three:
+./mvnw -pl tika-metadata-schema -Pregen-metadata-schema process-classes -Dmaven.repo.local=$(pwd)/.local_m2_repo
+```
+
+Then `git diff` the JSONs, run the gate tests, commit.
+
+**Trap — never use `exec:java`.** `SchemaGenerator` scans `java.class.path`, force-loads Property
+classes, and swallows load failures. `exec:java` runs in-process on Maven's classpath → finds zero
+keys → emits a near-empty registry that exits 0 and *passes* `MetadataNoUnderscoreTest`. The
+`-Pregen-metadata-schema` profile uses the forking `exec` goal (`<classpath/>`) for a real child-JVM
+classpath. A hand-rolled `java -cp` also works, but the full repo classpath (~4000 jars) overflows
+the 128 KB arg limit — use the profile.
+
+**Trap — incomplete classpath drops keys silently.** Always validate: `git diff` should show only the
+intended change (a big key-count drop = classes failed to load), and `MetadataSchemaTest` regenerates
+under the full test classpath and asserts a byte-match — the definitive completeness check.
+
+## Gate tests (run WITHOUT `-Pfast`, which skips execution)
+
+```bash
+./mvnw -pl tika-metadata-schema test -Dmaven.repo.local=$(pwd)/.local_m2_repo
+```
+
+- `MetadataSchemaTest` / `MetadataFieldTableTest` — regenerate in-memory, assert committed files match.
+- `MetadataNoUnderscoreTest` — no Tika-coined key/prefix may contain `_`. Scans the JSON, so it
+  reflects the *registry*, not live code — regenerate before trusting it.
+- `MetadataKeyValidatorTest` — registry-driven CLOSED/OPEN/TEMPLATE/UNKNOWN classifier.
+- `MetadataCoverageTest` — fails if a scanned module's keys are neither in scope nor listed out-of-scope.
+
+Failures with stale `X-TIKA:`/underscore/`SHA256` keys usually mean *regenerate*, not edit code.
+
+## Naming conventions (frozen for 4.0, TIKA-4794)
+
+- All keys are `Property` constants — no bare `String` keys (`metadata-string-keys.json` retired).
+- Tika-coined prefix is `tk:` (`X-TIKA:` is legacy); kebab-case, no underscores.
+- External-standard names verbatim, *including* the standard's prefix: `dc:`, `xmp:`, `cp:`, `extended-properties:`.
+- HTTP has no namespace → `Content-Type`, `Content-Encoding`, `Location` stay bare (no `http:`).
+- Tika-coined message keys *do* get a namespace: `message:`, `multipart:`.
+- HttpHeaders keys are SIMPLE — Content-Type is not a bag; parsers use `set()`, not `add()`.
+- Digest keys use the JCA name via `DigestDef.getJavaName()`: `tk:digest:SHA-256`, `SHA3-256` (not
+  `SHA256`/`SHA3_256`). Config *input* still takes the enum name (`"SHA256"`); only the output key changes.
+
+## After a rename: sweep for stale key literals
+
+The compiler won't catch `metadata.get("Message-From")`. Grep the whole repo for old strings and
+prefer replacing them with the constant, so the next rename fails to compile instead of at test time:
+
+```bash
+grep -rn '"Message-' --include=*.java . | grep -v /target/
+grep -rn 'tk:digest:SHA[0-9]' --include=*.java . | grep -v /target/
+```
+
+## XML comments can't contain `--`
+
+A double hyphen in an XML comment makes the POM non-parseable and breaks the module build. Reword.

.skills/tika-eval-encoding-regression.md

@@ -158,10 +158,10 @@ To see what one chain change actually did, Compare the new run against the
 (IBM850 / x-MacRoman → windows-1252) vs the prior run and nothing else. A
 bigger-than-expected diff means the change fired more broadly than intended.
 
-## Per-file detector attribution (`X-TIKA:encodingDetectionTrace`)
+## Per-file detector attribution (`tk:encoding-detection-trace`)
 
 Every JSON extract from a chain with multiple detectors carries
-`X-TIKA:encodingDetectionTrace` in metadata. It's a per-detector emission
+`tk:encoding-detection-trace` in metadata. It's a per-detector emission
 log with the META detector's arbitration tag at the end:
 
 ```
@@ -174,8 +174,8 @@ meta detector chose. If the trace shows ONLY Mojibuster firing with a CJK
 pick, the bug is in Mojibuster's emission (pool too narrow), not in
 JunkFilter's arbitration.
 
-`X-TIKA:encodingDetector` is the simple-name credit string;
-`X-TIKA:detectedEncoding` is the final answer (also in `Content-Encoding`).
+`tk:encoding-detector` is the simple-name credit string;
+`tk:detected-encoding` is the final answer (also in `Content-Encoding`).
 
 ## Reproducing a single-file detection without a full chain
 

.skills/tika-eval-h2-query.md

@@ -135,7 +135,7 @@ WHERE DETECTED_ENCODING REGEXP 'big5|gb|euc|shift|jis|2022|949'
 ```
 
 (`DETECTED_ENCODING` is on `ENCODINGS_A`/`ENCODINGS_B` — join to `PROFILES`/`CONTENTS`
-on `ID` — populated from `X-TIKA:detectedEncoding`.)
+on `ID` — populated from `tk:detected-encoding`.)
 
 ## Tip
 

docs/modules/ROOT/pages/advanced/embedded-documents.adoc

@@ -49,12 +49,12 @@ documents. They answer: "Which document contained this one?" All fields are defi
 
 === Nesting Identifiers
 
-`TikaCoreProperties.EMBEDDED_ID` (`X-TIKA:embedded_id`)::
+`TikaCoreProperties.EMBEDDED_ID` (`tk:embedded-id`)::
 A 1-indexed integer assigned by Tika to each embedded document during parsing. IDs are
 assigned in the order documents are encountered by the `RecursiveParserWrapper`. This ID
 uniquely identifies each embedded document within a single parse operation.
 
-`TikaCoreProperties.EMBEDDED_ID_PATH` (`X-TIKA:embedded_id_path`)::
+`TikaCoreProperties.EMBEDDED_ID_PATH` (`tk:embedded-id-path`)::
 A path showing the containment hierarchy using `EMBEDDED_ID` values. For example, `/1/3`
 indicates that the file with `EMBEDDED_ID=3` was contained within the file with
 `EMBEDDED_ID=1`. This is the most reliable field for tracking containment relationships.
@@ -64,22 +64,22 @@ folder structures or original paths within the containers themselves.
 
 === Synthetic Paths
 
-`TikaCoreProperties.EMBEDDED_RESOURCE_PATH` (`X-TIKA:embedded_resource_path`)::
+`TikaCoreProperties.EMBEDDED_RESOURCE_PATH` (`tk:embedded-resource-path`)::
 A synthetic path built by concatenating file names (from `RESOURCE_NAME_KEY`) at each
 nesting level. This provides a human-readable path through the containment hierarchy.
 +
 WARNING: Do not use this field for creating directory structures to write out attachments.
 There may be path collisions, illegal characters, or zip slip vulnerabilities. Use
 `EMBEDDED_ID_PATH` for reliable containment tracking.
 
-`TikaCoreProperties.FINAL_EMBEDDED_RESOURCE_PATH` (`X-TIKA:final_embedded_resource_path`)::
+`TikaCoreProperties.FINAL_EMBEDDED_RESOURCE_PATH` (`tk:final-embedded-resource-path`)::
 Similar to `EMBEDDED_RESOURCE_PATH`, but calculated at the end of the full parse. For some
 parsers, an embedded file's name isn't known until after its child files have been parsed.
 This field may have fewer "unknown" file names than `EMBEDDED_RESOURCE_PATH`.
 
 === Resource Naming
 
-`TikaCoreProperties.RESOURCE_NAME_KEY` (`X-TIKA:resourceName`)::
+`TikaCoreProperties.RESOURCE_NAME_KEY` (`tk:resource-name`)::
 The file name (not path) of the resource. Tika makes a best effort to determine a meaningful
 name from the container's metadata. When unavailable, Tika falls back to synthetic names
 such as `embedded-1.jpeg`.
@@ -95,7 +95,7 @@ the nesting structure. All fields below are defined in `TikaCoreProperties`.
 
 === Internal Paths
 
-`TikaCoreProperties.INTERNAL_PATH` (`X-TIKA:internalPath`)::
+`TikaCoreProperties.INTERNAL_PATH` (`tk:internal-path`)::
 The path (including file name) as literally stored within the container. This is what the
 container knows about where the file lives in its internal structure:
 +
@@ -108,7 +108,7 @@ This differs fundamentally from `EMBEDDED_RESOURCE_PATH`:
 * `INTERNAL_PATH` is what the _container stores_ about the file's location within itself
 * `EMBEDDED_RESOURCE_PATH` is what _Tika synthesizes_ from the nesting structure
 
-`TikaCoreProperties.ORIGINAL_RESOURCE_NAME` (`X-TIKA:origResourceName`)::
+`TikaCoreProperties.ORIGINAL_RESOURCE_NAME` (`tk:orig-resource-name`)::
 For some file formats, the file path where the document was last saved on the creator's
 system. For example, an `.xlsx` file named `budget.xlsx` may include a metadata property
 storing where it was last saved: `C:\Users\Alice\budget.xlsx`. This is not specific to
@@ -118,11 +118,11 @@ embedded files - it's a property that certain file formats preserve about themse
 
 Microsoft Office formats use additional identifiers for embedded objects.
 
-`TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID` (`X-TIKA:embeddedRelationshipId`)::
+`TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID` (`tk:embedded-relationship-id`)::
 A Microsoft-specific identifier used internally to reference embedded objects within
 Office documents. This is the relationship ID from the Office Open XML or OLE structure.
 
-`Office.EMBEDDED_STORAGE_CLASS_ID` (`msoffice:embeddedStorageClassId`)::
+`Office.EMBEDDED_STORAGE_CLASS_ID` (`msoffice:embedded-storage-class-id`)::
 A UUID that identifies the class of embedded object in Microsoft formats. While not
 exactly a MIME type, it provides similar information about what type of object is
 embedded. Defined in the `Office` metadata class.
@@ -134,39 +134,39 @@ embedded. Defined in the `Office` metadata class.
 |Property |Metadata Key |Source
 
 |`EMBEDDED_ID`
-|`X-TIKA:embedded_id`
+|`tk:embedded-id`
 |Containment
 
 |`EMBEDDED_ID_PATH`
-|`X-TIKA:embedded_id_path`
+|`tk:embedded-id-path`
 |Containment
 
 |`EMBEDDED_RESOURCE_PATH`
-|`X-TIKA:embedded_resource_path`
+|`tk:embedded-resource-path`
 |Containment
 
 |`FINAL_EMBEDDED_RESOURCE_PATH`
-|`X-TIKA:final_embedded_resource_path`
+|`tk:final-embedded-resource-path`
 |Containment
 
 |`RESOURCE_NAME_KEY`
-|`X-TIKA:resourceName`
+|`tk:resource-name`
 |Containment
 
 |`INTERNAL_PATH`
-|`X-TIKA:internalPath`
+|`tk:internal-path`
 |Container
 
 |`ORIGINAL_RESOURCE_NAME`
-|`X-TIKA:origResourceName`
+|`tk:orig-resource-name`
 |Container
 
 |`EMBEDDED_RELATIONSHIP_ID`
-|`X-TIKA:embeddedRelationshipId`
+|`tk:embedded-relationship-id`
 |Container (MS)
 
 |`Office.EMBEDDED_STORAGE_CLASS_ID`
-|`msoffice:embeddedStorageClassId`
+|`msoffice:embedded-storage-class-id`
 |Container (MS)
 |===
 

docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc

@@ -147,7 +147,7 @@ java -jar tika-app.jar -J -t /tmp/tika-app-test/batch-input /tmp/tika-app-test/b
 ls /tmp/tika-app-test/batch-output2
 ----
 
-*Expected:* Creates .json files with text content (X-TIKA:content_handler should be ToTextContentHandler).
+*Expected:* Creates .json files with text content (tk:content-handler should be ToTextContentHandler).
 
 === Test 11: Version Check
 
@@ -183,7 +183,7 @@ java -jar tika-app.jar --language testPDF.pdf
 java -jar tika-app.jar --digest=md5 --json testPDF.pdf
 ----
 
-*Expected:* JSON output includes `X-TIKA:digest:MD5` field.
+*Expected:* JSON output includes `tk:digest:MD5` field.
 
 === Test 15: URL Input
 

docs/modules/ROOT/pages/advanced/integration-testing/tika-eval-regression.adoc

@@ -106,7 +106,7 @@ match.
 ----
 
 The `parse-context > commons-digester-factory` block makes every
-extracted record carry an `X-TIKA:digest:SHA256` metadata field —
+extracted record carry an `tk:digest:SHA256` metadata field —
 required by `tika-eval` for embedded-document alignment across runs.
 
 === Config B (variant under test)
@@ -233,7 +233,7 @@ Open the `.xlsx` files directly, or use the `regression` skill
 == Tips
 
 * *Keep the digester identical between A and B.*  tika-eval uses the
-  `X-TIKA:digest:SHA256` field on embedded documents to align records
+  `tk:digest:SHA256` field on embedded documents to align records
   across the two extracts.  If A digests and B doesn't (or different
   algorithms), the embedded-doc alignment falls back to filename and
   produces false-positive diffs.
@@ -310,7 +310,7 @@ Compare step finishes in 2-5 minutes depending on extract size.
 . Open `reports/mimes/mime_diffs_A_to_B.xlsx` to see the headline
   MIME-detection differences; the encoding-detector chain change
   surfaces as charset diffs in `mimes/mime_diffs_A_to_B_details.xlsx`
-  (`X-TIKA:detected_encoding`).
+  (`tk:detected-encoding`).
 
 A regression analysis writeup goes in
 `~/Desktop/claude-todo/<reports-dir-name>-analysis.md` per the

docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc

@@ -100,7 +100,7 @@ curl -s -X PUT -T testPDF.pdf http://localhost:9998/tika/xml
 curl -s -X PUT -T testPDF.pdf http://localhost:9998/tika/json
 ----
 
-*Expected:* JSON object with metadata and X-TIKA:content field.
+*Expected:* JSON object with metadata and tk:content field.
 
 === Test 7: PUT /meta
 

docs/modules/ROOT/pages/advanced/setting-limits.adoc

@@ -103,7 +103,7 @@ siblings at the current level continue to be processed.
 |`throwOnMaxDepth`
 |false
 |Whether to throw an `EmbeddedLimitReachedException` when maxDepth is reached.
-If false, processing continues and `X-TIKA:maxDepthReached=true` is set in metadata.
+If false, processing continues and `tk:exception:embedded-depth-limit-reached=true` is set in metadata.
 
 |`maxCount`
 |-1 (unlimited)
@@ -113,7 +113,7 @@ stops immediately.
 |`throwOnMaxCount`
 |false
 |Whether to throw an `EmbeddedLimitReachedException` when maxCount is reached.
-If false, processing continues and `X-TIKA:maxEmbeddedCountReached=true` is set.
+If false, processing continues and `tk:exception:embedded-resource-limit-reached=true` is set.
 |===
 
 === maxDepth Behavior
@@ -380,7 +380,7 @@ Metadata metadata = Metadata.newInstance(context);
 |`maxTotalBytes`
 |10 MB
 |Maximum total estimated size of all metadata in UTF-16 bytes. When exceeded,
-additional metadata is silently dropped and `X-TIKA:WARN:truncated_metadata` is set.
+additional metadata is silently dropped and `tk:warn:truncated-metadata` is set.
 
 |`maxFieldSize`
 |100 KB
@@ -423,7 +423,7 @@ Use this to extract only the metadata you need.
       "maxKeySize": 1024,
       "maxValuesPerField": 100,
       "includeFields": ["dc:title", "dc:creator", "dc:subject"],
-      "excludeFields": ["pdf:unmappedUnicodeCharsPerPage"]
+      "excludeFields": ["pdf:unmapped-unicode-chars-per-page"]
     }
   }
 }
@@ -436,15 +436,15 @@ of `includeFields` or size limits:
 
 * `Content-Type` - Required for parser selection
 * `Content-Length`, `Content-Encoding`, `Content-Disposition`
-* `X-TIKA:content` - The extracted text content
-* `X-TIKA:Parsed-By` - Parser chain information
-* `X-TIKA:WARN:*` - Warning metadata
+* `tk:content` - The extracted text content
+* `tk:parsed-by` - Parser chain information
+* `tk:warn:*` - Warning metadata
 * Access permission fields
 
 === Detecting Truncation
 
 When metadata is truncated due to limits, Tika sets the metadata field
-`X-TIKA:WARN:truncated_metadata` to `true`. You can check for this in your code:
+`tk:warn:truncated-metadata` to `true`. You can check for this in your code:
 
 [source,java]
 ----
@@ -458,7 +458,7 @@ if ("true".equals(metadata.get(TikaCoreProperties.TRUNCATED_METADATA))) {
 
 1. **Always set limits** when processing untrusted content
 2. **Use `includeFields`** to capture only the metadata you need
-3. **Monitor for truncation** by checking `X-TIKA:WARN:truncated_metadata`
+3. **Monitor for truncation** by checking `tk:warn:truncated-metadata`
 4. **Combine with process isolation** - limits protect against memory issues,
    but xref:advanced/robustness.adoc[process isolation] protects against crashes
 5. **Test with adversarial files** - use Tika's `MockParser` to simulate extreme cases

docs/modules/ROOT/pages/advanced/zip-detection.adoc

@@ -90,10 +90,10 @@ parser.parse(inputStream, handler, metadata, context);  // May fail on truncated
 
 The detector also sets additional metadata hints for parsers:
 
-`zip:detectorZipFileOpened`:: `true` if the detector successfully opened the ZIP as a `ZipFile`.
+`zip:detector-zip-file-opened`:: `true` if the detector successfully opened the ZIP as a `ZipFile`.
 The `ZipFile` object is available via `TikaInputStream.getOpenContainer()` for reuse by parsers.
 
-`zip:detectorDataDescriptorRequired`:: `true` if streaming detection required
+`zip:detector-data-descriptor-required`:: `true` if streaming detection required
 DATA_DESCRIPTOR support. This hint helps parsers choose the correct streaming mode.
 
 == Related Topics