TIKA-4816 metadata key api

Author: tballison

.skills/metadata-schema/SKILL.md

@@ -35,19 +35,19 @@ this covers conventions, regeneration, and the traps.
 Under `tika-metadata-schema/src/main/resources/org/apache/tika/metadata/`:
 
 - `metadata-keys.json` — closed set: every `Property` constant + the synthesized `tk:digest:*` cross-product.
-- `metadata-open-namespaces.json` — `PassthroughPrefix` prefixes for runtime-minted names (`html:`, `message:raw-header:`, `mdb-prop:`).
+- `metadata-open-namespaces.json` — `KeyPrefix` prefixes for runtime-minted names (`html:`, `message:raw-header:`, `mdb-prop:`).
 - `metadata-key-fields.json` — TIKA-4797 `{class, field, key}` table for field-identity migration.
 
 Committed on purpose: they are the reviewable audit trail of the key space (a rename or a dropped key
 shows up as a diff). Don't switch to build-time-only generation — that loses the review signal.
 
-## Regenerate (after adding/changing a Property or PassthroughPrefix)
+## Regenerate (after adding/changing a Property or KeyPrefix)
 
 ```bash
 tika-metadata-schema/regen.sh
 ```
 
-This does the full sequence in one shot: `-am install` so newly added Property/PassthroughPrefix
+This does the full sequence in one shot: `-am install` so newly added Property/KeyPrefix
 classes are on the scan classpath, regenerate all three registries via the forked-exec profile, print
 a before/after key-count check (catches an incomplete classpath scan), `git diff --stat` the
 registries, then run the gate tests. Flags: `--skip-install` (only safe if nothing outside
@@ -91,6 +91,8 @@ Failures with stale `X-TIKA:`/underscore/`SHA256` keys usually mean *regenerate*
 ## Naming conventions (frozen for 4.0, TIKA-4794)
 
 - All keys are `Property` constants — no bare `String` keys (`metadata-string-keys.json` retired).
+  Sole exception: `TikaCoreProperties.EMBEDDED_RESOURCE_TYPE_KEY`, an internal building block
+  that constructs the `Property` name for `EMBEDDED_RESOURCE_TYPE` — not an independent key.
 - Tika-coined prefix is `tk:` (`X-TIKA:` is legacy); kebab-case, no underscores.
 - External-standard names verbatim, *including* the standard's prefix: `dc:`, `xmp:`, `cp:`, `extended-properties:`.
 - HTTP has no namespace → `Content-Type`, `Content-Encoding`, `Location` stay bare (no `http:`).

CHANGES.txt

@@ -2,6 +2,34 @@ Release 4.0.0 - ???
 
   BREAKING CHANGES
 
+   * Metadata's reserved tk: (and legacy X-TIKA:) namespace is now a trust
+     boundary for String-keyed writes. A String write to a reserved key throws
+     IllegalArgumentException instead of 3.x's silent success or silent drop,
+     and the public Property factories reject reserved names outright -- a
+     curated tk: constant can only be built from Tika's own package-private
+     factories.
+     Metadata#setAll(Properties) is removed with no replacement; use
+     putAll(Metadata) or individual set/add calls. PassthroughPrefix is
+     renamed to KeyPrefix, and document/tool-derived names are written via the
+     new Metadata#add(KeyPrefix, String, String) route (append-only, with
+     built-in skip-and-WARN bounds on hostile names) or its Instant overload
+     for source-typed dates. A batch of key families was also renamed into
+     namespaced spellings (audio:, ner:, envi:, ogg:streams-, grobid:,
+     iso19115:, gdal:, geotopic:, mif:, idml:, and GeoParser's Optional_NAME<n>
+     fields); see the metadata key migration guide (metadata-changes-4x.adoc)
+     for the full renamed-key table.
+     Metadata no longer implements CreativeCommons, Geographic, HttpHeaders,
+     Message, ClimateForcast, TIFF, or TikaMimeKeys: inherited constants like
+     Metadata.CONTENT_TYPE move to their home interface (HttpHeaders.CONTENT_TYPE,
+     etc.). The TikaMimeKeys and ClimateForcast interfaces are deleted outright;
+     ClimateForecast (corrected spelling) replaces the latter. The Property
+     factories internalClosedChoise/internalOpenChoise/externalClosedChoise/
+     externalOpenChoise are renamed to ...Choice (typo fix, no forwarders), and
+     the dead enum constants PropertyType.STRUCTURE and ValueType.{LOCALE,
+     MIME_TYPE, PROPER_NAME, URL, XPATH} are removed. The remaining metadata
+     API cleanups (cf: prefix, writelimiter package, smaller removals) are
+     enumerated in the migration guide, migrating-to-4x.adoc (TIKA-4816).
+
    * tika-server: the /translate endpoints have been removed (TIKA-4809).
 
    * tika-server: /detect/stream and /language/stream and /language/string have

docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc

@@ -364,10 +364,10 @@ curl -s -X PUT -H "X-Tika-OCRskipOcr: true" -T testPDF.pdf http://localhost:9998
 
 [source,bash]
 ----
-curl -s -X PUT -H "Content-Disposition: attachment; filename=myfile.pdf" -T testPDF.pdf http://localhost:9998/meta/resourceName
+curl -s -X PUT -H "Content-Disposition: attachment; filename=myfile.pdf" -T testPDF.pdf http://localhost:9998/meta/tk:resource-name
 ----
 
-*Expected:* Returns the filename from Content-Disposition header.
+*Expected:* Returns the `tk:resource-name` value taken from the Content-Disposition filename.
 
 == Error Handling
 

docs/modules/ROOT/pages/developers/index.adoc

@@ -22,7 +22,7 @@ with custom parsers, detectors, and other components.
 
 * xref:developers/serialization.adoc[Serialization and Configuration] - JSON configuration,
   @TikaComponent annotation, and creating custom components
-* xref:developers/metadata-keys.adoc[Adding a Metadata Key] - the Property/PassthroughPrefix
+* xref:developers/metadata-keys.adoc[Adding a Metadata Key] - the Property/KeyPrefix
   registry, naming conventions, and regenerating the schema
 
 == Coming Soon

docs/modules/ROOT/pages/developers/metadata-keys.adoc

@@ -16,17 +16,30 @@
 = Adding a Metadata Key
 
 Every metadata key Tika can emit is a `Property` constant (or, for runtime-minted names like scraped
-HTML `<meta>` tags, a `PassthroughPrefix`) — there are no bare `String` keys. That closed/open key
+HTML `<meta>` tags, a `KeyPrefix`) — there are no bare `String` keys naming a population of writable
+metadata. (One `String` constant remains as an exception: `TikaCoreProperties.EMBEDDED_RESOURCE_TYPE_KEY`
+is an internal building block that constructs the `Property` name for `EMBEDDED_RESOURCE_TYPE` — not an
+independent key — and is excluded from the registry's key count for that reason.) That closed/open key
 space is tracked in a generated, build-gated registry, so adding a key involves one extra step beyond
 writing the Java.
 
 == Add the constant
 
-Add the `Property` to its interface as usual:
+A curated `tk:` key can only be minted from inside `org.apache.tika.metadata`, via the
+package-private `reserved*` factories -- a public `Property` factory rejects a `tk:`/`X-TIKA:`
+name at construction time. Add it to `TikaCoreProperties` (or another class in that package)
+as usual:
 
 [source,java]
 ----
-Property MY_NEW_KEY = Property.internalText(TIKA_META_PREFIX + "my-new-key");
+Property MY_NEW_KEY = Property.reservedInternalText(TIKA_META_PREFIX + "my-new-key");
+----
+
+A parser module coining its own key uses a public factory in its own namespace instead:
+
+[source,java]
+----
+Property MY_NEW_KEY = Property.internalText("myformat:my-new-key");
 ----
 
 Naming conventions (frozen for 4.0):
@@ -36,14 +49,34 @@ Naming conventions (frozen for 4.0):
   `cp:`, `extended-properties:`).
 * HTTP headers stay bare — no `http:` namespace (`Content-Type`, `Content-Encoding`, `Location`).
 
+== Document-derived names: declare a KeyPrefix
+
+When the key *names* come from the document or an external tool (custom document properties,
+format-specific attribute names, NER labels), they can't be constants. Declare a `KeyPrefix`
+once, as a `static final` field — never per-parse, never from document text — and write through
+`Metadata#add(KeyPrefix, String, String)`:
+
+[source,java]
+----
+static final KeyPrefix MYFORMAT = KeyPrefix.file("myformat:", "myformat's own header fields");
+...
+metadata.add(MYFORMAT, nameFromDocument, value);              // String value
+metadata.add(MYFORMAT, nameFromDocument, date.toInstant());   // source-typed date
+----
+
+The route is append-only (repeated names accumulate, losslessly transcribing the source) and
+never throws on hostile input: blank, over-length, or flooding names are skipped with a WARN.
+Use `KeyPrefix.file(...)` for names read out of the document, `KeyPrefix.tool(...)` for names
+coined by an external tool or service.
+
 == Regenerate the registry
 
 The registry — three JSON files under `tika-metadata-schema/src/main/resources/`, listing every
 declared key, every open-namespace prefix, and a field-provenance table — is generated from the live
-`Property`/`PassthroughPrefix` declarations, never hand-edited. A committed copy is the reviewable
+`Property`/`KeyPrefix` declarations, never hand-edited. A committed copy is the reviewable
 audit trail (a rename or dropped key shows up as a diff), and CI fails if it's stale.
 
-Run this after adding, renaming, or removing a `Property` or `PassthroughPrefix`:
+Run this after adding, renaming, or removing a `Property` or `KeyPrefix`:
 
 [source,bash]
 ----

docs/modules/ROOT/pages/developers/serialization.adoc

@@ -329,19 +329,21 @@ framework keys.
 
 == Creating a Custom Component
 
-Complete example of a custom metadata filter:
+Complete example of a custom metadata filter. `fieldName` is config-supplied, so it could
+legitimately name a reserved `tk:` key; use `setTrusted` rather than `set` -- filters run
+inside the trusted bracket of `MetadataFilter`, the sanctioned route for a write keyed by
+a name that isn't a compile-time constant:
 
 [source,java]
 ----
 package com.example.tika;
 
 import org.apache.tika.annotation.TikaComponent;
-import org.apache.tika.exception.TikaException;
 import org.apache.tika.metadata.Metadata;
-import org.apache.tika.metadata.filter.MetadataFilter;
+import org.apache.tika.metadata.filter.MetadataFilterBase;
 
 @TikaComponent
-public class UpperCaseFilter implements MetadataFilter {
+public class UpperCaseFilter extends MetadataFilterBase {
 
     private String fieldName = "title";
 
@@ -354,18 +356,19 @@ public class UpperCaseFilter implements MetadataFilter {
     }
 
     @Override
-    public void filter(Metadata metadata) throws TikaException {
+    protected void filter(Metadata metadata) {
         String value = metadata.get(fieldName);
         if (value != null) {
-            metadata.set(fieldName, value.toUpperCase());
+            metadata.setTrusted(fieldName, value.toUpperCase());
         }
     }
 }
 ----
 
 Configure in JSON. Metadata filters are loaded via `parse-context` (they
-implement the `MetadataFilter` interface, which is a `ParseContext`-keyed
-component):
+extend the `MetadataFilter` abstract class -- here via `MetadataFilterBase`,
+whose per-`Metadata` hook suits context-free filters -- which is a
+`ParseContext`-keyed component):
 
 [source,json]
 ----