Skip to content

Commit e537ce0

Browse files
tballisontballison
authored
TIKA-1997 -- pkcs detection (#2955)
1 parent bb79091 commit e537ce0

33 files changed

Lines changed: 769 additions & 38 deletions

File tree

‎CHANGES.txt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
Release 4.0.0 - ???
2+
3+
NEW FEATURES
4+
5+
* Content-based detection of ASN.1/DER crypto containers at parse time. An
6+
opt-in Pkcs7Detector surfaces the subtype at detect() time,
7+
but must be enabled via configuration (TIKA-1997).
8+
9+
110
Release 4.0.0-beta-1 - 6/29/2026
211

312
BREAKING CHANGES

‎docs/modules/ROOT/examples/detector-pkcs7.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"detectors": [
3+
{ "pkcs-7-detector": {} },
4+
{ "default-detector": {} }
5+
]
6+
}

‎docs/modules/ROOT/pages/configuration/detectors.adoc‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
//
2+
// Licensed to the Apache Software Foundation (ASF) under one or more
3+
// contributor license agreements. See the NOTICE file distributed with
4+
// this work for additional information regarding copyright ownership.
5+
// The ASF licenses this file to You under the Apache License, Version 2.0
6+
// (the "License"); you may not use this file except in compliance with
7+
// the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing, software
12+
// distributed under the License is distributed on an "AS IS" BASIS,
13+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
// See the License for the specific language governing permissions and
15+
// limitations under the License.
16+
//
17+
18+
= Detector Configuration
19+
20+
A `detectors` list loads *only* the detectors it names — every other detector is dropped.
21+
To add a detector while keeping the default detection chain, include a `default-detector`
22+
entry (the same pattern as `default-parser`; see xref:configuration/index.adoc[Configuration]).
23+
24+
== Enabling the PKCS7/CMS detector
25+
26+
Content detection of ASN.1/DER cryptographic containers is *coarse by default*: a CMS/PKCS7
27+
message is detected as `application/pkcs7-signature`, and the exact CMS subtype
28+
(signed vs. enveloped vs. certs-only) is refined by `Pkcs7Parser` at *parse* time — it is set
29+
on the output content type, the same way `PDFParser` refines a PDF to `application/illustrator`.
30+
31+
If you need that subtype from `Tika.detect()` *without* parsing, enable the opt-in
32+
`pkcs-7-detector`. It is not loaded by default (so the common detect path stays cheap and does
33+
no ASN.1 work on every stream); add it ahead of `default-detector`:
34+
35+
[source,json]
36+
----
37+
include::example$detector-pkcs7.json[]
38+
----
39+
icon:github[] https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/detector-pkcs7.json[View source on GitHub]
40+
41+
[IMPORTANT]
42+
====
43+
List `pkcs-7-detector` *before* `default-detector`. It returns a parameterized
44+
`application/pkcs7-mime; smime-type=...`, which is a *sibling* of the magic detector's
45+
`application/pkcs7-signature` (neither is a specialization of the other). A `CompositeDetector`
46+
replaces the running type only with a strict specialization, so of two sibling results it keeps
47+
the first — the opt-in detector must run first to win.
48+
====
49+
50+
With this configuration, `Tika.detect()` on a signed CMS message returns
51+
`application/pkcs7-mime; smime-type=signed-data` instead of the coarse
52+
`application/pkcs7-signature`. This is verified end-to-end in
53+
link:https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/ConfigExamplesTest.java[`ConfigExamplesTest`].
54+
55+
NOTE: The default (magic-only) path already detects the crypto *families* — PKCS7/CMS,
56+
RFC 5544 timestamped-data, and PKCS12 — so most users do not need this detector. Enable it only
57+
when a routing or downstream decision depends on the CMS subtype before parsing.

‎docs/modules/ROOT/pages/configuration/index.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@ JSON uses the backslash as an escape character, so path options (e.g. `tesseract
131131

132132
=== Other Configuration
133133

134+
* xref:configuration/detectors.adoc[Detectors] — Configuring content (MIME) detection, incl. the opt-in PKCS7/CMS detector
134135
* xref:configuration/digesters.adoc[Digesters] — Computing cryptographic hashes of documents
135136
* xref:configuration/encoding-detectors.adoc[Encoding Detectors] — Configuring charset/encoding detection
136137

‎tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -893,6 +893,15 @@
893893
<mime-type type="application/pkcs7-mime">
894894
<glob pattern="*.p7m"/>
895895
<glob pattern="*.p7c"/>
896+
<!-- CMS content types in the 1.2.840.113549.1.9.16.1.* arc (compressedData .9, authData .2,
897+
authEnvelopedData .23, ...) with the final OID byte masked. timestamped-data (.31) has the
898+
same shape but its own higher-priority magic below. The .7 arc (signed/enveloped/...) is
899+
matched by the pkcs7-signature block. Pkcs7Parser refines all of these to the smime-type. -->
900+
<magic priority="50">
901+
<match value="0x30" offset="0">
902+
<match value="0x060B2A864886F70D0109100100" mask="0xFFFFFFFFFFFFFFFFFFFFFFFF00" type="string" offset="2:6"/>
903+
</match>
904+
</magic>
896905
</mime-type>
897906

898907
<mime-type type="application/pkcs7-signature">
@@ -926,8 +935,13 @@
926935

927936
<mime-type type="application/timestamped-data">
928937
<glob pattern="*.tsd"/>
929-
<magic priority="50">
930-
<match value="0x3080060B2A864886F7" type="string" offset="0"/>
938+
<!-- CMS ContentInfo whose contentType is id-ct-timestampedData (1.2.840.113549.1.9.16.1.31),
939+
matching the full OID across the DER SEQUENCE length forms (short + 1..4-byte long).
940+
Higher priority than the masked pkcs7-mime .9.16.1.* magic so this exact OID wins. -->
941+
<magic priority="60">
942+
<match value="0x30" offset="0">
943+
<match value="0x060B2A864886F70D010910011F" type="string" offset="2:6"/>
944+
</match>
931945
</magic>
932946
</mime-type>
933947

@@ -4867,6 +4881,20 @@
48674881
<mime-type type="application/x-pkcs12">
48684882
<glob pattern="*.p12"/>
48694883
<glob pattern="*.pfx"/>
4884+
<!-- PFX ::= SEQUENCE { version INTEGER (v3), authSafe ContentInfo SEQUENCE, ... }. The
4885+
INTEGER 3 then a SEQUENCE (02 01 03 30) tells it apart from CMS (OID next) and keys
4886+
(version 0). "02 01 03 30" alone also matches an SNMPv3 message (SEQUENCE { INTEGER 3,
4887+
SEQUENCE, ...}), so we additionally require authSafe's ContentInfo type OID: the pkcs7
4888+
arc 1.2.840.113549.1.7.x (id-data for password integrity, id-signedData for public-key),
4889+
last byte masked. Its offset varies with the two SEQUENCE length forms, hence 7:15. -->
4890+
<magic priority="50">
4891+
<match value="0x30" offset="0">
4892+
<match value="0x02010330" offset="2:6">
4893+
<match value="0x06092A864886F70D010701" mask="0xFFFFFFFFFFFFFFFFFFFF00"
4894+
type="string" offset="7:15"/>
4895+
</match>
4896+
</match>
4897+
</magic>
48704898
</mime-type>
48714899
<mime-type type="application/x-pkcs7-certificates">
48724900
<glob pattern="*.p7b"/>

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/ConfigExamplesTest.java‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
*/
1717
package org.apache.tika.config;
1818

19+
import static org.junit.jupiter.api.Assertions.assertEquals;
1920
import static org.junit.jupiter.api.Assertions.assertNotNull;
2021

2122
import java.io.InputStream;
@@ -27,6 +28,10 @@
2728
import org.junit.jupiter.api.io.TempDir;
2829

2930
import org.apache.tika.config.loader.TikaLoader;
31+
import org.apache.tika.detect.Detector;
32+
import org.apache.tika.io.TikaInputStream;
33+
import org.apache.tika.metadata.Metadata;
34+
import org.apache.tika.parser.ParseContext;
3035
import org.apache.tika.parser.Parser;
3136

3237
/**
@@ -94,4 +99,29 @@ public void testTesseractFullConfig() throws Exception {
9499
public void testFullMigrationExample() throws Exception {
95100
loadAndValidate("migration-full-example.json");
96101
}
102+
103+
/** The detectors.adoc example: enabling the opt-in pkcs-7-detector yields the CMS smime-type. */
104+
@Test
105+
public void testPkcs7DetectorConfig() throws Exception {
106+
Detector detector = loadDetectors("detector-pkcs7.json");
107+
assertEquals("application/pkcs7-mime; smime-type=signed-data",
108+
detect(detector, "testPKCS7_signed_data_def.p7m"));
109+
}
110+
111+
private Detector loadDetectors(String resourceName) throws Exception {
112+
try (InputStream is = getClass().getResourceAsStream(EXAMPLES_DIR + resourceName)) {
113+
assertNotNull(is, "Resource not found: " + resourceName);
114+
Path configFile = tempDir.resolve("tika-config.json");
115+
Files.writeString(configFile, new String(is.readAllBytes(), StandardCharsets.UTF_8),
116+
StandardCharsets.UTF_8);
117+
return TikaLoader.load(configFile).loadDetectors();
118+
}
119+
}
120+
121+
private String detect(Detector detector, String resource) throws Exception {
122+
try (TikaInputStream tis = TikaInputStream.get(
123+
getClass().getResourceAsStream("/test-documents/" + resource))) {
124+
return detector.detect(tis, new Metadata(), new ParseContext()).toString();
125+
}
126+
}
97127
}

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/mime/TestMimeTypes.java‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1242,10 +1242,9 @@ public void testCertificatesKeys() throws Exception {
12421242
// Parameters only have PEM form, always need data
12431243
assertTypeByData("application/x-x509-dsa-parameters", "testDSAPARAMS.pem");
12441244
assertTypeByData("application/x-x509-ec-parameters", "testECPARAMS.pem");
1245-
// PKCS12 wrappers of Certs+Keys cannot currently be identified
1246-
// Once solved, see TIKA-3784, ought to work for name or data
1247-
//assertType("application/x-pkcs12", "testRSAKEYandCERT.p12");
1248-
//assertTypeByData("application/x-pkcs12", "testRSAKEYandCERT.p12"); // pass=tika
1245+
// PKCS12 is now identified by content via the PFX version-3 anchor (TIKA-1997/TIKA-3784)
1246+
assertType("application/x-pkcs12", "testRSAKEYandCERT.p12");
1247+
assertTypeByData("application/x-pkcs12", "testRSAKEYandCERT.p12");
12491248
assertTypeByData("application/x-java-keystore", "KeyStore.jks");
12501249
}
12511250

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/crypto/Pkcs7EndToEndTest.java‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
package org.apache.tika.parser.crypto;
18+
19+
import static org.junit.jupiter.api.Assertions.assertEquals;
20+
import static org.junit.jupiter.api.Assertions.assertTrue;
21+
22+
import java.util.List;
23+
24+
import org.junit.jupiter.api.Test;
25+
26+
import org.apache.tika.TikaTest;
27+
import org.apache.tika.metadata.Metadata;
28+
import org.apache.tika.metadata.TikaCoreProperties;
29+
30+
public class Pkcs7EndToEndTest extends TikaTest {
31+
32+
/**
33+
* TIKA-1997: a CMS-signed XML is detected as PKCS7, routed to Pkcs7Parser, refined to
34+
* signed-data, and its inner XML payload is unwrapped and its content extracted.
35+
*/
36+
@Test
37+
public void testSignedXmlIsExtracted() throws Exception {
38+
List<Metadata> metadataList = getRecursiveMetadata("test.xml.p7m");
39+
// the container is refined from the coarse detected type to the CMS subtype
40+
assertEquals("application/pkcs7-mime; smime-type=signed-data",
41+
metadataList.get(0).get(Metadata.CONTENT_TYPE));
42+
// the signed XML payload is unwrapped and extracted
43+
StringBuilder content = new StringBuilder();
44+
for (Metadata m : metadataList) {
45+
String c = m.get(TikaCoreProperties.TIKA_CONTENT);
46+
if (c != null) {
47+
content.append(c);
48+
}
49+
}
50+
assertContains("TEST_APP", content.toString());
51+
assertContains("ESTRAZIONE", content.toString());
52+
}
53+
54+
/** CMS compressedData (RFC 3274) is inflated and its inner payload extracted (here a PDF). */
55+
@Test
56+
public void testCompressedContentIsExtracted() throws Exception {
57+
List<Metadata> metadataList = getRecursiveMetadata("testPKCS7_compressed_def_long.p7z");
58+
assertEquals("application/pkcs7-mime; smime-type=compressed-data",
59+
metadataList.get(0).get(Metadata.CONTENT_TYPE));
60+
boolean pdf = metadataList.stream()
61+
.map(m -> m.get(Metadata.CONTENT_TYPE))
62+
.anyMatch(ct -> ct != null && ct.startsWith("application/pdf"));
63+
assertTrue(pdf, "expected an embedded application/pdf inflated from the compressed payload");
64+
}
65+
}

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/detector-pkcs7.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"detectors": [
3+
{ "pkcs-7-detector": {} },
4+
{ "default-detector": {} }
5+
]
6+
}

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/test-documents/test.xml.p7m‎

2.45 KB
Binary file not shown.

0 commit comments

Comments
 (0)