Skip to content

Commit b5c52d4

Browse files
tballisontballison
authored
TIKA-4877: improve tmp file handling (#3125)
1 parent 4f0e11b commit b5c52d4

17 files changed

Lines changed: 452 additions & 163 deletions

File tree

‎CHANGES.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
11
Release 4.1.0 - unreleased
22

3+
* Temp files follow -Djava.io.tmpdir on the parent JVM (Tika, its
4+
libraries, and forks all honor it); TikaLoader fails at config load
5+
if it is unusable. pipes.tempDirectory is deprecated for removal in
6+
5.0: it only covered the forks. Do not use tmpfs: spool size is
7+
bounded by input, and an orphaned fork dir pins RAM (TIKA-4877).
8+
9+
* A fork whose parent dies deletes its own temp dir; the parent
10+
surfaces a fork's hs_err log before every delete. Failure-path temp
11+
file leaks fixed in PDFBoxRenderer, PopplerRenderer, truncated RTF,
12+
and MarianTranslator (TIKA-4877).
13+
314
* tika-eval Profile/Compare speedups: single-pass URL/mail stripping
415
replaces the bounded regexes in langdetect preprocessing (same output,
516
17-290x faster on web text), the default H2 db URL drops MVStore chunk

‎docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -576,7 +576,7 @@ xref:pipes/cpu-sizing.adoc[Forked-JVM CPU and Heap Sizing].
576576
=== Performance Optimizations
577577
578578
* TCP_NODELAY enabled for reduced latency on small requests
579-
* Configurable temp directory for RAM disk optimization (`pipes.tempDirectory`)
579+
* Temporary files go to `-Djava.io.tmpdir`; forks get private subdirectories under it (see xref:pipes/configuration.adoc#_where_temporary_files_go[Where temporary files go]; `pipes.tempDirectory` is deprecated, and do not use a RAM disk)
580580
581581
== Advanced: Shared Server Mode
582582

‎docs/modules/ROOT/pages/pipes/configuration.adoc‎

Lines changed: 29 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -57,14 +57,37 @@ how many forked JVMs to run, timeouts, memory management, and parse behavior.
5757

5858
|`tempDirectory`
5959
|_system default_
60-
|Directory for temporary files. Each fork gets a subdirectory here, and the fork's whole `java.io.tmpdir` points at it, so anything the fork spools while parsing, unpacked embedded files and JVM crash logs all land inside. It does *not* cover the host side: tika-server spools over-threshold request bodies into its own input temp directory, and `PipesForkParser` into the calling JVM's `java.io.tmpdir`. Consider a RAM-backed filesystem (e.g., `/dev/shm`) for better performance -- but see the caveat below.
60+
|*Deprecated since 4.1, removal planned for 5.0.* Set `-Djava.io.tmpdir` on the parent JVM instead; see below. Still honored: each fork gets a subdirectory here, and the fork's whole `java.io.tmpdir` points at it. It never covered the host side -- tika-server's request-body spool and `PipesForkParser` use the parent's `java.io.tmpdir` -- nor any library that calls `File.createTempFile` itself.
6161
|===
6262

63-
The parent deletes a fork's subdirectory when that fork is torn down or fails to start, so a
64-
crashing fork does not accumulate them. A parent killed abruptly (`SIGKILL`, container stop)
65-
cannot, and its subdirectories survive. On a RAM-backed filesystem those leaks consume memory
66-
rather than disk, and `/dev/shm` is commonly sized at half of RAM -- so if you point
67-
`tempDirectory` at one, sweep it on service start.
63+
=== Where temporary files go
64+
65+
Set `-Djava.io.tmpdir` on the parent JVM. It has to be a launch flag because of Tika's
66+
dependencies: POI, PDFBox and the rest create temp files through the JDK, which reads
67+
`java.io.tmpdir` once at JVM start, so nothing Tika sets at runtime reaches them.
68+
Tika, every library it calls, and its forked pipes servers all honor the flag: each fork gets a private subdirectory under it (its whole
69+
`java.io.tmpdir` points there, so spooled input, unpacked embedded files and JVM crash logs
70+
all land inside), and the parent deletes that subdirectory when the fork is torn down.
71+
Tika checks the directory exists and is writable at config load and refuses to start
72+
otherwise, rather than failing on the first document that spools.
73+
74+
If the parent dies, a surviving fork deletes its own subdirectory as it exits. Only when the
75+
whole process family is killed at once (`SIGKILL` of the group, container stop) do
76+
subdirectories survive, with whatever the forks were spooling inside; Tika never deletes
77+
directories another process created. Point `java.io.tmpdir` at a disk-backed directory on a
78+
volume where filling it does not take out the OS, and apply your own retention to
79+
`pipes-server-*` there.
80+
81+
[WARNING]
82+
====
83+
*DO NOT USE tmpfs* (`/dev/shm`, any RAM-backed filesystem) for `java.io.tmpdir`.
84+
85+
Spool size is bounded by the input, not by any Tika setting: one large archive expanding
86+
into tmpfs can exhaust memory for every process on the host or get a container evicted,
87+
and a fork directory orphaned by a killed parent pins that RAM until someone deletes it.
88+
A slow run is recoverable; a lost host is not. If you must, give tmpfs a dedicated mount
89+
with `size=` so the failure is a refused write rather than a lost host.
90+
====
6891

6992
=== Embedded-object cache memory budget
7093

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/rtf/jflex/RTFEmbeddedHandler.java‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
*/
1717
package org.apache.tika.parser.microsoft.rtf.jflex;
1818

19+
import java.io.Closeable;
1920
import java.io.IOException;
2021
import java.util.concurrent.atomic.AtomicInteger;
2122

@@ -26,6 +27,7 @@
2627
import org.apache.tika.exception.TikaException;
2728
import org.apache.tika.extractor.EmbeddedDocumentExtractor;
2829
import org.apache.tika.extractor.EmbeddedDocumentUtil;
30+
import org.apache.tika.io.TemporaryResources;
2931
import org.apache.tika.io.TikaInputStream;
3032
import org.apache.tika.metadata.HttpHeaders;
3133
import org.apache.tika.metadata.Metadata;
@@ -41,7 +43,7 @@
4143
* {@link RTFPictStreamParser}) so that large embedded objects are written
4244
* to temp files rather than buffered entirely in memory.</p>
4345
*/
44-
public class RTFEmbeddedHandler {
46+
public class RTFEmbeddedHandler implements Closeable {
4547

4648
private final ContentHandler handler;
4749
private final ParseContext context;
@@ -254,4 +256,12 @@ private static int hexValue(int ch) {
254256
return 10 + (ch - 'A');
255257
}
256258
}
259+
260+
/** Closes a stream parser left open by a document that ended inside its group. */
261+
@Override
262+
public void close() throws IOException {
263+
TemporaryResources.closeAll(objParser, pictParser);
264+
objParser = null;
265+
pictParser = null;
266+
}
257267
}

‎tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/rtf/jflex/RTFHtmlDecapsulator.java‎

Lines changed: 124 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -75,150 +75,155 @@ public String extract(byte[] rtfBytes) throws IOException, SAXException, TikaExc
7575
boolean inHtmlTag = false;
7676

7777
RTFToken tok;
78-
while ((tok = tokenizer.yylex()) != null) {
79-
RTFTokenType type = tok.getType();
80-
if (type == RTFTokenType.EOF) {
81-
break;
82-
}
78+
// EOF inside {\object or {\pict leaves a stream parser open
79+
try {
80+
while ((tok = tokenizer.yylex()) != null) {
81+
RTFTokenType type = tok.getType();
82+
if (type == RTFTokenType.EOF) {
83+
break;
84+
}
8385

84-
// Flush pending bytes before charset-changing events
85-
if (type == RTFTokenType.GROUP_CLOSE
86-
|| (type == RTFTokenType.CONTROL_WORD && "f".equals(tok.getName())
87-
&& tok.hasParameter())) {
88-
flushPendingBytes(pendingBytes, html, state);
89-
}
86+
// Flush pending bytes before charset-changing events
87+
if (type == RTFTokenType.GROUP_CLOSE
88+
|| (type == RTFTokenType.CONTROL_WORD && "f".equals(tok.getName())
89+
&& tok.hasParameter())) {
90+
flushPendingBytes(pendingBytes, html, state);
91+
}
9092

91-
boolean consumed = state.processToken(tok);
93+
boolean consumed = state.processToken(tok);
9294

93-
// Embedded handler processes objdata/pict/sp in the same pass
94-
if (!consumed) {
95-
RTFGroupState closingGroup =
96-
(type == RTFTokenType.GROUP_CLOSE) ? state.getLastClosedGroup() : null;
97-
try {
98-
embHandler.processToken(tok, state, closingGroup);
99-
} catch (TikaException | IOException e) {
100-
// don't let a bad embedded object kill decapsulation
95+
// Embedded handler processes objdata/pict/sp in the same pass
96+
if (!consumed) {
97+
RTFGroupState closingGroup =
98+
(type == RTFTokenType.GROUP_CLOSE) ? state.getLastClosedGroup() : null;
99+
try {
100+
embHandler.processToken(tok, state, closingGroup);
101+
} catch (TikaException | IOException e) {
102+
// don't let a bad embedded object kill decapsulation
103+
}
101104
}
102-
}
103105

104-
RTFGroupState group = state.getCurrentGroup();
106+
RTFGroupState group = state.getCurrentGroup();
105107

106-
// Skip tokens that are part of objdata/pict hex streams
107-
if (!consumed && (group.objdata || group.pictDepth > 0)) {
108-
continue;
109-
}
110-
111-
switch (type) {
112-
case GROUP_OPEN:
113-
sawIgnorable = false;
114-
break;
115-
116-
case GROUP_CLOSE:
117-
if (inHtmlTag && state.getDepth() < htmlTagDepth) {
118-
flushPendingBytes(pendingBytes, html, state);
119-
inHtmlTag = false;
120-
htmlTagDepth = -1;
121-
}
122-
break;
108+
// Skip tokens that are part of objdata/pict hex streams
109+
if (!consumed && (group.objdata || group.pictDepth > 0)) {
110+
continue;
111+
}
123112

124-
case CONTROL_SYMBOL:
125-
if (tok.getChar() == '*') {
126-
sawIgnorable = true;
127-
}
128-
if (!foundHtmlTag || inHtmlRtfSkip) {
113+
switch (type) {
114+
case GROUP_OPEN:
115+
sawIgnorable = false;
129116
break;
130-
}
131-
if (inHtmlTag || htmlTagDepth == -1) {
132-
char sym = tok.getChar();
133-
if (sym == '{' || sym == '}' || sym == '\\') {
117+
118+
case GROUP_CLOSE:
119+
if (inHtmlTag && state.getDepth() < htmlTagDepth) {
134120
flushPendingBytes(pendingBytes, html, state);
135-
html.append(sym);
121+
inHtmlTag = false;
122+
htmlTagDepth = -1;
136123
}
137-
}
138-
break;
139-
140-
case CONTROL_WORD:
141-
if (consumed) {
142124
break;
143-
}
144-
String name = tok.getName();
145125

146-
if ("fromhtml".equals(name)) {
147-
foundFromHtml = true;
148-
break;
149-
}
150-
if ("htmltag".equals(name) && sawIgnorable) {
151-
if (!foundFromHtml) {
126+
case CONTROL_SYMBOL:
127+
if (tok.getChar() == '*') {
128+
sawIgnorable = true;
129+
}
130+
if (!foundHtmlTag || inHtmlRtfSkip) {
152131
break;
153132
}
154-
foundHtmlTag = true;
155-
flushPendingBytes(pendingBytes, html, state);
156-
inHtmlTag = true;
157-
htmlTagDepth = state.getDepth();
158-
break;
159-
}
160-
if ("htmlrtf".equals(name)) {
161-
flushPendingBytes(pendingBytes, html, state);
162-
inHtmlRtfSkip = !(tok.hasParameter() && tok.getParameter() == 0);
163-
break;
164-
}
165-
if (!foundHtmlTag || inHtmlRtfSkip) {
133+
if (inHtmlTag || htmlTagDepth == -1) {
134+
char sym = tok.getChar();
135+
if (sym == '{' || sym == '}' || sym == '\\') {
136+
flushPendingBytes(pendingBytes, html, state);
137+
html.append(sym);
138+
}
139+
}
166140
break;
167-
}
168-
if (inHtmlTag || htmlTagDepth == -1) {
169-
flushPendingBytes(pendingBytes, html, state);
170-
switch (name) {
171-
case "par":
172-
case "pard":
173-
html.append('\n');
174-
break;
175-
case "tab":
176-
html.append('\t');
177-
break;
178-
case "line":
179-
html.append("<br>");
180-
break;
181-
default:
182-
break;
141+
142+
case CONTROL_WORD:
143+
if (consumed) {
144+
break;
183145
}
184-
}
185-
break;
146+
String name = tok.getName();
186147

187-
case HEX_ESCAPE:
188-
if (consumed || !foundHtmlTag || inHtmlRtfSkip) {
148+
if ("fromhtml".equals(name)) {
149+
foundFromHtml = true;
150+
break;
151+
}
152+
if ("htmltag".equals(name) && sawIgnorable) {
153+
if (!foundFromHtml) {
154+
break;
155+
}
156+
foundHtmlTag = true;
157+
flushPendingBytes(pendingBytes, html, state);
158+
inHtmlTag = true;
159+
htmlTagDepth = state.getDepth();
160+
break;
161+
}
162+
if ("htmlrtf".equals(name)) {
163+
flushPendingBytes(pendingBytes, html, state);
164+
inHtmlRtfSkip = !(tok.hasParameter() && tok.getParameter() == 0);
165+
break;
166+
}
167+
if (!foundHtmlTag || inHtmlRtfSkip) {
168+
break;
169+
}
170+
if (inHtmlTag || htmlTagDepth == -1) {
171+
flushPendingBytes(pendingBytes, html, state);
172+
switch (name) {
173+
case "par":
174+
case "pard":
175+
html.append('\n');
176+
break;
177+
case "tab":
178+
html.append('\t');
179+
break;
180+
case "line":
181+
html.append("<br>");
182+
break;
183+
default:
184+
break;
185+
}
186+
}
189187
break;
190-
}
191-
if (inHtmlTag || htmlTagDepth == -1) {
192-
pendingBytes.write(tok.getHexValue());
193-
}
194-
break;
195188

196-
case UNICODE_ESCAPE:
197-
if (!foundHtmlTag || inHtmlRtfSkip) {
189+
case HEX_ESCAPE:
190+
if (consumed || !foundHtmlTag || inHtmlRtfSkip) {
191+
break;
192+
}
193+
if (inHtmlTag || htmlTagDepth == -1) {
194+
pendingBytes.write(tok.getHexValue());
195+
}
198196
break;
199-
}
200-
if (inHtmlTag || htmlTagDepth == -1) {
201-
flushPendingBytes(pendingBytes, html, state);
202-
int cp = tok.getParameter();
203-
if (Character.isValidCodePoint(cp)) {
204-
html.appendCodePoint(cp);
197+
198+
case UNICODE_ESCAPE:
199+
if (!foundHtmlTag || inHtmlRtfSkip) {
200+
break;
205201
}
206-
}
207-
break;
202+
if (inHtmlTag || htmlTagDepth == -1) {
203+
flushPendingBytes(pendingBytes, html, state);
204+
int cp = tok.getParameter();
205+
if (Character.isValidCodePoint(cp)) {
206+
html.appendCodePoint(cp);
207+
}
208+
}
209+
break;
208210

209-
case TEXT:
210-
if (consumed || !foundHtmlTag || inHtmlRtfSkip) {
211+
case TEXT:
212+
if (consumed || !foundHtmlTag || inHtmlRtfSkip) {
213+
break;
214+
}
215+
if (inHtmlTag || htmlTagDepth == -1) {
216+
flushPendingBytes(pendingBytes, html, state);
217+
html.append(tok.getChar());
218+
}
211219
break;
212-
}
213-
if (inHtmlTag || htmlTagDepth == -1) {
214-
flushPendingBytes(pendingBytes, html, state);
215-
html.append(tok.getChar());
216-
}
217-
break;
218220

219-
default:
220-
break;
221+
default:
222+
break;
223+
}
221224
}
225+
} finally {
226+
embHandler.close();
222227
}
223228

224229
flushPendingBytes(pendingBytes, html, state);

0 commit comments

Comments
 (0)