Description
Problem
The Processing Engine in InfluxDB 3 can’t write files directly to object storage, despite object storage being the only supported persistence layer in diskless deployments. This limits use cases like EDR durability, report generation, or plugin outputs that require persisted file storage. While direct uploads over the network are possible, they’re cumbersome and require custom handling outside the engine or plugin system.
Proposed Solution
Enable plugins to output files to a designated location in object storage. This capability should support both local (node-level) and global (cluster-level) file outputs, depending on the deployment type (Core or Enterprise).
To visually explain the hierarchy, the file structure in Core might look like:
process_outputs /
- node_id /
-- [output_files]
In Enterprise:
process_outputs /
- cluster_id /
-- node_id /
--- [output_files]
-- node_id /
--- [output_files]
-- [global_output_files]
Acceptance Criteria
1. Plugin File Writes
Writes are only permitted to object storage with this functionality, not local file storage (unless if file storage is being used as the object store structure).
2. Scoped Directory Structure
Core: files are written to the node’s section only
Enterprise: files can be written to the node’s section or the cluster’s section as well.
3. Write Boundaries
Plugins should not be able to define where they want to write. They should only be able to write to their node or to the global cluster, which should be defined API functions.
A plugin file write should have three modes for writing files with the same name:
Auto -- append the time on the end of the file name (default), log [file name] saved to object store.
- Example: output.json → output-20250328T141230Z.json
Reject -- don't write the file to the object store, log [file name] not saved to object store. File name already exists.
Replace -- overwrite the file in the object store, log [file name] saved to object store, overwriting previous file.
4. File Visibility and Durability
Files are not ingested into the database or queryable via SQL — these files should be treated as completely separate outputs from the database.