sincedb_path needs to accept a path not a file

[jameshueston]

Discovered by coolacid in the irc chat today (7/15/14)
https://github.com/jordansissel/ruby-filewatch/blob/master/lib/filewatch/tail.rb#L209
the input file filter sincedb_path looks for a file and not a path.
I couldn't specify a directory for sincedb files to go into and coolacid said this is why.
Ideally, I'd like to see the ability to specify a relative path with a dynamic filename, like: ../sincedb/%{path}.sincedb

The output file filter path already works this way, dynamically. I can grok the inputFilename and then beautifully do:
output {
file {
path => "../output/%{inputFilename}.out"
...

[colinsurprenant]

in what situation would you want to use a dynamic path for the sincedb?

[jordansissel]

I was also present for part of this discussion on IRC.

The problem seems to be some combination of:

• The default sincedb is somewhere under $HOME w/ unpredictable file names, but predictable directory ($HOME).
• The setting sincedb_path is a file, not a directory.
• The "YOu get a file or you get the default directory" is inconsistent. Both settings should be either files or directories, imo, not exclusionary "this one is a directory and this other thing is a file"
• Requiring $HOME set to effect a change on the default directory is inconsistent with the rest of logstash generally being configured via config file.
• $HOME, by name, is completely unrelated to any terms used by the file input plugin.

Some thoughts -

• Make sincedb_path accept a directory. Perhaps hinting at directory usage witha trailing slash, example: sincedb_path => "/some/path/"
• Do validation at startup to verify that two sincedb_paths are not equal iff the paths point to a file.
• More broadly, there's consistency in stateful plugins. file input is different than s3 input and different than multiline filter. Maybe worth providing an API here to do state w/ different backends (local file, remote file (s3), in memory for multiline)

[jordansissel]

My best preference would be to have Logstash globally configured with "All plugins that need to store state should do so in " similar to Elasticsearch's path.data setting. Individual plugins should NOT need such a setting.

[jordansissel]

Future discussion on having a simpler way to describe where the pipeline should do work and store state: #1535

[phutchins]

Did this go anywhere? My issue here is that I cannot supply a wildcard to the file input resource and then specify a sincedb directory. I still want file names to be generated programmatically but want to specify the directory. Even having some sort of way to specify a base of the file name and then a bit in the filename that will expand out to the hash so all sincedb files are unique.

[andrewvc]

@phutchins I'm not sure if we ever resolved this, but your request makes a ton of sense. @jordansissel the link you closed the issue with is to this same issue. Where should it be pointed?

[jordansissel]

No movement yet on any particular new solution since this was closed. Most folks seem OK with setting HOME and calling it a day. I would probably be happy to merge a patch adding directory support to the sincedb_path :)

[phutchins]

Awesome.

I was under the impression that there was a sincedb file for each file which is not the case. I tested it and I get one sincedb file per logstash instance. This works for my current implementation thankfully.

It does not however add the hash as we discussed which would be really nice... I'll see if I can find time to write the patch.

[phutchins]

@jordansissel I just submitted a MR for these changes. Take a look at...

logstash-plugins/logstash-input-file#85

...and let me know what you think of my implementation. Definitely open for input. This should continue to be backwards compatible and will let the @sincedb_path override the new options I added.

I also took a look at the tests and they seem to cover the additional changes well enough to avoid issue.