Skip to content

Zipped filewatching #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

rajsahae
Copy link

Created a filewatch for gzipped text files. Necessary to implement the file_gz logstash plugin that I'm working on. Seems to be working well for me so far. Thought you may want it as part of the gem.

Raj Sahae added 2 commits February 27, 2014 11:14
Initially, I started refactoring everything but as this isn't my gem I
ended up simply inheriting FileWatch::Tail as to maintain backwards
compatibility as much as possible.

Unfortunately, if Tail ever changes, Gzip will need to be retested.
@jordansissel
Copy link
Owner

Howdy!

Thanks for your effort on this.

It might be helpful to know the future roadmap a bit for logstash to see how this all fits together.

In logstash 1.2.0 we added the codec system which allows us to decouple "where data comes from" from "how data is serialized". In my mind, the best way forward with this codec system is:

  • fix filewatch to emit chunks of bytes read from a file instead emitting lines
  • add a gzip_lines codec to logstash

This would allow filewatch to do what it is good at (stream file data) and a separate, small 'gzip' codec worry about decompression.

With that said, I think this patch is a step away from the codec system, not towards it.

@rajsahae
Copy link
Author

Ok, that all sounds fine and dandy. Feel free to close this pull request then if you think it's a step backwards. I can just use it locally until codec is mature enough to handle this use case.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants