-
Notifications
You must be signed in to change notification settings - Fork 0
andrepl/shotgun
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
shotgun is a unix shell utility which can be used to parallelize long-running processes that operate on data from stdin.
Examples:
lets say we have a fast-growing log file. we also have a program which
reads lines from stdin, does some CPU intensive work, and then inserts
them into a database. if we can parallelize the expensive work, we can
save a lot of time. thats where shotgun comes in.
$ tail -f test.log | shotgun.py -p 4 -c "crunch"
in the above example, shotgun launches 4 instances of the 'crunch'
program. it then reads lines from the tail -f command and will distribute
those lines across the pool of subprocesses, writing to their stdin.
About
unix shell utility to parallelize stream processing.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published