-
Notifications
You must be signed in to change notification settings - Fork 6
Optionally use ClickHouse DB to log process, run, and file metadata #378
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
This way a single DB entry or Arrow file entry can be made for all channels at once.
The clickhouse-go seems to drop subsecond info when you INSERT a go `time.Time` value into a table, unless you convert to a string. So we do that.
It works by appending binary data, then rewriting the "size" value in the file header.
|
What benefit do we gain by merging this vs just keeping it in a branch/PR until we actually want to test the features? |
I am not sure. My motivation for this PR is simply that both this work and the possible work on simplifying our data writing touch/would touch the same lines of Go. If we're going to change how we write LJH and OFF files, I don't want to make incompatible changes for that purpose and for the DB writer. I'd rather launch the one branch from the other. Of course, that doesn't require us to merge this PR yet. We can talk in person.... |
Not sure you'll be excited to add an optional DB-logging feature, but I do believe that it's feature-complete for a certain first set of tasks. I made enough changes that I'd like to bake this into the main branch. It shouldn't have any effect on computers where DB use was not explicitly asked for in the past.
This PR does not log pulse data, but only:
Again, there are no pulse records stored in the DB--no LJH-like or OFF-like data. I experimented with that possibility, but the performance was not acceptable. (See
branch:clickhouse_pulse_storagefor that code.)