Extract key value pairs during demuxing #1410
Replies: 2 comments 9 replies
-
By this, do you mean exporting search results as mentioned in the following link? https://go2docs.graylog.org/current/interacting_with_your_log_data/export_search_results.html If so, is the export format being used the "Log file/plain text" one? Perhaps life would be easier if the export was NDJSON and the demux support was updated to work with that?
Yes, the keys at the end, without a clear demarcation of where the message is, makes life kinda hard. |
Beta Was this translation helpful? Give feedback.
-
|
You are totally right, it would probably make much more sense to go back to the source on this .. I'll see what I can find out and report back. |
Beta Was this translation helpful? Give feedback.
-
|
I went back and requested the logs to be exported as NDJSON, which has improved the situation a lot! I created a log format and get it all nicely structured to a certain extent. All of my remaining complaints fall squarely into the category "garbage in, garbage out", and will probably need manual cleanup. The log lines that I now get look like this (newlines for readability): {
"kubernetes_pod_ip": "10.0.0.1",
"source": "k8shost1",
"message": "2025-03-14 06:06:28,629 INFO ...",
"kubernetes_host": "k8shost1",
"kubernetes_pod_name": "hdfs-namenode-default-0",
"timestamp": "2025-03-14T07:06:29.568+01:00"
}The issues with this are:
I have tried updating the timestamp and loglevel via sql with something like ;update
graylog_json
set
log_level = regexp_match('^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2},\d+\s+([A-Z]+)\s+.*', log_body)which didn't throw an error, but also didn't work. I suspect it is due to the fields My current idea would be to just pipe the input file through I just wanted to see if there are more elegant ways of achieving this in lnav that I have missed during my investigation? The ideal solution that ocurred to me was that I'd like to be able to specify "calculated" fields in a json logformat, which I can fill with capture groups from regexs or similar things ... but I am aware that I am probably far of the beaten path with what I am doing here and would probably remain the sole user for this :) Update: Does what I was looking for and adds |
Beta Was this translation helpful? Give feedback.
-
Ah, sorry, I think I misunderstood your meaning there.. that sounds great! While I'd personally be happy to have a single field as the mux_id, I'm sure there are use cases for multiple fields .. random brainfart with almost no thought put into it: maybe a generated |
Beta Was this translation helpful? Give feedback.
-
|
seems really minimal, but should serve its purpose, unless I missed other places that need updating.. |
Beta Was this translation helpful? Give feedback.
-
I've made a first pass at this and it's available on the top-of-tree. I tested with the following file, which I think is similar to what you are working with: https://github.com/tstack/lnav/blob/master/test/logfile_muxed_syslog.0 The configuration for this JSON-demux is in the builtin config: Lines 45 to 52 in 957361c I imagine the functionality still needs some tweaks. When you find some time, let me know, thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Hey Tim, I played around with it a bit and it works well in principle, I think I just have a few questions to better understand how it fits together with the log formats.. Before the json demuxing feature I came up with this log format which worked well for my use case:
I'm currently playing around with an external script to join multi-line logs that were divorced by the log gathering mechanism, but I think this probably too much of an edge case for lnav and will remain something for logs to be piped through before hitting lnav. As for the json demuxing config, I used this: "demux-json": {
"graylog": {
"enabled": true,
"timestamp": "timestamp_parsed",
"mux_id": "kubernetes_pod_name",
"body": "message"
}
} And while that worked well in principle and showed the value of I now get "out of order" warnings again, because the timestamp parsed from the log line isn't what the file is sorted by. I presume that I could fix this by patching the The extra fields from the json line are gone, I think because by specifying the body message only that is passed on and scheduled for processing? Maybe taking a step back and locking at the larger picture, is my understanding correct, that the demux-json config "replaces" json log formats (unless I stick json in the content of the "body" field? ). If this is correct, I presume this is due to the way that demuxing is handled in the codebase. But it does feel a bit counter-intuitive to me, because I now loose the ability to define the log format for json (extract fields and define line format mostly) - plus, some things are duplicated in the demux config from the json logformat config - while some things are missing. It think it would have felt more natural to me to add a field "demux"-id to the logformat config and keep the rest unchanged. |
Beta Was this translation helpful? Give feedback.
-
The demux should split the single input file into separate files and then those files are treated like normal files that go through the log format detection steps.
I wouldn't think the
Hmm, that is surprising to me. When you turn on the parser details (press When the demux is happening, the timestamp from the input file is included with the lines in the output files. But, those timestamps should only used be used if the output lines don't include a timestamp of their own. Since your input has real log messages, the log format should extract the timestamp from the message and that should be used for sorting.
Ah, I didn't realize you were interested in bubbling that up to the display.
The extra fields are associated with the output file. You can run the following query to view them: I'm not sure the best way to add that to the display. Maybe the demux definition should be able to specify the format of what is added to the display? I could make it available through SQL so the
It's not meant to replace JSON log formats. It's just a different input format supported by the demuxer. Originally, only plain text that matched a pattern was supported and now the functionality was extended for a different style of input. I could also see adding CSV as another format recognized by the demuxer.
The "timestamp found in the log body" is what should be used. We need to debug why it isn't. |
Beta Was this translation helpful? Give feedback.
-
I have a question on how to treat key value pairs which occur in a file that needs to be demuxed.
I'll explain the scenario first, to give a bit of context, which may make it a bit easier to understand why I would ever consider something like this :)
We run distributed systems on kubernetes and logs are gathered in a central Greylog, by reading the console output of pods.
So in Greylog, some metadata is captured, but this is all based on kubernetes
Plus, the actual line that was logged (I'm not even gonna start talking about multiline handling here, that went out the window a long time ago).
Now, if something breaks that needs investigating, we can ask for logs to be exported from Greylog, and we'll receive them in roughly this format:
2024-12-11T05:32:47.143+01:00 source=hostname1 <the actual log line> kubernetes_pod_ip=<ip> kubernetes_pod_name=<podname> kubernetes_pod_id=<podip> kubernetes_host=<host>With the key=value pairs at the end being somewhat flexible, they can change order, some can be missing, there can be extra fields .. not really predictable, based on what the person who does the export clicks..
The first timestamp is from greylog, when the log line was received, the actually interesting timestamp will be in the body though, so we can more or less ignore that.
I have written a demux pattern to target this, which looks like this:
So basically what it does is remove the prefix (greylog timestamp and source=...) and uses the entire rest of the line as the log body.
In this body it then looks for a key=value pair where the key is "kubernetes_pod_name" and extracts that, but keeps all key=value pairs as part of the body.
This way all key value pairs will be extracted by lnav from the log line and shown in the pretty view (see screenshot).
However, it does change the logline, because these fields will now be displayed at the end of the line, but don't really "belong" there, I'd much rather remove them during demuxing.
What I'd like to happen is to extract these key value pairs during the demux stage. I know I can do that with named capture groups in the demux pattern, but I found no way of doing this for keys where I do not know the name up front (dynamic capture group names I guess..).
So basically I'd need to include every possible key that can occur in the regex, but I'd rather just have the exact same behavior that lnav already does when applying the format in the subsequent step do demuxing.
I am probably doing a horrible way of explaining this .. hopefully you can make a rough guess at what I am trying to say and I can elaborate further on questions :)
Beta Was this translation helpful? Give feedback.
All reactions