Feature: Map Graylog fields to Jira Fields

[123dev]

Hi Gerd,
This is a future feature / enhancement consideration that could further enrich this great plugin.
Consider it at your leisure and don't feel obligated to add.

It would be useful to have a way of mapping Graylog field values to Jira field values.
A perfect example is:
Affects Version
We have a field in Graylog that captures the current version of the Application.
If there was a way to use this value to automatically populate Affects Version in Jira, it would cut down on manual ticket updates.

To keep this feature generic, perhaps add one more row to the settings page

Graylog-To-Jira-Mappings (just an example)
where we can enumerate a list of Graylog Field Name and Jira Field Name pairs (it is perfectly ok to use something similar to the graylog_md5 if the jira field is Custom Field, example: customfield_10501)

[LAST_MESSAGE:GraylogField1][JIRA:FIELD1], [LAST_MESSAGE:GraylogField2][JIRA:FIELD2], [LAST_MESSAGE:GraylogField3][JIRA:FIELD3] ...

Thanks

[magicdude4eva]

Great idea. I have implemented this as part of 1.0.7 and I have done some very basic testing, as I do not have any use cases or any data I could map within a Graylog message.

I would appreciate it if you could help testing it - I do log some extra information into the Graylog log if things go wrong and hopefully this will be enough to troubleshoot.

In theory you should be able to literally now set any field within JIRA from the Graylog message-field value.

A few possible caveats:

• The JIRA field name is case-sensitive (i.e. https://github.com/rcarz/jira-client/blob/master/src/main/java/net/rcarz/jiraclient/Field.java#L120)
• Some JIRA fields require to be Iterable - by the looks of it any field where you can specify multiple values (such as versions, fixVersions) - for those cases you need to append #i to the JIRA-fieldname in the configuration
• I am using the #?-notation to preempt any other data-type issues (I am unsure what will happen if you pass in a date or any other data-type from Graylog to JIRA) and we can build this up if you find other issues.
• For now all Graylog fields are treated as String. I am not sure if JIRA is able to convert a String to another data-type.

Affects Version
Your above use-case to auto-populate the version is really simple now. If your Graylog message contains a field "myapplicationversion" your config for jira_graylog_message_field_mapping would be as follows:

myapplicationversion=versions#i

You can have multiple mappings by separating them with a comma as a delimiter.

[123dev]

Hi Gerd,

Awesome for the quick turnaround.
I have to apologize to you.
I was under the wrong impression that our Graylog field would map 1 to 1 to the Jira field for "Affects Version"
It turns out that I need to do some transformation so that Jira would accept it, hence I'm exploring the best way of getting that before I could fully test this feature.

• Using extractors in graylog (does not work if values need to be picked up from multiple fields)
• Using drool in graylog (probably the best)
• Changing Jira Affects Version values to match (ouch, not likely)

And then there is the burden of yet one more field in graylog to store this.

I did some basic tests, just to see it working, and so far the only issue I ran into is when using a field with multiple values (labels).

This is what I tried

application=labels#i

I also had set JIRA Labels setting to graylog

Are you doing all of this in one step during creation? or you use update to do the mappings?
If it is an update, then fields with multivalue need to be appended rather than replaced, I'm not even sure this option is available to you.

If it is during creation, I can see why this would be the case, as it probably does not take into account special fileds such as labels / components or any other multi-value field that the plugin itself sets during the creation.
Not a big issue at all, I'm just reporting it.

I promise I will test it more thoroughly once I get to address our 1-1 mapping.

Thanks again for all your work.

[magicdude4eva]

Thanks for the initial feedback. I basically prepare the JIRA issue and then create it in one go. This means that the auto-mapping would overwrite for example labels or components you have set via the plugin configuration (this is not an issue, I could very well just go and merge the plugin-configured labels/components and any auto-mapping of the same JIRA field)

I would think that you have to store the transformed field into Graylog as the stream-alert would fire much later. Does this only affect your version? Perhaps give me some examples of what is logged in Graylog and what is in JIRA - possibly we could code some generic transformation - i.e. if you supply some "transformation regex":

graylogversionfield=versions#i-SOMEREGEXTOTRANSFORMTHEGRAYLOGFIELDVALUE

[123dev]

Thanks Gerd,

I could very well just go and merge the plugin-configured labels/components and any auto-mapping of the same JIRA field)

Yeah that would fix that issue, or if you allow non field text to be included then I can set the label graylog in JIRA/Graylog field mapping
Not sure how you would specify it though, it gets tricky to pass multi value to a jira field, you probably need to rework the format.
Not worth it.

I would think that you have to store the transformed field into Graylog as the stream-alert would fire much later.

Yep, unless the transformation is done in the plugin, but that complicates the plugin.

Does this only affect your version? Perhaps give me some examples of what is logged in Graylog and what is in JIRA - possibly we could code some generic transformation - i.e. if you supply some "transformation regex":

Here's an example:

Graylog

application: API
app_version: 5.8.0

Jira

Affects Version/s: Public API 5.8

Thanks for looking into this.

[magicdude4eva]

I see what you mean. It is probably better than to use an extractor or something similar to transform your version into the proper format.

How do you log into Graylog? If you use Logstash it is quite simple to just write a small mutate via Grok to add a field with the right content.

Another option (with Graylog 2) is to write a Message Processor plugin which could then on the fly inject those transformed fields. An example is the GeoIP processor plugin in Graylog 2 which scans fields for IP addresses and then injects GeoIP info: https://github.com/Graylog2/graylog-plugin-map-widget/blob/42268f6e9baad978fe064158a05f934d4bd6f3a9/src/main/java/org/graylog/plugins/map/geoip/processor/GeoIpProcessor.java / https://github.com/Graylog2/graylog-plugin-map-widget/blob/42268f6e9baad978fe064158a05f934d4bd6f3a9/src/main/java/org/graylog/plugins/map/geoip/GeoIpResolverEngine.java

[123dev]

Hi Gerd,
Thanks for getting back.

We log to Graylog in various forms, but in this particular use case it's an application that is logging using GELF libraries.

We can update the application to output another static field having the right values.
For this particular case we are still on stable line 1.3.x of Graylog as this is production instance of Graylog, hence no pipelines.
We are also using Graylog 2 internally for development.

I think I'll write a drool rule and get around it that way.

Thanks again.

[enz1ey]

Any updates on this? I'm trying to simply extract the field (message.fields.winlogbeat_event_data_TargetUserName) from Graylog and enter that in Jira as a request participant, but I'm not sure of the syntax for the field mapping. I tried message.fields.winlogbeat_event_data_TargetUserName=requestParticipants and it didn't work.