Native Grok Reader Implementation

[bangtim]

Description

Native reader implementation for Grok format.

This PR is implementing a GrokDeserializer as well as porting over the entire Grok library (Athena depends on release 0.1.4 with some minor bug fixes and changes to support date data type).

The Java Grok library can be found here: https://github.com/thekrakken/java-grok/tree/grok-0.1.4

• The library includes an api that allows us to parse logs as well as some basic unit tests

Questions/concerns:

• One thing to pay attention to is the LICENSE
• The header is different, thus the build fails (with the same header as other files, the build succeeds locally) - How should we make sure the header is properly citing the authors/contributors of the open source grok library? cc: @martint
• What should the getHiveSerDeClassNames value be?

The implementation(everything aside from java grok library) for the reader was done in the following files:

• trino-hive-formats module:
  • GrokDeserializer + GrokDeserializerFactory --> our implementation of the Deserializer
    • Very similar to regex
  • TestGrokFormat --> some additional unit tests + tests against examples found in athena docs (reading line, following format of other native reader tests)
  • pom.xml
• trino-hive module:
  • HiveModule
  • HiveClassNames
  • HiveMetadata
  • HiveStorageFormat
  • HiveTableProperties
  • GrokFileWriterFactory
  • GrokPageSourceFactory
  • BaseHiveConnectorTest
  • HiveTestUtils
  • TestGrokTable
  • TestHivePageSink
  • pom.xml

Additional context and related issues

Athena supports the GrokSerde and this is a bug-for-bug implementation for what Athena currently has.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Section
* Add native Grok file format reader. ({issue}`25205 `)

[martint]

We need to preserve the copyright notice in those files, but it doesn't need to be laid out verbatim. See how we do it in other places, such as:

trino/core/trino-spi/src/main/java/io/trino/spi/predicate/Primitives.java

Line 22 in daa0a44

// Copyright (C) 2007 The Guava Authors

[pettyjamesm]

Why is this block commented out?

[bangtim]

in the portion above(getRegexPattern), there's a check done to make sure that the regex pattern passed is a valid pattern regex.

At first, I wanted to follow a similar approach and make sure the grok input format passed in is valid regex, but remembered that Pattern.compile() doesn't support when the named capture groups have an underscore

I wasn't sure how to approach this and wanted to bring this to reviewer's attention (I probably should have included that in the PR details) but I think there's two ways to approach:

1. Remove this all check all together
2. Add this check back in (should probably also be checking the grok custom pattern too) but implement something similar to what we did in Grok.java where we remove the underscores from the named capture groups before passing it in to Pattern.compile()

[zhaner08]

Apart from the other comments, would suggest to go through the copied code and remove any code that is not actually being used

[findinpath]

What about the other types out there?

Use RegexDeserializer as inspiration.

Add coverage in TestGrokTable for all the types in this method.

[bangtim]

Initially had all the types that regex covered, but reduced it down to cover all the ones that we currently support in Athena

cc: @zhaner08 @pettyjamesm should we expand to match the types we have for the RegexDeserializer?

[bangtim]

will go ahead and tests for other data types in TestGrokTable

[dain]

some comments... I'm not done yet

[dain]

I don't understand what is going on here. It appears that grok is building a map of name/value pairs. Then the code assumes the values of the map are in specific order that happens to match the order of values in the reader. Then it converts the value to string, so it can be parsed again as the final value type.

This seems like a lot of work when instead we could just read the values into the final expected type directly.

[bangtim]

From my understanding there are two types of casting occurring.

The first happens when we specify the data type in the input format. Since we're reading the log lines as strings, the default data type is string. However, we can specify the data type in the input format.

For example, let's say we have a log:
1
with an 'input.format' = "%{NUMBER:num:double}"

We are specifying that we want whatever NUMBER captures to be represented as a type double, so 1 becomes 1.0

The second type casting occurs with the column type. With the example above, let's say the column that will hold the value captured by NUMBER has a type of String. That means the value shown in that column will be "1.0" as a string representation.

The conversion of value to string does seem a bit unnecessary but I wanted to be able to utilize the primitive type wrapper classes like we do in regex deserializer. For instance, if value was "123" and the column type was BIGINT then it would beneficial to make use of the Long.parseLong(value)

Would you instead suggest having value be a type of Object and then casting value to be whatever we specify the column type to be? I believe some additional casting would have to take place to handle the case mentioned above

[github-actions]

This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack.