Support for custom databases

[icb-]

MaxMind publish their database specification and publish a Perl module for writing your own custom databases.

I would like to be able to use the GeoIP plugin to lookup custom data about IP addresses in Logstash, but that's not possible without pretending to be one of the supported database types. The specification states "[n]ames starting with "GeoIP" are reserved for use by MaxMind (and “GeoIP” is a trademark anyway)." so I really shouldn't have to do that. If I want to use my own custom fields, I don't have a lot of options other than sticking a JSON object in one of the fields and the json filter to pull out my custom data.

The database format is self-describing, so you should be able to fetch the data without a pre-defined schema for the custom database.

[kvmuralidhar]

We have a similar requirement and are now storing the data in certain geoip database fields like city_name, country_name & time_zone.

[lisaens]

We are also

[bionicspy]

As are we. Starting with 7.1.3 this is no longer working either....

[ViperGDC]

Is there a way to use custom databases? I used the newer Go module to build a custom mmdb of internal ip's and I was initially getting unsupported database errors until i used City for default database type. But i still cant get logstash to use my custom mmdb to enrich my internal IPs.

[ViperGDC]

It would be really nice if this plugin also supported custom mmdb's. Or if someone could tell us how to get a custom mmdb to work with the filter. i have spent hours trying to get my custom mmdb to follow the tree structure of the GeoLite2-City.mmdb and just using a mutate to rename the fields but still no luck.

[icb-]

Or if someone could tell us how to get a custom mmdb to work with the filter.

1. Make sure your database (not just the file) is named GeoLite2-City. The plugin looks for specific database names to decide what field names are available.
2. Make sure you're setting the latitude and longitude fields, even if you aren't using them and just set them to 0. The plugin checks that those aren't empty when doing a lookup in what it thinks is the City database.

[ViperGDC]

Hey @icb- thanks for the info i do have the DBType set to GeoLite2-City as well as the name of the DB but still having issues. Did you happen to do this in your environment? I used the methods detailed here (https://blog.maxmind.com/2020/09/01/enriching-mmdb-files-with-your-own-data-using-go/) to create an mmdb using GO. I think i have the right tree structure but i am not sure.

here is a screen shot of a snip of my GO code...

[icb-]

I'm creating my own database, rather than augmenting an existing one. This is a stripped down version of what I'm doing.

package main

import (
    "github.com/maxmind/mmdbwriter"
    "github.com/maxmind/mmdbwriter/mmdbtype"
    "net"
    "os"
)

func main() {
    writer, _ := mmdbwriter.New(
        mmdbwriter.Options{
            DatabaseType:            "GeoLite2-City",
            RecordSize:              28,
            IncludeReservedNetworks: true,
        },
    )
    _, network, _ := net.ParseCIDR("8.8.8.0/24")
    record := mmdbtype.Map{
        "city": mmdbtype.Map{
            "names": mmdbtype.Map{
                "en": mmdbtype.String(`{"json":"would go here"}`),
            },
        },
        "location": mmdbtype.Map{
            "latitude":  mmdbtype.Uint32(0),
            "longitude": mmdbtype.Uint32(0),
        },
    }
    writer.Insert(network, record)
    fh, _ := os.Create("test.mmdb")
    defer fh.Close()
    writer.WriteTo(fh)
}

Paired with a Logstash config something like this:

filter {
  # ...
  geoip {
    database => "test.mmdb"
    source => "clientip"
  }
  json {
    source => "[geoip][city_name]"
  }
}

I get events that look something like:

{
  // ...
  "clientip": "8.8.8.8",
  "json": "would go here",
  "geoip": {
    "longitude": 0,
    "latitude": 0,
    "city_name": "{\"json\":\"would go here\"}",
    "location": {
      "lon": 0,
      "lat": 0
    }
  }
}