filter.py - trim does not cut as expected

[johnhtodd]

I'd like to use standard JSON formatting to specify what fields or arrays should be trimmed.

I had an arguments file that looks like this:

(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018# cat arguments
continent
country.confidence
country.geoname_id
location.time_zone
postal
registered_country.geoname_id
traits.autonomous_system_number
traits.autonomous_system_organization
traits.isp
traits.organization
traits.domain
subdivisions.geoname_id

and applied it to an MMDB file that had objects in it that should have matched those namespaces.

../mrt2mmdb/mrt2mmdb/filter.py --trim `cat arguments` --mmdb GeoIP2-Enterprise.mmdb.trim1

This leads to objects that still contain fields I expected to be trimmed:

{
  "city": {
    "confidence": 20,
    "geoname_id": 6698658,
    "names": {
      "en": "Ko Pha Ngan"
    }
  },
  "country": {
    "confidence": 59,
    "geoname_id": 1635651,
    "iso_code": "TH",
    "names": {
      "en": "Thailand"
    }
  },
  "location": {
    "accuracy_radius": 100,
    "latitude": 9.4167,
    "longitude": 101,
    "time_zone": "Asia/Bangkok"
  },
  "range": "1.0.163.56/29",
  "registered_country": {
    "geoname_id": 1675651,
    "iso_code": "TH",
    "names": {
      "en": "Thailand"
    }
  },
  "subdivisions": [
    {
      "confidence":90,
      "geoname_id": 1130524,
      "iso_code": "84",
      "names": {
        "en": "Surat Thani"
      }
    }
  ],
  "traits": {
    "autonomous_system_number": 22969,
    "autonomous_system_organization": "TOT Public Company Limited",
    "connection_type": "Cable/DSL",
    "domain": "totinternet.net",
    "isp": "TOT",
    "organization": "TOT",
    "user_type": "residential"
  }
}

The "continent" and "postal" and other top-level objects were deleted, but the second-level items were not.

If an array or object matches the name given, then it should be deleted.

[johnhtodd]

Here's another full example where you can see the fields not being deleted that I've specified in the "arguments" file. Some are correctly deleted like "registered_country" but apparently there is no way to delete sub-objects or sub-arrays like "country.confidence"

(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018# dir
total 369948
     4 drwxrwxr-x 2 root  root       4096 Nov 19 22:22 .
     4 drwxr-x--- 9 jtodd jtodd      4096 Nov 19 02:18 ..
     4 -rw-r--r-- 1 root  root        294 Nov 19 22:21 arguments
369936 -rw-r--r-- 1 root  root  378810717 Oct 18 16:29 GeoIP2-Enterprise.mmdb
(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018# cat arguments
continent
country.confidence
country.geoname_id
country.confidence
location.time_zone
postal
traits.autonomous_system_number
traits.autonomous_system_organization
traits.isp
traits.organization
traits.domain
traits.is_anycast
subdivisions.geoname_id
location.accuracy_radius
registered_country
(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018# ../mrt2mmdb/mrt2mmdb/filter.py --trim `cat arguments` --mmdb GeoIP2-Enterprise.mmdb
 Apply filter to trim mmdb file                                                    : 1399 prefixes [00:01, 791.56 prefixes/s] ^C
Traceback (most recent call last):
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 169, in <module>
    rewrite(fname, db, pb)
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 105, in rewrite
    for prefix, record in dic_data:
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 86, in <genexpr>
    res = (((prefix.compressed), filter_dict(data)) for prefix, data in mreader)
                                 ^^^^^^^^^^^^^^^^^
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 75, in filter_dict
    return fnc(raw)
           ^^^^^^^^
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 70, in <lambda>
    key1: fnc(val1) if isinstance(val1, (dict, list)) else val1
          ^^^^^^^^^
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 64, in <lambda>
    fnc(item) if isinstance(item, dict) else item
    ^^^^^^^^^
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 70, in <lambda>
    key1: fnc(val1) if isinstance(val1, (dict, list)) else val1
          ^^^^^^^^^
  File "/home/jtodd/GeoIP2-Enterprise_20241018/../mrt2mmdb/mrt2mmdb/filter.py", line 70, in <lambda>
    key1: fnc(val1) if isinstance(val1, (dict, list)) else val1
KeyboardInterrupt

(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018# cat booz|jq -r|more^C
(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018#  mmdbctl export --format json GeoIP2-Enterprise.mmdb.trim |jq -r|more
{
  "range": "1.0.0.0/24",
  "traits": {
    "autonomous_system_number": 13335,
    "autonomous_system_organization": "CLOUDFLARENET",
    "connection_type": "Corporate",
    "is_anycast": true,
    "isp": "Cloudflare",
    "organization": "Cloudflare",
    "user_type": "content_delivery_network"
  }
}
{
  "country": {
    "confidence": 99,
    "geoname_id": 1814991,
    "iso_code": "CN",
    "names": {
      "en": "China"
    }
  },
  "location": {
    "accuracy_radius": 1000,
    "latitude": 34.7732,
    "longitude": 113.722,
    "time_zone": "Asia/Shanghai"
  },
  "range": "1.0.1.0/24"
}
{
  "country": {
    "confidence": 99,
    "geoname_id": 1814991,
    "iso_code": "CN",
    "names": {
      "en": "China"
    }
  },
  "location": {
    "accuracy_radius": 1000,
    "latitude": 34.7732,
    "longitude": 113.722,
    "time_zone": "Asia/Shanghai"
  },
  "range": "1.0.2.0/23"
}
{
  "country": {
    "confidence": 99,
    "geoname_id": 2077456,
    "iso_code": "AU",
    "names": {
      "en": "Australia"
    }
  },
  "location": {
    "accuracy_radius": 1000,
    "latitude": -33.494,
    "longitude": 143.2104,
    "time_zone": "Australia/Sydney"
(.venv) root@mmdbpython:/home/jtodd/GeoIP2-Enterprise_20241018#

[sbng]

current filter implementation is base on common key. Hence, if you like to filter country.geoname_id, you can use
geoname_id as a filter term. However, this would filter bother country.geoname_id and subdivisions.geoname_id. Is this acceptable?

[johnhtodd]

I guess this works for now, but it seems like this will result in future unexpected behaviors.

[sbng]

Ok. Let me add another filter to filter the specific key as well.

[johnhtodd]

Can you please make it a bit more flexible:

.foo # deletes any object or array ending in ".foo" such as blah.bar.foo
foo # deletes any object or array matching "foo" such as foo.bar or foo.baz
.foo. # deletes any object or array containing foo such as blah.foo.baz

Also: please remove the "default" deletions, such as languages. With this new syntax, it should be easy enough to specify all the things for deletion, and having default deletions seems inflexible.

[sbng]

Please test commit 37be655 under filter_issue branch. This fix allows both key word filter as well as nested dictionary key filtering (eg. traits.isp or country.geoname_id). Additonally, the default lanaguage been remove from the code. All language filter needs to be part of the --trim list. Once it's verified correct. I will merge this to the main branch.