Corrupted Database

Open gustavohellwig opened this issue 1 year ago • 0 comments

Hi, I'm trying to update the mmdb database, as there is missing data in some ips, as Graylog needs those information to treat the logs. However, after I update the mmdb, the graylog lookup table doesn't work. The result from the test is this:

{
  "single_value": null,
  "multi_value": null,
  "string_list_value": null,
  "has_error": false,
  "ttl": 9223372036854776000
}

normally, it would be this:

{
  "single_value": "TZ",
  "multi_value": {
    "continent": {
      "code": "AF",
      "geoname_id": 6255146,
      "names": {
        "de": "Afrika",
        "ru": "Африка",
        "pt-BR": "África",
        "ja": "アフリカ",
        "en": "Africa",
        "fr": "Afrique",
        "zh-CN": "非洲",
        "es": "África"
      }
    },
    "country": {
      "confidence": null,
      "geoname_id": 149590,
      "is_in_european_union": false,
      "iso_code": "TZ",
      "names": {
        "de": "Tansania",
        "ru": "Танзания",
        "pt-BR": "Tanzânia",
        "ja": "タンザニア連合共和国",
        "en": "Tanzania",
        "fr": "Tanzanie",
        "zh-CN": "坦桑尼亚",
        "es": "Tanzania"
      }
    },
    "traits": {
      "autonomous_system_number": null,
      "autonomous_system_organization": null,
      "connection_type": null,
      "domain": null,
      "ip_address": "41.59.208.37",
      "is_anonymous": false,
      "is_anonymous_proxy": false,
      "is_anonymous_vpn": false,
      "is_anycast": false,
      "is_hosting_provider": false,
      "is_legitimate_proxy": false,
      "is_public_proxy": false,
      "is_residential_proxy": false,
      "is_satellite_provider": false,
      "is_tor_exit_node": false,
      "isp": null,
      "mobile_country_code": null,
      "mobile_network_code": null,
      "network": "41.59.0.0/16",
      "organization": null,
      "user_type": null,
      "user_count": null,
      "static_ip_score": null
    },
    "represented_country": {
      "confidence": null,
      "geoname_id": null,
      "is_in_european_union": false,
      "iso_code": null,
      "names": {},
      "type": null
    },
    "registered_country": {
      "confidence": null,
      "geoname_id": 149590,
      "is_in_european_union": false,
      "iso_code": "TZ",
      "names": {
        "de": "Tansania",
        "ru": "Танзания",
        "pt-BR": "Tanzânia",
        "ja": "タンザニア連合共和国",
        "en": "Tanzania",
        "fr": "Tanzanie",
        "zh-CN": "坦桑尼亚",
        "es": "Tanzania"
      }
    }
  },
  "string_list_value": null,
  "has_error": false,
  "ttl": 9223372036854776000
}

what I have done:

installed the app:

wget "https://github.com/ipinfo/mmdbctl/releases/download/mmdbctl-1.4.4/mmdbctl_1.4.4_linux_amd64.tar.gz" && tar xvf mmdbctl_1.4.4_linux_amd64.tar.gz && rm mmdbctl_1.4.4_linux_amd64.tar.gz && mv mmdbctl_1.4.4_linux_amd64 /usr/local/bin/mmdbctl
sudo apt-get update && sudo apt-get install jq

Exported: mmdbctl export --in ${importfrom} --out ${exportto}
Changed the content using python:

import json

input_file = '/opt/scripts/tmp/GeoLite2-City.json'
output_file = '/opt/scripts/tmp/GeoLite2-City-Changes.json'

def add_subdivisions_to_json_line(json_line):
    data = json.loads(json_line)
    original_data = data.copy()  # Make a copy of the original data for comparison

    if 'subdivisions' not in data:
        country_info = data.get('country', {})
        if country_info:
            subdivisions = [{
                "geoname_id": country_info.get("geoname_id"),
                "iso_code": country_info.get("iso_code"),
                "names": {"en": country_info["names"]["en"]}
            }]
            data['subdivisions'] = subdivisions

    return data, data != original_data  # Return both the data and a flag indicating if a change was made

with open(input_file, 'r') as infile, open(output_file, 'w') as outfile:
    for line in infile:
        updated_data, is_changed = add_subdivisions_to_json_line(line)
        if is_changed:
            updated_line = json.dumps(updated_data)
            outfile.write(updated_line + '\n')

print(f"Processed file saved as {output_file}")

Imported the changes to the mmdb: mmdbctl import --disable-metadata-pointers false --in ${updatedjson} --out ${mmdbbkp}
Tests mmdbctl metadata GeoLite2-City-updated.mmdb

Binary Format 2.0
Database Type ipinfo GeoLite2-City-updated.mmdb
IP Version 6
Record Size 32
Node Count 3971781
Description en ipinfo GeoLite2-City-updated.mmdb
Languages en
Build Epoch 1721934123

mmdbctl metadata GeoLite2-City.mmdb

Binary Format 2.0
Database Type GeoLite2-City
IP Version 6
Record Size 28
Node Count 3971880
Description en GeoLite2City database
Languages de, en, es, fr, ja, pt-BR, ru, zh-CN
Build Epoch 1721747951

mmdbctl read -f json-pretty 41.59.208.37 GeoLite2-City.mmdb # Original File { "continent": { "code": "AF", "geoname_id": 6255146, "names": { "de": "Afrika", "en": "Africa", "es": "África", "fr": "Afrique", "ja": "アフリカ", "pt-BR": "África", "ru": "Африка", "zh-CN": "非洲" } }, "country": { "geoname_id": 149590, "iso_code": "TZ", "names": { "de": "Tansania", "en": "Tanzania", "es": "Tanzania", "fr": "Tanzanie", "ja": "タンザニア連合共和国", "pt-BR": "Tanzânia", "ru": "Танзания", "zh-CN": "坦桑尼亚" } }, "location": { "accuracy_radius": 1000, "latitude": -6.8227, "longitude": 39.2936, "time_zone": "Africa/Dar_es_Salaam" }, "registered_country": { "geoname_id": 149590, "iso_code": "TZ", "names": { "de": "Tansania", "en": "Tanzania", "es": "Tanzania", "fr": "Tanzanie", "ja": "タンザニア連合共和国", "pt-BR": "Tanzânia", "ru": "Танзания", "zh-CN": "坦桑尼亚" } } }

mmdbctl read -f json-pretty 41.59.208.37 GeoLite2-City-updated.mmdb # New File { "continent": { "code": "AF", "geoname_id": 6255146, "names": { "de": "Afrika", "en": "Africa", "es": "África", "fr": "Afrique", "ja": "アフリカ", "pt-BR": "África", "ru": "Африка", "zh-CN": "非洲" } }, "country": { "geoname_id": 149590, "iso_code": "TZ", "names": { "de": "Tansania", "en": "Tanzania", "es": "Tanzania", "fr": "Tanzanie", "ja": "タンザニア連合共和国", "pt-BR": "Tanzânia", "ru": "Танзания", "zh-CN": "坦桑尼亚" } }, "location": { "accuracy_radius": 1000, "latitude": -6.8227, "longitude": 39.2936, "time_zone": "Africa/Dar_es_Salaam" }, "network": "41.59.208.0/20", "registered_country": { "geoname_id": 149590, "iso_code": "TZ", "names": { "de": "Tansania", "en": "Tanzania", "es": "Tanzania", "fr": "Tanzanie", "ja": "タンザニア連合共和国", "pt-BR": "Tanzânia", "ru": "Танзания", "zh-CN": "坦桑尼亚" } }, "subdivisions": [ { "geoname_id": 149590, "iso_code": "TZ", "names": { "en": "Tanzania" } } ] }

Graylog log:

WARN [MaxmindDataAdapter] Unable to look up city data for IP address /123.185.49.44, returning empty result. java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "parameters[index]" is null at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:450) ~[graylog.jar:?] at com.maxmind.db.Decoder.decodeMap(Decoder.java:341) ~[graylog.jar:?] at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[graylog.jar:?] at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[graylog.jar:?] at com.maxmind.db.Decoder.decode(Decoder.java:89) ~[graylog.jar:?] at com.maxmind.db.NoCache.get(NoCache.java:17) ~[graylog.jar:?] at com.maxmind.db.Decoder.decode(Decoder.java:116) ~[graylog.jar:?] at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:434) ~[graylog.jar:?] at com.maxmind.db.Decoder.decodeMap(Decoder.java:341) ~[graylog.jar:?] at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[graylog.jar:?] at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[graylog.jar:?] at com.maxmind.db.Decoder.decode(Decoder.java:76) ~[graylog.jar:?] at com.maxmind.db.Reader.resolveDataPointer(Reader.java:411) ~[graylog.jar:?] at com.maxmind.db.Reader.getRecord(Reader.java:185) ~[graylog.jar:?] at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:280) ~[graylog.jar:?] at com.maxmind.geoip2.DatabaseReader.getCity(DatabaseReader.java:365) ~[graylog.jar:?] at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:348) ~[graylog.jar:?] at org.graylog.plugins.map.geoip.MaxMindIPLocationDatabaseAdapter.maxMindCity(MaxMindIPLocationDatabaseAdapter.java:39) ~[graylog.jar:?] at org.graylog.plugins.map.geoip.MaxmindDataAdapter.doGet(MaxmindDataAdapter.java:186) ~[graylog.jar:?] at org.graylog2.plugin.lookup.LookupDataAdapter.get(LookupDataAdapter.java:143) ~[graylog.jar:?] at org.graylog2.lookup.LookupTable.lambda$lookup$0(LookupTable.java:73) ~[graylog.jar:?] at org.graylog2.lookup.caches.CaffeineLookupCache.lambda$get$2(CaffeineLookupCache.java:161) ~[graylog.jar:?] at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167) ~[graylog.jar:?] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688) ~[graylog.jar:?] at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) [?:?] at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686) [graylog.jar:?] at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669) [graylog.jar:?] at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) [graylog.jar:?] at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) [graylog.jar:?] at org.graylog2.lookup.caches.CaffeineLookupCache.get(CaffeineLookupCache.java:182) [graylog.jar:?] at org.graylog2.lookup.LookupTable.lookup(LookupTable.java:73) [graylog.jar:?] at org.graylog2.lookup.LookupTableService$Function.lookup(LookupTableService.java:635) [graylog.jar:?] at org.graylog2.inputs.extractors.LookupTableExtractor.run(LookupTableExtractor.java:66) [graylog.jar:?] at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:226) [graylog.jar:?] at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:79) [graylog.jar:?] at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:105) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:167) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:137) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:107) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:52) [graylog.jar:?] at org.graylog2.shared.buffers.PartitioningWorkHandler.onEvent(PartitioningWorkHandler.java:52) [graylog.jar:?] at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) [graylog.jar:?] at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.base/java.lang.Thread.run(Unknown Source) [?:?]

mmdb Sizes: -rw-r--r-- 1 root root 237M Jul 26 00:38 GeoLite2-City-updated.mmdb -rw-r--r-- 1 root root 50M Jul 25 07:28 GeoLite2-City.mmdb

json sizes: -rw-r--r-- 1 root root 963M Jul 25 14:35 GeoLite2-City-Changes.json -rw-r--r-- 1 root root 3.8G Jul 25 08:36 GeoLite2-City-Dump.json

Why is the database in some way corrupted that Graylog can't read but the command line can??

Jul 26 '24 21:07 gustavohellwig