getGeoDataSync so slow with 3k ip list

[nvcken]

I try getGeoDataSync vs geoip.lookup of geoip-lite module with the same 3k IP list getGeoDataSync run so slow what wrong?

[the-eater]

I think this is because of the way the MaxMind provided his data, while geoip-lite uses the legacy database, we use the new MaxMind DB (mmdb). the legacy database allows to predict the position where the record may be, while node-maxmind-db needs to read the whole tree till found.

am I correct @oschwald / @paddek?

[oschwald]

With the new format, cache misses are more likely due to how the data is stored, but I think the primary issue is that the new format provides significantly more data. One possible way to alleviate that is to allow looking up a subset of the data. For instance the C library allows looking up data by a path and the Go reader will only look up data specified in the struct passed in.

That said, I think the easiest speed improvement that I am not seeing in the code currently is to cache the IPv4 start node. This will save you 96 node reads when doing a lookup and is simple to implement. There may also be other simple improvements that profiling would detect.

[mahnunchik]

+1

[vbauer]

:+1:

[vivekkrbajpai]

So is there any option or config to get better throughput form mmdb format ?

[the-eater]

Sorry for reacting late :( Im currently working on a version that does ipv4 caching and allows a path like suggested by @oschwald, you can see the progress on that here: https://github.com/EaterOfCode/node-maxmind-db/tree/moreSpeed

You can use paths currently by giving the path argument with a path to a value you want to know, f.x. if you only want to have the us name of a ip you can retrieve that by

mmdbreader.getGeoDataSync('8.8.8.8', ['country', 'names', 'en']);

Any feedback would be more then welcome!

[knoxcard]

https://github.com/runk/node-maxmind