DeepMergeWith inserter function causes high memory usage

[mandar-01]

Hi,

I have been using the mmdbwriter package in golang to insert some records into an MMDB file. I observed that the script to insert records into MMDB consumes a lot of memory. I did some memory profiling using pprof and I found that a couple of MMDB writer functions were consuming most of the memory. I have attached a screenshot below for reference.

These functions consume around 600MB each whereas the size of the final MMDB file was 146MB. Overall, the program consumed around 3.8GB for producing a file of 146MB. I think these functions, specially the Map.Copy(), stores the records in-memory and is not garbage collected since their references are still in use. This profiling is done just before the writer writes the MMDB file on disk.

Here's how I have defined the MMDB writer:

writer, err := mmdbwriter.New(
	mmdbwriter.Options{
		DatabaseType:            "V1",
		IncludeReservedNetworks: true,
		RecordSize:              32,
	},
)

I am using the DeepMergeWith inserter to insert the MMDB records