OOM issue for loading kb.sling

[luke217]

Hi Mr. Ringgaard!

It's great to see your wonderful work! Recently, I've been trying to run templama scripts with sling installed as instructed but later on find my laptop unable to handle the huge memory request of the process. To summarize the logic that I've run, there are basically two parts involved:

1. bash script

pip install https://ringgaard.com/data/dist/sling-3.0.0-py3-none-linux_x86_64.whl

# Download SLING KB and en wikipedia mapping.
sling fetch --dataset kb,mapping

and from this I got a kb.sling file with 16GB big.

2. python script

commons = sling.Store()
commons.load(sling_kb_file)

and this gives me an oom error when loading that 16GB kb.sling

Here's the log for the oom error:

[Tue Oct 22 01:29:34 2024] Out of memory: Killed process 1462 (sling2facts.py) total-vm:22444652kB, anon-rss:15766396kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:38828kB oom_score_adj:0

My laptop only has 32GB memory in total. Is there a possibility that I can load the kb.sling partially? Thanks!

PS: To run the second script templama.py in templama, I would need more resources since the following errors are found:

I1022 01:31:30.817656 140702686967616 templama.py:154] Extracting entity names and num-facts from SLING KB.
[Tue Oct 22 01:38:03 2024] Out of memory: Killed process 2766 (templama.py) total-vm:59056860kB, anon-rss:15760604kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:40012kB oom_score_adj:0

[ringgaard]

The knowledge base has grown a bit over time, so 32 GB RAM is a bit on the low side these days.

You could try to download a KB snapshot from here: kb.sling.snap

Make sure that kb.sling.snap is in the same directory as kb.sling and that it has a newer modification data. Then the SLING loader should detect this and use it for loading the KB. Using a snapshot uses less memory because you don't have to build intermediate structures for loading the KB.