[doc] creating large SharedMemory crashes on MacOs

[bf8b85a2-9df0-412b-b6c3-5481df7521c8]

BPO	39584
Nosy	@ronaldoussoren, @pitrou, @tiran, @ned-deily, @applio, @vinay0410
PRs	python/cpython#21877

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-02-08.14:18:24.896>
labels = ['OS-mac', '3.8', '3.9', 'expert-C-API', 'type-crash', 'docs']
title = 'multiprocessing.shared_memory: MacOS crashes by running attached Python code'
updated_at = <Date 2020-11-17.10:55:29.496>
user = 'https://github.com/vinay0410'

bugs.python.org fields:

activity = <Date 2020-11-17.10:55:29.496>
actor = 'ronaldoussoren'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'macOS', 'C API']
creation = <Date 2020-02-08.14:18:24.896>
creator = 'vinay0410'
dependencies = []
files = []
hgrepos = []
issue_num = 39584
keywords = ['patch']
message_count = 9.0
messages = ['361629', '361924', '373806', '375229', '375245', '375249', '381221', '381222', '381223']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'pitrou', 'christian.heimes', 'ned.deily', 'docs@python', 'davin', 'vinay0410']
pr_nums = ['21877']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39584'
versions = ['Python 3.8', 'Python 3.9']

[bf8b85a2-9df0-412b-b6c3-5481df7521c8]

Consider the following python Code.

from multiprocessing.shared_memory import SharedMemory
shm = SharedMemory(name='test-crash', create=True, size=1000000000000000000)

This causes macOS Catalina, Mojave to freeze and then crash. Although, this works fine on ubuntu.

After, debugging I realised that this is due to the ftruncate call. I could replicate the same by calling os.ftruncate and also using ftruncate in C code.

Following C++ code also crashes, which confirms that ftruncate in macOS is broken:

#include <fcntl.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <iostream>
#include <sys/mman.h>
#include <unistd.h>
#include <sys/types.h>
#include <errno.h>
#include <time.h>

int main() {

    int shm_fd = shm_open("/test-shm2", O_CREAT | O_RDWR, 0666);

    if (shm_fd == -1) {
        throw "Shared Memory Object couldn't be created or opened";
    }

    int rv = ftruncate(shm_fd, (long long)1000000000000000000);

}

Should python, in any way handle this, so as to prevent any crashes using python code.

[applio]

My sense is that it would be nice if we can catch this before ftruncate does something nasty.

Where else is ftruncate used in CPython that this could similarly trigger a problem? How is it handled there (or not)?

[bf8b85a2-9df0-412b-b6c3-5481df7521c8]

Hi, I tried replicating this by truncating normal files but that doesn't crash. The above mentioned call of ftruncate only crashes for when the file descriptor passed points to a shared memory segment. And only, multiprocessing.shared_memory is currently creating shared_memory using _posixshmem.shm_open.

So, it can be fixed in ftruncate implementation or this can also be handled by multiprocessing.shared_memory.

Please let me know your thoughts about the same.

[ronaldoussoren]

I expect that this is more a problem with how memory management works on macOS, the freeze and crash is likely an indication of eager allocation of memory by the system.

It is far from sure if implementing a workaround is feasible, the upper limit for safe calls to ftruncate likely depends on the amount of RAM in the system.

BTW. The sample code tries to allocate well over a petabyte of memory, how realistic is that code? It might be good enough to document that using very large amounts of memory with multiprocessing.shared_memory causes problems on macOS.

[bf8b85a2-9df0-412b-b6c3-5481df7521c8]

I have 8GB of ram and 128 GB of hard disk.

Now, creating a shared memory segment of size 10^12 (1 terabyte) somehow succeeds.

Creating a shared memory segment of 10^15 (1 petabyte), mmap (not ftruncate) throws an error stating cannot allocate memory.

Creating a shared memory segment of 10^18 (1 exabyte), causes the system to crash.

Now, I understand that this should be documented for a genuine user.

But, if documented this can be used by malicious softwares using python to crash systems abruptly.

Also, I understand that this is an issue with macos, but shouldn't python handle this so that atleast python's APIs are safe.

Creating shared memory segments of size 1 exabyte are not reasonable, but if some makes a mistake then, we must throw an error instead of a crash.

Also, can we set a max limit on creating shared memory segments to 1TB ? because no one would genuinily need to create a segment of that size on a single machine.

[ronaldoussoren]

A workaround in the implementation of multiprocessing.SharedMemory is IMHO acceptable, tweaking os.ftruncate less so.

Note that a Python script can already cause problems on systems by using APIs as intended (such as using shutil.rmtree on the user's home directory).

There's balance between compensating for platform deviancies and having clean and performant implementation.

BTW. We should file an issue with Apple about this. I'll do so when I've had time to crash a test VM using the C code you provided.

[tiran]

I'm strong -1 on any kind of arbitrary memory limit. Python is used on all sorts of hardware from RPi to super computers. 1 TB sounds like a lot, but it really is not. These days you can buy workstation computers with more than 1 TB RAM.

I would be ok with a limit of 2**48 on X86_64 platforms. Current X86_64 CPUs have a virtual address size limit of 48 bits. Other hardware platforms may have a similar limit. I don't have access to AArch64 or PPC64 to verify the address size.

[ronaldoussoren]

Having thought about this a little more: I agree, we shouldn’t hard code a limit. I’m not against working around misfeatures, but in this case that’s hard to do: Even if we’d limit the size of a single shared memory segment the user can create a number of them to once again use enough memory to crash the machine.

IMHO We should document that this can happen on macOS (preferable with some indication of how much shared memory can safely be used). I’ve therefore added the “Documentation” component.

[ronaldoussoren]

I’ve filed an issue with Apple about this: FB8903019

[dversoza]

@ronaldoussoren, do we have any updates with Apple on the filled issue FB8903019?

@tiran, I agree with you on not hard-codding any arbitrary memory limit, but considering that creating large SharedMemory will continue to crash on MacOS, I'd like to hear from you if documenting about the MacOS multiprocessing limit be enough to handle this issue?

[ronaldoussoren]

@ronaldoussoren, do we have any updates with Apple on the filled issue FB8903019?

There are no updates on the issue with Apple.

[esatormi]

Couldn't reproduce the crash with Python3.12 and MacOS Sonoma (14.5).

Running the original code:

>>> from multiprocessing.shared_memory import SharedMemory
>>> shm = SharedMemory(name='test-crash', create=True, size=1000000000000000000)
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 239, in main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/shared_memory.py", line 115, in __init__
    self._mmap = mmap.mmap(self._fd, size)
    cache[rtype].remove(name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/test-crash'
OSError: [Errno 12] Cannot allocate memory

Smaller amount works:

shm = SharedMemory(name='test-crash', create=True, size=100000000000000)

Given C++ code works too.

@ronaldoussoren is there any updates about the issue?

[picnixz]

Friendly ping @ronaldoussoren (you can also close the issue if you think it's not worth the additional doc effort).

[gpshead]

I do not think this is worth documenting in any specific platform sense. All hardware platforms have address space size limits, and the underlying OSes tend to further restrict the amount of that possible to claim as shared memory. Often configurable to a much smaller limit on a system, process, or container basis.

This is just something people willing to delve into using shared memory need to know. But the specifics aren't something Python docs are appropriate to contain, at most we could just mention "most platforms have specific sometimes configurable limits in the amount of shared memory address possible to allocate".