Billiard is leaving pipes open

[luederm]

I am using Celery to initiate long running pipelines and I override multiprocessing imports with Biliard. I notice as a pipeline runs, the number of open files (pipes) steadly increases and never decreases. This occurs until tens of thousands of pipes are opened and I get the error OSError: [Errno 24] Too many open files. I ran this pipeline outside of Celery without overriding multiprocessing imports and this issue does not occur. Processes are spawned by Luigi workers and I am using the latest version of Celery.

[auvipy]

were you able to figure out the root of this in billiard?

[luederm]

I use the following code to override multiprocessing imports in a 3rd party package (luigi) with biliard:

def _import(name, *args, **kwargs):
    if name == 'multiprocessing':
        name = 'billiard'
    return original_import(name, *args, **kwargs)

original_import = builtins.__import__
builtins.__import__ = _import

I performed 2 tests by running my luigi pipeline manually, outside of Celery. In test A, I ran my pipeline without the above code (so luigi uses multiprocessing). In this test the number of open pipes as described by the lsof commnd remained stable and the pipeline completed successfully. I test B, I ran the pipeline with the above code (so luigi uses billiard). This causes the number of open pipes to continually increase until the program crashes.

All code referencing billiard/multiprocessing is in luigi (version 3.0.2). I did not write multiprocessing code of my own.

Edit: I originally misread you question, but will leave this here incase the information is useful. I did not figure out what code in billiard was causing the issue.

[darkl1ght]

I am using Celery to initiate long running pipelines and I override multiprocessing imports with Biliard. I notice as a pipeline runs, the number of open files (pipes) steadly increases and never decreases. This occurs until tens of thousands of pipes are opened and I get the error OSError: [Errno 24] Too many open files. I ran this pipeline outside of Celery without overriding multiprocessing imports and this issue does not occur. Processes are spawned by Luigi workers and I am using the latest version of Celery.

How do you override multiprocessing imports with Billiard? I am new to billiard and trying to figure out where to begin with. I am also stuck with this question: https://stackoverflow.com/questions/66135819/running-subprocess-popen-in-celery-task-gives-status-code-134sigabrt-error?noredirect=1#comment116933899_66135819

Please see, if you could help me in any way or direction with my question.

[luederm]

@darkl1ght That is what is shown in the code in my comment above. That code needs to run before you import multiprocessing.

WRT you stackoverflow question (I don't have enough rep to comment), I doubt that is a Celery issue. To my knowledge, billiards does not affect subprocess calls. Does the subprocess command fail when not using celery?