numaflow-python icon indicating copy to clipboard operation
numaflow-python copied to clipboard

[WIP]feat: add multiproc async

Open kohlisid opened this issue 9 months ago • 1 comments

UDS Mode

(venv11) skohli@macos-JQWR9T560R async_multiproc_map % kubectl logs simple-pipeline-mult-0-ssrln -f | head -n 10 
+ python example.py
2025-05-21 23:03:15 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/multiproc2.sock with max threads: 4
2025-05-21 23:03:15 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/multiproc0.sock with max threads: 4
2025-05-21 23:03:15 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/multiproc1.sock with max threads: 4

TCP Mode

(venv11) skohli@macos-JQWR9T560R async_multiproc_map % kubectl logs simple-pipeline-mult-0-wi104 -f | head -n 10 
+ python example.py
2025-05-21 23:04:44 INFO     Async GRPC Server listening on: 0.0.0.0:42285 with max threads: 4
2025-05-21 23:04:44 INFO     Async GRPC Server listening on: 0.0.0.0:35685 with max threads: 4
2025-05-21 23:04:44 INFO     Async GRPC Server listening on: 0.0.0.0:33085 with max threads: 4

kohlisid avatar May 20 '25 03:05 kohlisid

Codecov Report

:x: Patch coverage is 52.94118% with 40 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 92.80%. Comparing base (6ccd49e) to head (62b7a85). :warning: Report is 28 commits behind head on main.

Files with missing lines Patch % Lines
pynumaflow/mapper/async_multiproc_server.py 34.69% 32 Missing :warning:
pynumaflow/mapper/_servicer/_async_servicer.py 70.00% 5 Missing and 1 partial :warning:
...numaflow/reducestreamer/servicer/async_servicer.py 33.33% 2 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #224      +/-   ##
==========================================
- Coverage   94.19%   92.80%   -1.39%     
==========================================
  Files          58       59       +1     
  Lines        2359     2419      +60     
  Branches      119      124       +5     
==========================================
+ Hits         2222     2245      +23     
- Misses        100      136      +36     
- Partials       37       38       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar May 20 '25 03:05 codecov[bot]