Share vertex map with the workers on the same host using shared memory

[wenyuanyu]

Is your feature request related to a problem? Please describe. When multiple worker processes are started on the same host (e.g., for auto workers), global vertex maps of fragments will be created multiple times

Describe the solution you'd like Global vertex maps can be placed in shared memory, and getting share across workers on the same host to save the unnecessary memory overhead

Describe alternatives you've considered To support multiple concurrent workers for each process?

[tomzhang]

maybe we can using concurrent queue, each worker should be binding with its "workingID" and work in the producer-consumer way to separate with each others thread.

[tomzhang]

and this project much more like an "demo" rather than a real project which already working in production environment.

[wenyuanyu]

@tomzhang Thanks very much for the comments!

maybe we can using concurrent queue, each worker should be binding with its "workingID" and work in the producer-consumer way to separate with each others thread.

This issue is specifically about "auto" version of apps, where we hide the multithread programming from programmers. For other types of apps, there will be a ParallelEngine provided for parallelization that could maximize the utilization of local cores, therefore there is no need for starting multiple MPI workers on the same host.

and this project much more like an "demo" rather than a real project which already working in production environment.

Actually, there are many applications in production at Alibaba built upon this library :) You are welcome to try this library yourself. Any feedback or suggestions are GREATLY appreciated!

Thanks!