Yi Cheng

Thanks for fixing this. As synced offline, my only concern is that we make the code_search_path and the new introduced ones doing similar works. Maybe we should try to unify...

Btw, could you also delete import thread which is not useful anymore with this work.

Question: will this slow down the perfs? I think this adds runtime env to all task specs (previously, only one). Do you mind benchmarking the perf regression? Besides this, do...

@scv119 the default microbenchmark doesn't use job config. What concerns me is this: ``` https://sourcegraph.com/github.com/ray-project/ray@master/-/blob/src/ray/protobuf/gcs.proto?L311 ``` Could you add some runtime envs there, for example, a list of OS envs...

``` import ray ray.init() @ray.remote def process(data): return b"\0" * 100_000_000 while True: data = ray.put(b"\0" * 100_000_000) ref = process.remote(data) ray.get(ref) del ref ``` Won't have the leak. Maybe...

``` [2023-01-04 21:17:14,005 D 3290269 3290314] (raylet) store.cc:401: Received create request for object 6d83b3e72b6da8ce4f1fb3b11f2a02d5aeeedd2e6459ec7100000000 assigned request ID 1, 8 bytes [2023-01-04 21:17:14,005 D 3290269 3290314] (raylet) object_lifecycle_manager.cc:38: attempting to create...

``` [2023-01-04 22:03:49,667 D 3319491 3319536] (raylet) store.cc:401: Received create request for object c8ef45ccd0112571ffffffffffffffffffffffff0100000001000000 assigned request ID 19, 100000003 bytes [2023-01-04 22:03:49,667 D 3319491 3319536] (raylet) object_lifecycle_manager.cc:38: attempting to create...

I feel the root cause is related to the lineage reconstruction. Basically, A->B and when B is deleted, A is still there because the lineage count 1 (fail to decrease)....

verified, turn off lineage reconstruction spilling disappeared: ``` RAY_lineage_pinning_enabled=0 python ./leak.py ```

@ericl verified that your theory is correct. Summary: - del ref ==> no leak - del data or del nothing ==> leak - disable lineage ==> no leak I think...