Chi Wang

Can you patch eda0fe4399b85d3bbc39fda2701287e014ecb919 and check whether this issue is still reproducible?

For the second invocation, `--remote_cache` and `--remote_execution` point to different endpoints. I don't know how the remote executor was setup but it seems like it doesn't read the AC/CAS from...

The repro doesn't make sense to me: you cannot combine multiple disk caches, only the last `--disk_cache` is used. Things have changed a lot since Bazel 5. For example we...

It could also be that Bazel was downloading toplevel outputs in the background if you set `--remote_download_toplevel` (although I have no ideas why there wasn't progress report). For presubmits, can...

Thanks for the stack trace. It looks like the the hanging is caused by `--experimental_remote_upload_async`. I will work on a fix.

@snakethatlovesstaticlibs Can you please share the stack trace when Bazel is hanging? (you can capture the stack trace by `jcmd Thread.dump_to_file `.)

~~Probably this: https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/skyframe/FilesystemValueChecker.java;l=121;drc=e03de754ec15096fcd1f701ea0b2d4c09c4b97a1~~ Can you repro with `--noexperimental_merged_skyframe_analysis_execution`?

> so they are still missing on the next invocation. Are these outputs should be downloaded (e.g. toplevel outputs, or specified by `--remote_download_regex`)? If Bazel marks an action dirty to...

skymeld calls `skyframeExecutor.detectModifiedOutputFiles` when the first toplevel target is analyzed. However, the `RemoteArtifactChecker` for BwoB needs full analyze result to be able to correctly determine which action should be marked...

> This is the root cause: Argh, I intended to post this link but somehow my clipboard is corrupted.