facebook
sl is very slow in git repo
Hello, I know dotgit mode is still being developed. But sl command is very slow in my large mono git repo. Here is a truncated trace. It spends quite some time on a git status command.
If I run git status manually, it does spend quite some time on refreshing index, then sl command becomes fast again.
But if I did some sl change, e.g. amend/commit, sl will be slow again on git status. I'm guessing it spends some time on refreshing index again.
I'm using latest release. Let me know what else info you need.
Glad that dotgit is being worked on, thank you so much!
➜ EDENSCM_LOG=debug sl status
...
2024-08-04T04:39:02.603878Z INFO run:status: workingcopy::workingcopy: enter
2024-08-04T04:39:02.603903Z DEBUG run:status: workingcopy::filesystem::dotgit: pending_changes (DotGitFileSystem) include_ignored=false
2024-08-04T04:39:02.603920Z DEBUG run:status: gitcompat::rungit: git command: ["--git-dir=MY_REPO_PATH/.git", "--no-optional-locks", "status", "--porcelain=1", "--ignore-submodules=dirty", "--untracked-files=all", "--no-renames", "-z", "--ignored=no"]
2024-08-04T04:39:34.597473Z INFO run:status:compute_status: workingcopy::status: enter
It's --no-optional-locks (added by https://github.com/facebook/sapling/commit/6b1912bd137ed2375fcefa1ec570802e855884ca). The difficult choice is:
- Without
--no-optional-locks, status triggered by automation might create the "lock" and other non-automation commands will error out. - With
--no-optional-locks,statuswill eventually be slower, maybe until a checkout (I'm not an expert in this area, and not sure).
Do you use the CLI or ISL? If you use the CLI we can change --no-optional-locks to automation (ISL) only. Then CLI status would probably be okay. If you use ISL exclusively then it's tricker to fix.
I mostly use ISL for commit, stacking, but I do use CLI sometimes, but basically just sl status/diff for some linters that we have.
In addition to the locks, I've noticed that --untracked-files=all adds a substantial amount of time, ~500-700ms, to status. e.g running EDENSCM_LOG=debug sl status, then I take the git args and run them manually, both the git status and sl status take ~900ms. But if I remove --untracked-files all from git status it drops to ~250ms. Weirdly, the man page says that all is the default so I'm not sure why it changes things.
I'd wanted to setup rs-git-fsmonitor to maybe bypass this issue, but ran into the above.
I use CLI exclusively so I would be really interested if --no-optional-locks was applied only to automation 👀
@tpcstld before dotgit, I had a set of sapling hooks configured to sync the state (HEAD etc) between sapling and git. It was a little flaky, but was fast. But the most common problem is that you'd have some background task like VSCode calling git that would create the lock and then an interactive use fails. Point being, I don't think you can necessarily know if something is automation or not.
It looks like the underlying lockfile handling supports a timeout w/ exponential backoff, but status specifically uses the default 0 which fails if the lock is held. It seems like a relatively small change to add an option to add a timeout, and then we can get the best of both worlds.
In addition to the locks, I've noticed that
--untracked-files=alladds a substantial amount of time, ~500-700ms
Do you have lots of untracked files? Would adding them to .gitignore help performance?
I dont recall having a lot of untracked files when I did that test, but I've just re-tested and am not seeing the difference (which would make sense since all is the default). However, with --untracked-files=no the time goes from 1.6s -> 800ms. This is with 2 untracked files.
@tpcstld before dotgit, I had a set of sapling hooks configured to sync the state (HEAD etc) between sapling and git. It was a little flaky, but was fast. But the most common problem is that you'd have some background task like VSCode calling git that would create the lock and then an interactive use fails. Point being, I don't think you can necessarily know if something is automation or not.
Does that mean that you essentially had two copies of the repo, one in "git mode" and another in sapling mode? How did you deal with unpushed changes? But it seems pretty smart I might try something like that.
This seems to be the same issue as I reported in https://github.com/facebook/sapling/issues/1117.
Can you please ELI5, why git amend && git st is fast (0.5s) and seems like everyone is happy with it, and sl amend && sl st is slow (15s, 30x slower)? Why can't sl run the same work as git?
