microsoft
Very poor performance when opening/searching for very large files (2GB)
-
Search performance problem: When performing keyword search, a card appears, and the response latency is significant, which affects the operation fluency.
-
Long line display defect: The current display support for long lines is insufficient, and it is recommended to add a horizontal scrollbar for the implementation of complete content browsing.
Very poor performance when opening
So, is the performance issue with opening files or with searching in them? Your description only talks about the latter.
Search performance problem: When performing keyword search, a card appears, and the response latency is significant, which affects the operation fluency.
We rely on ICU for Unicode-correct, case-insensitive text search. It caps out at <500MB/s so searching in a 2GB file can take up to 4s (or more if your CPU is slow). Our focus is on common file sizes first, because we also want to focus on a small file size. Integrating a fast case-insensitive searcher, particularly with Regex support, would triple the binary size.
Long line display defect: The current display support for long lines is insufficient, and it is recommended to add a horizontal scrollbar for the implementation of complete content browsing.
Computing the horizontal scrollbar width is fairly expensive. The reason other editors can do it is because they either have existed for a long time or have a large team behind it to work on such optimizations. In our case we got 1 part-time developer (me), so such a feature is considered lowest priority (compared to e.g. syntax highlighting, etc.).
https://github.com/cessen/ropey seems like its purpose is to resolve this sort of issue. But I imagine it would require a lot of refactoring, and probably inflate the binary size too much.
The overwhelming majority of CPU time is spent in the read syscall, even if the file is fully cached in memory. Whether ropey is used or not would not make any difference (i.e. neither good nor bad).
That said, it would not require a lot of refactoring. All you have to do is implement the WriteableDocument trait on it and use it in the TextBuffer struct.
I'll close this issue for now due to no response to my question. We currently have no plans to address the search performance. Editors specialized for large files may fare better in that regard.