Slow fish completions

[Kevin-Mok]

Context (Environment)

• Terminal: rxvt-unicode-pixbuf 9.22-2
• Shell: fish-3.0.0-1
• Version: watson 1.6.0-1
• Completion: latest version from https://github.com/TailorDev/Watson/commit/f1a2c94c1878a6f0cace8bdd509628990e319c65

Expected Behavior

Completions as fast as zsh. Here I am using watson add (wa is my alias) in zsh: The projects almost come up instantly, and the + for the tags right after comes almost immediately.

Current Behavior

Pretty unexpectedly slow completions in fish. The projects take a few seconds to come up, and the + for the tags right takes bizarrely long to show up, even though it's not actually displaying any tags yet. I should also note that my other completions for fish aren't experiencing the same problem.

Proposal

Could the speed of this be improved in any way, or is it just a problem on my own machine?

[jmaupetit]

Hi @Kevin-Mok ! Thanks for the report. We will need relevant inputs from Fish users (which I am not).

[andrewmeyer]

Out of curiosity, how many projects do you have? I only have a few so it could be tied to the time to parse and print the whole list.

edit: I tested again on my local machine and found project and tag completion to be a little slow, but nowhere near as slow as the images you posted. I have maybe 1-1.5 seconds of lagtime

[Kevin-Mok]

I have 4 projects (you can see also this in the GIF's :wink:) and 10 tags, but it's weird that just the + symbol for the tags takes awhile to show up and not even the actual tags. And on zsh, the tags and projects show up almost immediately.

[andrewmeyer]

-- wanted to drop an update on this:

I did notice that zsh caches the output of watson tags, watson projects, and watson frames, where fish (from what I can tell) does not have such a feature. in watson.fish we collect tags, frames, and projects by running the respective watson command, but watson edit does not appear to have the same issues that watson start does.

I'm still looking at ways of speeding this up. It is possible that the completion is being slowed down by iterating over the command looking for projects, and then iterating over it again looking for tags.

I'll keep digging

[jmaupetit]

Thanks @andrewmeyer!

[Kevin-Mok]

Yeah, so I think that the actual watson projects and watson tags commands are the bottleneck for the completion. Is watson scanning all the frames on-the-fly for the names?

[jmaupetit]

Is watson scanning all the frames on-the-fly for the names?

Yes.

[andrewmeyer]

It might be worthwhile to explore adding a limit (or an option to limit) the depth watson scans for projects and tags. In the case of this completion, There is a high probability that we would only be interested in the last 7-30 days worth of frames (or the last 100 frames).

I was also thinking last night about the feasibility of using memcached for either the completion side or for watson if it is installed on the system.

[jmaupetit]

AFAIR @k4nar battle tested JSON blob (frames) loading performance and we were rather surprised by the json module parsing efficiency. Thus, we decided to stick to a simple JSON file-based storage. And I have to say that it works pretty well (even after many years using it!). But maybe it's time to re-think this component to be more efficient.

I've not a clear idea yet about what could be the alternatives we should consider to store/request the frames database, but I have some requirements in mind to keep the original simplicity of Watson:

1. it should be a lightweight dependency (not sure Pandas is for example),
2. it should not imply running a server or a daemon,
3. it should not be cache-based, as IMO it would cause more troubles than the problem it will solve.

Of course, we can discuss those requirements. Feel free to suggest alternatives to benchmark! :heart:

[andrewmeyer]

I don't think I have enough frames to stress-test (42 frames). I am only seeing ~200ms responses for watson tags watson projects and watson frames

I agree that running a separate daemon for storing the frames would be difficult requirement and should be avoided.

[jmaupetit]

We have a script to generate random fixtures [1], I think it could be modified to generate fake data to test loading/querying performances.

[1] https://github.com/TailorDev/Watson/blob/master/scripts/fuzzer.py

[Kevin-Mok]

Is SQLite a viable storage alternative? Also, what about the possibility of keeping a list of projects/names in a list and updating it whenever a new frame is added? Then, all the frames wouldn't needed to be scanned every time projects or tags is called, and it could be used for completions as zsh already does.

[jmaupetit]

Is SQLite a viable storage alternative?

It could, but it changes many things. I would prefer keeping data unstructured.

Also, what about the possibility of keeping a list of projects/names in a list and updating it whenever a new frame is added?

And also whenever we edit frames or rename projects, tags... This is more or less equivalent to cache invalidation :sweat_smile:

[Kevin-Mok]

So based on watson frames | wc -l, I'm at 56 frames now, and the problem is compounding. Here's what the completion looks like at the moment:

So as a temporary workaround, I just created a manual cache with watson projects > projects, changed this line to command cat /home/kevin/.config/watson/projects and did the same with tags. Here's the result:

:sweat_smile:

EDIT: I was looking into the zsh completions for watson, and it seems that the author invalidated the cache when the frame file is older than the cache file. It seems to be working fine like this too. @jmaupetit What do you think about using this method for cache invalidation?

[andrewmeyer]

I wrote a much larger fuzzer to test your PR with, I'm hoping it will finish adding frames in an hour or two (I might have made it too large, hah)