Add prefetch functionality for paginator
TL;DR
The get function takes more time to fetch data from the remote server as the amount increases, hence we only fetch 100 rows initially, and prefetch 100 rows of data at once while the user scrolls through pages to enhance the user experience.
Type
- [ ] Bug Fix
- [X] Feature
- [ ] Plugin
Are all requirements met?
- [X] Code completed
- [X] Smoke tested
- [ ] Unit tests added
- [ ] Code documentation added
- [ ] Any pending items have an associated Issue
Complete description
For testing run:
make compile
./bin/flytectl get execution -p my-project -d development -i
We initially fetch the first 100 rows when the paginator is called and start to prefetch the next batch (next 100 rows).
Prefetch will be triggered when the current page exceeds a prefetch threshold.
The first or last batch will be discarded when it exceeds the cache limit (currently set to 1000 rows).
We added a loading animation to indicate the fetching process is ongoing.
Users can use --filter.page flag to specify their starting page, if not specified, page will start from 1.
Tracking Issue
https://github.com/flyteorg/flyte/issues/4440
Follow-up issue
Follow-up of https://github.com/flyteorg/flytectl/pull/473
Big thanks to @troychiu for discussing and debugging with me!! :raised_hands: :heart:
Thank you for opening this pull request! 🙌
These tips will help get your PR across the finish line:
- Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
- Sign off your commits (Reference: DCO Guide).
Codecov Report
Attention: Patch coverage is 0% with 6 lines in your changes are missing coverage. Please review.
Project coverage is 61.10%. Comparing base (
cb57beb) to head (228b346).
| Files | Patch % | Lines |
|---|---|---|
| flytectl/cmd/get/execution.go | 0.00% | 5 Missing and 1 partial :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #5310 +/- ##
=======================================
Coverage 61.10% 61.10%
=======================================
Files 794 794
Lines 51213 51213
=======================================
Hits 31295 31295
Misses 17037 17037
Partials 2881 2881
| Flag | Coverage Δ | |
|---|---|---|
| unittests-datacatalog | 69.31% <ø> (ø) |
|
| unittests-flyteadmin | 58.90% <ø> (ø) |
|
| unittests-flytecopilot | 17.79% <ø> (ø) |
|
| unittests-flytectl | 68.30% <0.00%> (ø) |
|
| unittests-flyteidl | 79.30% <ø> (ø) |
|
| unittests-flyteplugins | 61.94% <ø> (ø) |
|
| unittests-flytepropeller | 57.32% <ø> (ø) |
|
| unittests-flytestdlib | 65.75% <ø> (ø) |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@zychen5186 One of the flytectl tests is failing. Mind taking a look?
@zychen5186 One of the flytectl tests is failing. Mind taking a look?
No problem
Congrats on merging your first pull request! 🎉