flyte icon indicating copy to clipboard operation
flyte copied to clipboard

Add prefetch functionality for paginator

Open zychen5186 opened this issue 1 year ago • 2 comments

TL;DR

The get function takes more time to fetch data from the remote server as the amount increases, hence we only fetch 100 rows initially, and prefetch 100 rows of data at once while the user scrolls through pages to enhance the user experience.

Type

  • [ ] Bug Fix
  • [X] Feature
  • [ ] Plugin

Are all requirements met?

  • [X] Code completed
  • [X] Smoke tested
  • [ ] Unit tests added
  • [ ] Code documentation added
  • [ ] Any pending items have an associated Issue

Complete description

For testing run:

make compile
./bin/flytectl get execution -p my-project -d development -i

We initially fetch the first 100 rows when the paginator is called and start to prefetch the next batch (next 100 rows). Prefetch will be triggered when the current page exceeds a prefetch threshold. The first or last batch will be discarded when it exceeds the cache limit (currently set to 1000 rows). We added a loading animation to indicate the fetching process is ongoing. Apr-27-2024 17-32-22

Users can use --filter.page flag to specify their starting page, if not specified, page will start from 1. Apr-28-2024 16-21-42

Tracking Issue

https://github.com/flyteorg/flyte/issues/4440

Follow-up issue

Follow-up of https://github.com/flyteorg/flytectl/pull/473

Big thanks to @troychiu for discussing and debugging with me!! :raised_hands: :heart:

zychen5186 avatar May 01 '24 06:05 zychen5186

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

welcome[bot] avatar May 01 '24 06:05 welcome[bot]

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 61.10%. Comparing base (cb57beb) to head (228b346).

Files Patch % Lines
flytectl/cmd/get/execution.go 0.00% 5 Missing and 1 partial :warning:
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5310   +/-   ##
=======================================
  Coverage   61.10%   61.10%           
=======================================
  Files         794      794           
  Lines       51213    51213           
=======================================
  Hits        31295    31295           
  Misses      17037    17037           
  Partials     2881     2881           
Flag Coverage Δ
unittests-datacatalog 69.31% <ø> (ø)
unittests-flyteadmin 58.90% <ø> (ø)
unittests-flytecopilot 17.79% <ø> (ø)
unittests-flytectl 68.30% <0.00%> (ø)
unittests-flyteidl 79.30% <ø> (ø)
unittests-flyteplugins 61.94% <ø> (ø)
unittests-flytepropeller 57.32% <ø> (ø)
unittests-flytestdlib 65.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar May 01 '24 08:05 codecov[bot]

@zychen5186 One of the flytectl tests is failing. Mind taking a look?

pingsutw avatar May 07 '24 02:05 pingsutw

@zychen5186 One of the flytectl tests is failing. Mind taking a look?

No problem

zychen5186 avatar May 07 '24 02:05 zychen5186

Congrats on merging your first pull request! 🎉

welcome[bot] avatar May 09 '24 08:05 welcome[bot]