integrations-core icon indicating copy to clipboard operation
integrations-core copied to clipboard

Speed up the Hudi test by caching a downloaded file

Open nubtron opened this issue 3 months ago • 3 comments

What does this PR do?

Hudi tests can sometimes take more than 30 minutes because downloading the Spark archive from https://archive.apache.org/ is slow. This PR introduces caching for that download at the GitHub Actions level.

The cache can also be reused by other integrations. It’s configured via a .github-cache.yml file located in each integration directory.

The .github-cache.yml file defines a list of files (with paths relative to the integrations directory) and their associated download URLs.

GitHub Actions uses this configuration to cache and restore the files, which helps with slow or unreliable download sources.

The cache has no impact on tests running locally on the developer's environment.

Motivation

Speed up CI.

Review checklist (to be filled by reviewers)

  • [ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • [ ] Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • [ ] If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

nubtron avatar Oct 21 '25 12:10 nubtron

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 88.99%. Comparing base (46169ca) to head (82c6fbf). :warning: Report is 40 commits behind head on master.

Additional details and impacted files
:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Oct 21 '25 13:10 codecov[bot]

⚠️ The qa/skip-qa label has been added with shippable changes

The following files, which will be shipped with the agent, were modified in this PR and the qa/skip-qa label has been added.

You can ignore this if you are sure the changes in this PR do not require QA. Otherwise, consider removing the label.

List of modified files that will be shipped with the agent
hudi/hatch.toml

github-actions[bot] avatar Oct 24 '25 13:10 github-actions[bot]

This PR doesn't actually touch files owned by windows-products. Windows-products was added in the reviewer list when I added some temporary testing code to check Windows compatibility.

nubtron avatar Nov 04 '25 10:11 nubtron