chainlink icon indicating copy to clipboard operation
chainlink copied to clipboard
verified publisher icon smartcontractkit

Optimize GitHub CI to Skip Unnecessary Tests Based on File Changes

Open lukaszcl opened this issue 1 year ago • 2 comments

This PR optimizes the GitHub CI workflows by implementing the following changes:

  1. Skip E2E tests on Markdown changes: E2E tests will not run when only Markdown files are modified.
  2. Isolate unit and E2E tests:
    • Unit tests will not run when only end-to-end test files are changed.
    • E2E tests will not run when only unit test files are changed.

Tests conducted for this PR:

  • [x] Unit tests only changes: Verified that a PR with unit test changes does not build the Chainlink image or run E2E tests.
    • https://github.com/smartcontractkit/chainlink/pull/14593
  • [x] E2E tests only changes: Verified that a PR with E2E test changes runs only E2E tests and skips unit tests.
    • https://github.com/smartcontractkit/chainlink/pull/14587
  • [x] Documentation changes: Confirmed that a PR with documentation updates does not trigger any tests.
    • https://github.com/smartcontractkit/chainlink/pull/14590
  • [x] Workflow updates: Ensured that a PR with workflow-related changes runs the affected workflow.
    • https://github.com/smartcontractkit/chainlink/pull/14591

NEW tests:

  • [x] E2E test changed: Does not trigger unit tests. But triggers lint - https://github.com/smartcontractkit/chainlink/pull/14792
  • [x] Unit test changed: Does not trigger E2E tests - https://github.com/smartcontractkit/chainlink/pull/14793
  • [x] README changed: Triggers unit tests and lint. Does not trigger E2E tests - https://github.com/smartcontractkit/chainlink/pull/14794
  • [x] Both E2E and unit tests changed: Triggers all tests - https://github.com/smartcontractkit/chainlink/pull/14795

lukaszcl avatar Sep 27 '24 08:09 lukaszcl

Below is an analysis created by an LLM. Be mindful of hallucinations and verify accuracy.

WF: CI Core#8d34ce3

1. Push request failed due to server unavailability:Flakey Test Detection

Source of Error: 
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2104111Z 2024/10/16 10:46:39 Error re-running flakey tests: push request failed: status=503, body=<html>
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2106172Z <head><title>503 Service Temporarily Unavailable</title></head>
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2107272Z <body>
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2108281Z <center><h1>503 Service Temporarily Unavailable</h1></center>
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2109377Z <hr><center>nginx</center>
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2109994Z </body>
Flakey Test Detection	Re-run tests	2024-10-16T10:46:39.2110424Z </html>

Why: The error occurred because the server that was supposed to handle the push request was temporarily unavailable, as indicated by the HTTP 503 status code.

Suggested fix: Ensure the server is running and can handle requests. If the issue persists, consider implementing retry logic in the workflow to handle temporary server unavailability.

github-actions[bot] avatar Oct 14 '24 10:10 github-actions[bot]

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

cl-sonarqube-production[bot] avatar Oct 21 '24 10:10 cl-sonarqube-production[bot]

AER Report: CI Core

aer_workflow , commit , Detect Changes , Clean Go Tidy & Generate , Core Tests (go_core_tests) , Core Tests (go_core_ccip_deployment_tests) , Core Tests (go_core_race_tests) , Core Tests (go_core_fuzz) , lint , SonarQube Scan

1. Test failures encountered:

[Run tests] [2024-10-21T10:54:11.2700446Z] [Encountered test failures.]

Source of Error: 
Run tests	2024-10-21T10:54:11.2700446Z Encountered test failures.
Run tests	2024-10-21T10:54:11.2701129Z go_core_tests exiting with code 1
Run tests	2024-10-21T10:54:11.2720911Z ##[error]Process completed with exit code 1.

Why: The test suite encountered one or more test failures, causing the overall test run to exit with a non-zero status code.

Suggested fix: Review the individual test cases to identify and fix the failing tests. Ensure all dependencies and configurations are correctly set up.

2. Panic due to port allocation failure:

[Print Filtered Test Results] [2024-10-21T10:54:12.9240853Z] [panic: freeport: cannot allocate port block]

Source of Error: 
Print Filtered Test Results	2024-10-21T10:54:12.9240853Z [0;31mpanic: freeport: cannot allocate port block [recovered] [0m
Print Filtered Test Results	2024-10-21T10:54:12.9241593Z 	panic: freeport: cannot allocate port block
Print Filtered Test Results	2024-10-21T10:54:12.9242153Z goroutine 388 [running]:
Print Filtered Test Results	2024-10-21T10:54:12.9243017Z testing.tRunner.func1.2({0x2bb1ce0, 0x37023b0})
Print Filtered Test Results	2024-10-21T10:54:12.9243853Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1631 +0x24a
Print Filtered Test Results	2024-10-21T10:54:12.9244538Z testing.tRunner.func1()
Print Filtered Test Results	2024-10-21T10:54:12.9245190Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1634 +0x377
Print Filtered Test Results	2024-10-21T10:54:12.9246647Z panic({0x2bb1ce0?, 0x37023b0?})
Print Filtered Test Results	2024-10-21T10:54:12.9247320Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/panic.go:770 +0x132
Print Filtered Test Results	2024-10-21T10:54:12.9247952Z github.com/hashicorp/consul/sdk/freeport.alloc()
Print Filtered Test Results	2024-10-21T10:54:12.9248743Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:274 +0xad
Print Filtered Test Results	2024-10-21T10:54:12.9249530Z github.com/hashicorp/consul/sdk/freeport.initialize()
Print Filtered Test Results	2024-10-21T10:54:12.9250371Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:124 +0x2d7
Print Filtered Test Results	2024-10-21T10:54:12.9251100Z sync.(*Once).doSlow(0x2cf8c20?, 0x4f3732?)
Print Filtered Test Results	2024-10-21T10:54:12.9252062Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/once.go:74 +0xc2
Print Filtered Test Results	2024-10-21T10:54:12.9252543Z sync.(*Once).Do(...)
Print Filtered Test Results	2024-10-21T10:54:12.9252971Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/once.go:65
Print Filtered Test Results	2024-10-21T10:54:12.9253533Z github.com/hashicorp/consul/sdk/freeport.Take(0x1)
Print Filtered Test Results	2024-10-21T10:54:12.9254346Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:303 +0xe9
Print Filtered Test Results	2024-10-21T10:54:12.9255267Z github.com/hashicorp/consul/sdk/freeport.GetN({0x372d2f8, 0xc0013589c0}, 0x1)
Print Filtered Test Results	2024-10-21T10:54:12.9256185Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:427 +0x48
Print Filtered Test Results	2024-10-21T10:54:12.9257086Z github.com/hashicorp/consul/sdk/freeport.GetOne({0x372d2f8, 0xc0013589c0})
Print Filtered Test Results	2024-10-21T10:54:12.9257976Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:449 +0x36
Print Filtered Test Results	2024-10-21T10:54:12.9259148Z github.com/smartcontractkit/chainlink/v2/core/services/p2p/wrapper_test.TestPeerWrapper_CleanStartClose(0xc0013589c0)
Print Filtered Test Results	2024-10-21T10:54:12.9260271Z 	/home/runner/work/chainlink/chainlink/core/services/p2p/wrapper/wrapper_test.go:25 +0xae
Print Filtered Test Results	2024-10-21T10:54:12.9260959Z testing.tRunner(0xc0013589c0, 0x3490fe0)
Print Filtered Test Results	2024-10-21T10:54:12.9261808Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1689 +0xfb
Print Filtered Test Results	2024-10-21T10:54:12.9262385Z created by testing.(*T).Run in goroutine 1
Print Filtered Test Results	2024-10-21T10:54:12.9262996Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1742 +0x390

Why: The test TestPeerWrapper_CleanStartClose panicked because it could not allocate a port block using the freeport library, likely due to exhaustion of available ports.

Suggested fix: Ensure that the system has enough available ports. Consider configuring the test environment to avoid port conflicts or increasing the range of available ports.

github-actions[bot] avatar Oct 21 '24 10:10 github-actions[bot]