origin icon indicating copy to clipboard operation
origin copied to clipboard

OPNET-582,OPNET-585: Add monitor test for on-prem keepalived

Open cybertron opened this issue 1 year ago • 76 comments

This will provide a timeline of VIP moves while also verifying that the correct priority is present in the keepalived logs.

cybertron avatar Dec 19 '24 21:12 cybertron

@cybertron: This pull request references OPNET-582 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

This pull request references OPNET-585 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

This will provide a timeline of VIP moves while also verifying that the correct priority is present in the keepalived logs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot avatar Dec 19 '24 21:12 openshift-ci-robot

/hold

Looks like I'm still missing a piece to construct the timeline.

cybertron avatar Dec 20 '24 15:12 cybertron

/test e2e-metal-ipi

cybertron avatar Dec 20 '24 17:12 cybertron

@cybertron: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

/test e2e-aws-jenkins
/test e2e-aws-ovn-edge-zones
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-image-registry
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial
/test e2e-gcp-ovn
/test e2e-gcp-ovn-builds
/test e2e-gcp-ovn-image-ecosystem
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test images
/test lint
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
/test e2e-agnostic-ovn-cmd
/test e2e-aws
/test e2e-aws-csi
/test e2e-aws-disruptive
/test e2e-aws-etcd-certrotation
/test e2e-aws-etcd-recovery
/test e2e-aws-ovn
/test e2e-aws-ovn-cgroupsv2
/test e2e-aws-ovn-etcd-scaling
/test e2e-aws-ovn-ipsec-serial
/test e2e-aws-ovn-kube-apiserver-rollout
/test e2e-aws-ovn-kubevirt
/test e2e-aws-ovn-single-node
/test e2e-aws-ovn-single-node-serial
/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-single-node-techpreview-serial
/test e2e-aws-ovn-single-node-upgrade
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-rollback
/test e2e-aws-ovn-upi
/test e2e-aws-ovn-virt-techpreview
/test e2e-aws-proxy
/test e2e-azure
/test e2e-azure-ovn-etcd-scaling
/test e2e-azure-ovn-upgrade
/test e2e-baremetalds-kubevirt
/test e2e-external-aws
/test e2e-external-aws-ccm
/test e2e-external-vsphere-ccm
/test e2e-gcp-csi
/test e2e-gcp-disruptive
/test e2e-gcp-fips-serial
/test e2e-gcp-ovn-etcd-scaling
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-ovn-techpreview
/test e2e-gcp-ovn-techpreview-serial
/test e2e-hypershift-conformance
/test e2e-metal-ipi-ovn
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-dualstack-local-gateway
/test e2e-metal-ipi-ovn-kube-apiserver-rollout
/test e2e-metal-ipi-serial
/test e2e-metal-ipi-serial-ovn-ipv6
/test e2e-metal-ipi-virtualmedia
/test e2e-metal-ovn-single-node-live-iso
/test e2e-metal-ovn-single-node-with-worker-live-iso
/test e2e-openstack-ovn
/test e2e-openstack-serial
/test e2e-vsphere
/test e2e-vsphere-ovn-dualstack-primaryv6
/test e2e-vsphere-ovn-etcd-scaling
/test okd-e2e-gcp
/test okd-scos-e2e-aws-ovn
/test okd-scos-images

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd
pull-ci-openshift-origin-master-e2e-aws-csi
pull-ci-openshift-origin-master-e2e-aws-ovn-cgroupsv2
pull-ci-openshift-origin-master-e2e-aws-ovn-edge-zones
pull-ci-openshift-origin-master-e2e-aws-ovn-fips
pull-ci-openshift-origin-master-e2e-aws-ovn-kube-apiserver-rollout
pull-ci-openshift-origin-master-e2e-aws-ovn-microshift
pull-ci-openshift-origin-master-e2e-aws-ovn-microshift-serial
pull-ci-openshift-origin-master-e2e-aws-ovn-serial
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade
pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade
pull-ci-openshift-origin-master-e2e-gcp-csi
pull-ci-openshift-origin-master-e2e-gcp-ovn
pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade
pull-ci-openshift-origin-master-e2e-gcp-ovn-upgrade
pull-ci-openshift-origin-master-e2e-hypershift-conformance
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout
pull-ci-openshift-origin-master-e2e-openstack-ovn
pull-ci-openshift-origin-master-images
pull-ci-openshift-origin-master-lint
pull-ci-openshift-origin-master-okd-scos-e2e-aws-ovn
pull-ci-openshift-origin-master-unit
pull-ci-openshift-origin-master-verify
pull-ci-openshift-origin-master-verify-deps

In response to this:

/test e2e-metal-ipi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Dec 20 '24 17:12 openshift-ci[bot]

/test e2e-metal-ipi-serial

cybertron avatar Dec 20 '24 17:12 cybertron

/test e2e-metal-ipi-ovn-dualstack

cybertron avatar Jan 03 '25 18:01 cybertron

/test e2e-metal-ipi-ovn-dualstack

cybertron avatar Jan 06 '25 23:01 cybertron

/test e2e-vsphere /test e2e-openstack-ovn

Looks like this is now working well in the metal job. Just make sure it's working for the other on-prem platforms as well.

cybertron avatar Jan 07 '25 15:01 cybertron

/hold cancel /cc @mkowalski

Apparently the VSphere job in this repo is UPI so there was nothing to be tested, but it looks like it worked fine in OpenStack too so I think we're good to go.

cybertron avatar Jan 07 '25 20:01 cybertron

/lgtm

mkowalski avatar Feb 19 '25 16:02 mkowalski

/retest-required

cybertron avatar Mar 11 '25 14:03 cybertron

/retest-required

cybertron avatar Mar 17 '25 20:03 cybertron

/retest-required

cybertron avatar Apr 03 '25 16:04 cybertron

/assign @dgoodwin

cybertron avatar Apr 07 '25 17:04 cybertron

/payload-job periodic-ci-openshift-release-master-nightly-4.18-upgrade-from-stable-4.17-e2e-metal-ipi-upgrade-ovn-ipv6

cybertron avatar Apr 18 '25 21:04 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.18-upgrade-from-stable-4.17-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3e2aacc0-1c9e-11f0-9032-79be10f37f93-0

openshift-ci[bot] avatar Apr 18 '25 21:04 openshift-ci[bot]

/payload-job periodic-ci-openshift-release-master-nightly-4.18-upgrade-from-stable-4.17-e2e-metal-ipi-upgrade-ovn-ipv6

Previous job failed for what appear to be unrelated reasons and there seem to be no intervals generated.

cybertron avatar Apr 21 '25 20:04 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.18-upgrade-from-stable-4.17-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5db5c2f0-1ef1-11f0-8dfa-0e0b80a27251-0

openshift-ci[bot] avatar Apr 21 '25 20:04 openshift-ci[bot]

/payload-job periodic-ci-openshift-release-master-nightly-4.19-upgrade-from-stable-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

Same result, trying a different release.

cybertron avatar Apr 22 '25 13:04 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.19-upgrade-from-stable-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3b8d0d20-1f80-11f0-9cbd-bd2d9ab46ecc-0

openshift-ci[bot] avatar Apr 22 '25 13:04 openshift-ci[bot]

/test e2e-metal-ipi-ovn-dualstack

Payload jobs keep failing, just a sanity check that the regular test job still works.

cybertron avatar Apr 22 '25 19:04 cybertron

/payload-job periodic-ci-openshift-release-master-nightly-4.19-upgrade-from-stable-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

Regular dual stack job passed so I don't think I broke something with the new tests. One more try and then I'll go looking for help on why these upgrade jobs keep failing.

cybertron avatar Apr 23 '25 14:04 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.19-upgrade-from-stable-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4b975700-204f-11f0-8f94-f76b24df8afc-0

openshift-ci[bot] avatar Apr 23 '25 14:04 openshift-ci[bot]

/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-upgrade-ovn-ipv6

cybertron avatar May 02 '25 19:05 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c1c2e520-278e-11f0-90c8-493c8035b498-0

openshift-ci[bot] avatar May 02 '25 19:05 openshift-ci[bot]

Risk analysis has seen new tests most likely introduced by this PR. Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 166162a2a64f0334ed86f419ba81b09fd9288e09

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Load Balancer"] on-prem loadbalancer must achieve full priority" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived cleanup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived collection" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived interval construction" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived setup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived test evaluation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-edge-zones High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived writing to storage" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Load Balancer"] on-prem loadbalancer must achieve full priority" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived cleanup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived collection" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived interval construction" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived setup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived test evaluation" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived writing to storage" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[Jira:"Networking / On-Prem Load Balancer"] on-prem loadbalancer must achieve full priority" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived cleanup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived collection" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived interval construction" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived setup" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived test evaluation" is a new test that was not present in all runs against the current commit.
(...showing 20 of 28 rows)

New tests seen in this PR at sha: 166162a2a64f0334ed86f419ba81b09fd9288e09

  • "[Jira:"Networking / On-Prem Load Balancer"] on-prem loadbalancer must achieve full priority" [Total: 41, Pass: 41, Fail: 0, Flake: 32]
  • "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived cleanup" [Total: 41, Pass: 41, Fail: 0, Flake: 0]
  • "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived collection" [Total: 41, Pass: 41, Fail: 0, Flake: 0]
  • "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived interval construction" [Total: 41, Pass: 41, Fail: 0, Flake: 0]
  • "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived setup" [Total: 41, Pass: 41, Fail: 0, Flake: 0]
  • "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived test evaluation" [Total: 41, Pass: 41, Fail: 0, Flake: 0]
  • "[Jira:"Networking / On-Prem Loadbalancer"] monitor test on-prem-keepalived writing to storage" [Total: 41, Pass: 41, Fail: 0, Flake: 0]

openshift-trt[bot] avatar May 03 '25 00:05 openshift-trt[bot]

/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm-upgrade

Trying an upgrade job without ipv6 because I'm seeing logs that suggest there might be an issue with it in the payload jobs.

cybertron avatar May 07 '25 19:05 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4ab4cca0-2b7b-11f0-92de-9c85e328fe47-0

openshift-ci[bot] avatar May 07 '25 19:05 openshift-ci[bot]

/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm-upgrade

cybertron avatar May 14 '25 13:05 cybertron

@cybertron: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e88a5390-30c7-11f0-8d98-07cd8bd77e21-0

openshift-ci[bot] avatar May 14 '25 13:05 openshift-ci[bot]