normandy icon indicating copy to clipboard operation
normandy copied to clipboard
verified publisher icon mozilla

Normandy CDN Migration

Open tiftran opened this issue 3 years ago • 4 comments

Migrate over to GCP

  • [x] Use nginx/varnish to replicate the exact same responses we serve now, as an interim solution, as a diff dns first
  • [x] Have QA do a sanity check by setting desktop to point to diff dns, run test suite for normandy
  • [ ] ~Use uptake telemetry to get healthy baseline of normandy calls~ ~Start with a 20% switch over and look at uptake telemetry after a day to ensure normandy calls are as expected~
  • [x] Switch over to original dns(100% if previous step is fine) Note: for the interest of time and cost and deciding it was somewhat low risk, we rolled out incrementally throughout a single day
  • [x] Roll out 1.45.1 to prod which include caching changes to the v1 endpoint

Follow up work.....

  • [ ] Identify consumers of Normandy

    • [ ] Analyze user agents to understand the distribution of clients/versions/domains calling/being called
    • [ ] identify all the remote calls made by Normandy desktop client

  • [ ] Make the code changes for headers that we can get rid of

    • [ ] Run a dummy experiment to get baseline
    • [ ] Explore making server side changes to deprecate the dependency on certain headers ref:docs
    • [ ] Implement changes to remove headers from investigation (Normandy Server)
    • [ ] Implement changes (if any) to Normandy Client
    • [ ] Migrate CDN again to reflect code changes
    • [ ] Run a dummy experiment to ensure there are no breaking changes
    • [ ] Setup manual QA process to test clients explicitly hitting old and new CDN

  • [ ] Identity which endpoints are no longer used and which arent for deprecation plan

tiftran avatar Nov 23 '22 21:11 tiftran

Uptake telemetry:

https://sql.telemetry.mozilla.org/queries/67658#219874

jaredlockhart avatar Nov 23 '22 21:11 jaredlockhart

image

jaredlockhart avatar Nov 23 '22 21:11 jaredlockhart

QA Jira: https://mozilla-hub.atlassian.net/browse/QA-1701?atlOrigin=eyJpIjoiNDBmODZlMzUzNmU5NDM1OWIzZDg5NGFhYWNhMWVlZDQiLCJwIjoiamlyYS1zbGFjay1pbnQifQ

tiftran avatar Dec 06 '22 19:12 tiftran

Summary of what has happened so far:

Nov 29th: Normandy new CDN setup for Normandy was available in the stage environment

Nov 30th: tested 1%, 10%, 25% of requests were moved to the new CDN

Dec 1st: 100% of requests were moved to the new CDN

Dec 2nd: #2342 PR was merged

Dec 5th: v1.145.1 was tagged and pushed to stage, QA request was made

tiftran avatar Dec 13 '22 08:12 tiftran