fides
fides copied to clipboard
Published
20 hours ago •
ethyca
ethyca
Find a way to cache the `npm install` step during `nox -s dev`
Is your feature request related to a specific problem?
nox -s dev takes a really long time....this appears to be primarily due to the fact that npm install step isn't getting cached as node_modules/ is included in the .dockerignore file
Describe the solution you'd like
For the sake of development, we should find a way to cache or use locally installed npm modules to majorly cut down on the time needed to spin up nox -s dev
Describe alternatives you've considered, if any
Not sure what other options might exist yet
Additional context
N/A
@allisonking @ssangervasi do either of you have any thoughts/experience here?
I was thinking we could load local node modules to speed this process up...some kind of check to see if local modules exist than use those, otherwise run an npm install
This might get messy though because we need to guarantee that the versions are the same and such? Maybe npm has some features here I just don't know about that could help solve this problem
It looks like right now any time any file in clients/admin-ui/ is changed it will cause everything blow to be rebuild and skip cache. Would it solve the issue to do the npm install like with pip? So just copy the package.json file and do the npm install, then later copy the rest of the files?
So
# install node modules
COPY clients/admin-ui/ .
RUN npm install
Changes to
COPY clients/admin-ui/package.json .
RUN npm install
Then later do the COPY clients/admin-ui/ .. This copy would probably have to come in the prod step of the multi stage build to make a difference...assuming it would would at all.
Hmm yeah I tried to multi-stage it to make the frontend get out of the backend's way, and only build itself on prod, but I guess now that we have fidesctl-ui, it's getting lumped together in the nox command. In the past I've always avoided using docker for developing frontend stuff, and only used it to build the production pages.
Also, for what it's worth, here are the times it took for my last nox -s dev start up:
➜ fides git:(aking-829-db-dataset) nox -s dev
nox > Running session dev
nox > Creating virtual environment (virtualenv) using python3.9 in .nox/dev
nox > docker build --target=dev --tag ethyca/fidesctl:local .
[+] Building 106.3s (22/22) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 423B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8-slim-buster 0.7s
=> [internal] load build context 14.4s
=> => transferring context: 165.62MB 13.8s
=> [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db7 0.0s
=> CACHED [base 2/2] RUN pip install -U pip 0.0s
=> CACHED [builder 1/14] RUN : && apt-get update && apt-get ins 0.0s
=> CACHED [builder 2/14] RUN : && apt-get update && apt-get ins 0.0s
=> CACHED [builder 3/14] RUN curl https://packages.microsoft.com/keys/m 0.0s
=> CACHED [builder 4/14] RUN curl https://packages.microsoft.com/config 0.0s
=> CACHED [builder 5/14] RUN : && apt-get update && apt-get ins 0.0s
=> CACHED [builder 6/14] COPY dev-requirements.txt dev-requirements.txt 0.0s
=> CACHED [builder 7/14] RUN pip install -r dev-requirements.txt 0.0s
=> CACHED [builder 8/14] COPY requirements.txt requirements.txt 0.0s
=> CACHED [builder 9/14] RUN pip install -r requirements.txt 0.0s
=> CACHED [builder 10/14] COPY optional-requirements.txt optional-requir 0.0s
=> CACHED [builder 11/14] RUN pip install -r optional-requirements.txt 0.0s
=> [builder 12/14] COPY . /fides 23.7s
=> [builder 13/14] WORKDIR /fides 0.0s
=> [builder 14/14] RUN mkdir -p src/fidesapi/build/static 0.9s
=> [dev 1/1] RUN pip install -e ".[all,mssql]" 53.5s
=> exporting to image 12.8s
=> => exporting layers 12.8s
=> => writing image sha256:f9273853b2144bdf8ef68b92b32be9ff94730db6ab9a7 0.0s
=> => naming to docker.io/ethyca/fidesctl:local 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker build --target=frontend --tag ethyca/fidesctl:local-ui .
[+] Building 87.3s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/library/node:16 0.8s
=> [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8f 0.0s
=> => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25ecc 0.0s
=> [internal] load build context 3.6s
=> => transferring context: 4.81MB 3.3s
=> CACHED [frontend 2/5] WORKDIR /fides/clients/admin-ui 0.0s
=> [frontend 3/5] COPY clients/admin-ui/ . 15.0s
=> [frontend 4/5] RUN npm install 2.2s
=> [frontend 5/5] RUN npm run export 57.9s
=> exporting to image 7.7s
=> => exporting layers 7.7s
=> => writing image sha256:7ff4cf97e577247f5ba8c0d7bf9642c518f5fc4e03c87 0.0s
=> => naming to docker.io/ethyca/fidesctl:local-ui 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker-compose up -d fidesctl-ui
So the slowest parts for me are:
-
pip install -e ".[all,mssql]" -
npm run export
In fact, npm install only took 2s! If we want to speed up npm run export, we just shouldn't run it at all in dev mode since we really don't use it in development (like what I suggested here: https://github.com/ethyca/fides/pull/757#discussion_r900158484)
@allisonking out of curiosity did npm install run from cache? That would make a big difference if it did.
@allisonking good catch there! definitely would be low-hanging fruit to move npm run export to another stage
I'm also a big curious why the pip install took so long, since all of those packages should be cached 🤔
my run times:
docker build --target=dev --tag ethyca/fidesctl:local .
[+] Building 53.1s (22/22) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8-slim-buster 1.5s
=> [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a 0.0s
=> [internal] load build context 12.0s
=> => transferring context: 100.50MB 11.5s
=> CACHED [base 2/2] RUN pip install -U pip 0.0s
=> CACHED [builder 1/14] RUN : && apt-get update && apt-get instal 0.0s
=> CACHED [builder 2/14] RUN : && apt-get update && apt-get instal 0.0s
=> CACHED [builder 3/14] RUN curl https://packages.microsoft.com/keys/micr 0.0s
=> CACHED [builder 4/14] RUN curl https://packages.microsoft.com/config/de 0.0s
=> CACHED [builder 5/14] RUN : && apt-get update && apt-get instal 0.0s
=> CACHED [builder 6/14] COPY dev-requirements.txt dev-requirements.txt 0.0s
=> CACHED [builder 7/14] RUN pip install -r dev-requirements.txt 0.0s
=> CACHED [builder 8/14] COPY requirements.txt requirements.txt 0.0s
=> CACHED [builder 9/14] RUN pip install -r requirements.txt 0.0s
=> CACHED [builder 10/14] COPY optional-requirements.txt optional-requireme 0.0s
=> CACHED [builder 11/14] RUN pip install -r optional-requirements.txt 0.0s
=> [builder 12/14] COPY . /fides 21.3s
=> [builder 13/14] WORKDIR /fides 0.1s
=> [builder 14/14] RUN mkdir -p src/fidesapi/build/static 1.2s
=> [dev 1/1] RUN pip install -e ".[all,mssql]" 10.2s
=> exporting to image 6.6s
=> => exporting layers 6.6s
=> => writing image sha256:25122a8a620a2977cbe534426777033b8440ed853407645e 0.0s
=> => naming to docker.io/ethyca/fidesctl:local 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker build --target=frontend --tag ethyca/fidesctl:local-ui .
[+] Building 63.7s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/library/node:16 1.1s
=> [internal] load build context 6.7s
=> => transferring context: 4.96MB 6.4s
=> [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9e 0.0s
=> CACHED [frontend 2/5] WORKDIR /fides/clients/admin-ui 0.0s
=> [frontend 3/5] COPY clients/admin-ui/ . 12.8s
=> [frontend 4/5] RUN npm install 35.2s
=> [frontend 5/5] RUN npm run export 0.8s
=> exporting to image 7.0s
=> => exporting layers 7.0s
=> => writing image sha256:15ac5b4abd31efb09c2354ffa4968341b7a5ef60c8a2f6b9 0.0s
=> => naming to docker.io/ethyca/fidesctl:local-ui
curious to see that we got some drastically different times for some steps, our caches must've been different
@allisonking out of curiosity did
npm installrun from cache? That would make a big difference if it did.
How do I tell?
And yeah that's strange we got such different results @ThomasLaPiana 🤔 presumably I work more on the frontend files and you on the backend, but then I'd expect the reverse since we're modifying those files more 😵
some of it might be down to platform differences (i believe mac's run on a VM of some sort, I'm on windows so its running directly on linux subsystem) but I'm not sure what impact that would have functionally on different steps. It is curious though
How do I tell?
I don't know of a way to know for sure other than to delete the container all together. docker system prune --all -f. Then do the build. That way you are sure it isn't using cache.
I'd be curious to know what results you both get with no cache at all, then with a rebuild right after when there is cache. You would be sure to be in the same states of cache that way so in theory you would see close to the same times. That would at least tell if the initial difference you saw was related to cache.
nox -s clean -> nox -s "build(test)"
nox > docker build --target=prod --tag ethyca/fidesctl:local .
[+] Building 244.9s (30/30) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 2.85kB 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 459B 0.0s
=> [internal] load metadata for docker.io/library/node:16 1.9s
=> [internal] load metadata for docker.io/library/python:3.8-slim-buster 1.4s
=> [internal] load build context 126.4s
=> => transferring context: 691.02MB 126.2s
=> [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9 33.1s
=> => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc08 0.0s
=> => sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf9118 1.21kB / 1.21kB 0.0s
=> => sha256:fa6e77edbfa850fbcdc38800b429b7d89bf5eff0fae858 2.21kB / 2.21kB 0.0s
=> => sha256:e6b2a509415bb3adc86fc84ed6b3d9d896e40b34f0dd3a 7.74kB / 7.74kB 0.0s
=> => sha256:ea267e4631a981caf2841a7e9a1faf29cef9d020c437 50.44MB / 50.44MB 7.9s
=> => sha256:8a014c92148934973210d840dc7cfed53e0afba38d839a 7.86MB / 7.86MB 2.5s
=> => sha256:293ff1be7001d642a624409e2d5f90e7708ef7e6f1a7 10.00MB / 10.00MB 2.4s
=> => sha256:4e42533e7311ad0a85fe19e9bc5fe3138f608853eeaa 51.84MB / 51.84MB 8.9s
=> => sha256:e56e70ed0dfb39711bca67c13b10e546d3d79ce4a 192.49MB / 192.49MB 16.4s
=> => sha256:43ccf1ebdd00dc1f1037d0133dfa1fd3fc1bab3eea4957 4.20kB / 4.20kB 8.4s
=> => extracting sha256:ea267e4631a981caf2841a7e9a1faf29cef9d020c4378fc6484 5.0s
=> => sha256:6804f7f2f678435f42a46264fb87bd9e0c519c7bac0 34.08MB / 34.08MB 11.7s
=> => sha256:d1a94681f5387b8a5cbf41dc2fb814c9ed1a6a72bb50bd 2.29MB / 2.29MB 9.5s
=> => sha256:1715a18f9097ff1184d586e695b776f52314dc65329e87ea87 450B / 450B 9.6s
=> => extracting sha256:8a014c92148934973210d840dc7cfed53e0afba38d839afaa48 0.8s
=> => extracting sha256:293ff1be7001d642a624409e2d5f90e7708ef7e6f1a75f4eb73 0.6s
=> => extracting sha256:4e42533e7311ad0a85fe19e9bc5fe3138f608853eeaaea70ee0 3.5s
=> => extracting sha256:e56e70ed0dfb39711bca67c13b10e546d3d79ce4a6b0fa528c 11.6s
=> => extracting sha256:43ccf1ebdd00dc1f1037d0133dfa1fd3fc1bab3eea495718828 0.0s
=> => extracting sha256:6804f7f2f678435f42a46264fb87bd9e0c519c7bac07fb1c7a2 1.7s
=> => extracting sha256:d1a94681f5387b8a5cbf41dc2fb814c9ed1a6a72bb50bd591fa 0.2s
=> => extracting sha256:1715a18f9097ff1184d586e695b776f52314dc65329e87ea873 0.0s
=> [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a 0.0s
=> CACHED [base 2/2] RUN pip install -U pip 0.0s
=> CACHED [builder 1/14] RUN : && apt-get update && apt-get instal 0.0s
=> CACHED [builder 2/14] RUN : && apt-get update && apt-get instal 0.0s
=> CACHED [builder 3/14] RUN curl https://packages.microsoft.com/keys/micr 0.0s
=> CACHED [builder 4/14] RUN curl https://packages.microsoft.com/config/de 0.0s
=> CACHED [builder 5/14] RUN : && apt-get update && apt-get instal 0.0s
=> CACHED [builder 6/14] COPY dev-requirements.txt dev-requirements.txt 0.0s
=> CACHED [builder 7/14] RUN pip install -r dev-requirements.txt 0.0s
=> CACHED [builder 8/14] COPY requirements.txt requirements.txt 0.0s
=> CACHED [builder 9/14] RUN pip install -r requirements.txt 0.0s
=> CACHED [builder 10/14] COPY optional-requirements.txt optional-requireme 0.0s
=> CACHED [builder 11/14] RUN pip install -r optional-requirements.txt 0.0s
=> [builder 12/14] COPY . /fides 14.4s
=> [builder 13/14] WORKDIR /fides 0.1s
=> [builder 14/14] RUN mkdir -p src/fidesapi/build/static 0.8s
=> [prod 1/3] RUN python setup.py sdist 1.7s
=> [prod 2/3] RUN pip install dist/fidesctl-*.tar.gz 13.1s
=> [frontend 2/5] WORKDIR /fides/clients/admin-ui 0.3s
=> [frontend 3/5] COPY clients/admin-ui/ . 6.3s
=> [frontend 4/5] RUN npm install 2.5s
=> [frontend 5/5] RUN npm run export 58.6s
=> [prod 3/3] COPY --from=frontend /fides/clients/admin-ui/out/ /fides/src/ 0.1s
=> exporting to image 14.7s
=> => exporting layers 14.6s
=> => writing image sha256:b862cea1d9110e72a07c2392a6b6ba986ed4a6c79d4e1f00 0.0s
=> => naming to docker.io/ethyca/fidesctl:local
My results on Mac with the same commands. Interesting yours still mentions cache.
nox > Running session build(test)
nox > Creating virtual environment (virtualenv) using python in .nox/build-test
nox > docker build --target=prod --tag ethyca/fidesctl:local .
[+] Building 593.9s (30/30) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.74kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 423B 0.0s
=> [internal] load metadata for docker.io/library/node:16 1.0s
=> [internal] load metadata for docker.io/library/python:3.8-slim-buster 1.0s
=> [internal] load build context 23.0s
=> => transferring context: 494.75MB 22.9s
=> [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de60 7.0s
=> => resolve docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de607b8b8 0.0s
=> => sha256:6258dcdb5fea7b710bfcfc3c889e022e4c6e9dd0ea962cfa73fbc130eff2 1.37kB / 1.37kB 0.0s
=> => sha256:8efad32061c2124947b46db2be6969b6698a00f374fbd57048fc2f701ac6 7.53kB / 7.53kB 0.0s
=> => sha256:65db770a0126c5f032de607b8b8ae7207ec2adc17420e54af4d129b1d3b0 1.86kB / 1.86kB 0.0s
=> => sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f47 11.29MB / 11.29MB 4.8s
=> => sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08 233B / 233B 4.9s
=> => extracting sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f4797e53d 0.8s
=> => sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d 3.17MB / 3.17MB 5.3s
=> => extracting sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08 0.0s
=> => extracting sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d467e 0.9s
=> [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37 25.6s
=> => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf 0.0s
=> => sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1f 49.23MB / 49.23MB 2.7s
=> => sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27 7.72MB / 7.72MB 1.1s
=> => sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf911888ff13c7622346 1.21kB / 1.21kB 0.0s
=> => sha256:407013b47e765f5e900729141d6668a3588b484af4082b232e586041e5bb 2.21kB / 2.21kB 0.0s
=> => sha256:90930c7a6dd7f8531d230a2206f35582a1ac1bfb480063ca55dc4e4899dc 7.75kB / 7.75kB 0.0s
=> => sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b6 9.77MB / 9.77MB 0.7s
=> => sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6 52.17MB / 52.17MB 3.5s
=> => sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5 184.07MB / 184.07MB 8.8s
=> => extracting sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1fd5a234 3.7s
=> => sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42 4.09kB / 4.09kB 2.8s
=> => sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05b 34.22MB / 34.22MB 5.2s
=> => sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80 2.29MB / 2.29MB 3.7s
=> => sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482 451B / 451B 3.8s
=> => extracting sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27a5f1 0.5s
=> => extracting sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b65f3a 0.6s
=> => extracting sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6f415d6 4.7s
=> => extracting sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5f3f0afe7 11.2s
=> => extracting sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42a86e 0.0s
=> => extracting sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05ba8417a 0.9s
=> => extracting sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80185b 0.1s
=> => extracting sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482 0.0s
=> [base 2/2] RUN pip install -U pip 26.8s
=> [frontend 2/5] WORKDIR /fides/clients/admin-ui 0.6s
=> [frontend 3/5] COPY clients/admin-ui/ . 3.6s
=> [frontend 4/5] RUN npm install 22.8s
=> [builder 1/14] RUN : && apt-get update && apt-get install -y --no-instal 89.8s
=> [frontend 5/5] RUN npm run export 49.2s
=> [builder 2/14] RUN : && apt-get update && apt-get install -y --no-instal 16.5s
=> [builder 3/14] RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key a 6.4s
=> [builder 4/14] RUN curl https://packages.microsoft.com/config/debian/10/prod.list | t 0.6s
=> [builder 5/14] RUN : && apt-get update && apt-get install -y --no-instal 20.8s
=> [builder 6/14] COPY dev-requirements.txt dev-requirements.txt 0.0s
=> [builder 7/14] RUN pip install -r dev-requirements.txt 76.3s
=> [builder 8/14] COPY requirements.txt requirements.txt 0.0s
=> [builder 9/14] RUN pip install -r requirements.txt 212.7s
=> [builder 10/14] COPY optional-requirements.txt optional-requirements.txt 0.0s
=> [builder 11/14] RUN pip install -r optional-requirements.txt 81.6s
=> [builder 12/14] COPY . /fides 5.5s
=> [builder 13/14] WORKDIR /fides 0.0s
=> [builder 14/14] RUN mkdir -p src/fidesapi/build/static 0.3s
=> [prod 1/3] RUN python setup.py sdist 3.1s
=> [prod 2/3] RUN pip install dist/fidesctl-*.tar.gz 37.9s
=> [prod 3/3] COPY --from=frontend /fides/clients/admin-ui/out/ /fides/src/fidesapi/build 0.1s
=> exporting to image 7.2s
=> => exporting layers 7.2s
=> => writing image sha256:2581a16d6683a60685240853568b0f8f1ce58b2829dc6f2b98a78dfef5b1ca 0.0s
=> => naming to docker.io/ethyca/fidesctl:local 0.0s
yeah that seems kinda weird...
oh well, sounds like this PR will just be some tinkering around stream-lining whatever parts of the build that we can, specifically for dev related commands
Here's mine after docker system prune --all -f then nox -s dev
➜ fides git:(aking-829-db-dataset) nox -s dev
nox > Running session dev
nox > Creating virtual environment (virtualenv) using python3.9 in .nox/dev
nox > docker build --target=dev --tag ethyca/fidesctl:local .
[+] Building 637.0s (22/22) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 2.74kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 423B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8-slim-buster 1.1s
=> [internal] load build context 32.1s
=> => transferring context: 769.18MB 31.8s
=> [base 1/2] FROM docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de607b8b8 7.0s
=> => resolve docker.io/library/python:3.8-slim-buster@sha256:65db770a0126c5f032de607b8b8ae720 0.0s
=> => sha256:b0bff26542d3686b2ab27a0717c161d81eaf10369688d0e880f1c646c3dd7163 2.78MB / 2.78MB 0.9s
=> => sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f4797e53 11.29MB / 11.29MB 3.7s
=> => sha256:65db770a0126c5f032de607b8b8ae7207ec2adc17420e54af4d129b1d3b0745a 1.86kB / 1.86kB 0.0s
=> => sha256:6258dcdb5fea7b710bfcfc3c889e022e4c6e9dd0ea962cfa73fbc130eff2c174 1.37kB / 1.37kB 0.0s
=> => sha256:8efad32061c2124947b46db2be6969b6698a00f374fbd57048fc2f701ac6cbfc 7.53kB / 7.53kB 0.0s
=> => sha256:824b15f81d6568adc139263c39805e52d9880758b907f40144bbb1938ca5932 27.14MB / 27.14MB 2.8s
=> => sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08 233B / 233B 1.0s
=> => sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d467e 3.17MB / 3.17MB 1.7s
=> => extracting sha256:824b15f81d6568adc139263c39805e52d9880758b907f40144bbb1938ca59323 2.2s
=> => extracting sha256:b0bff26542d3686b2ab27a0717c161d81eaf10369688d0e880f1c646c3dd7163 0.2s
=> => extracting sha256:d59e69a18eecf7f63d9379a602a0f904b298202d4fb9afdd34270c2f4797e53d 0.7s
=> => extracting sha256:1525f74216cf07879be0c575e5ed58c1ae4058f4c5d7ab3cdca10d657f49ab08 0.0s
=> => extracting sha256:301f2616d6071fe9be7e9dbe77c4829ece48519e10b8d27de853160bc18d467e 0.5s
=> [base 2/2] RUN pip install -U pip 34.0s
=> [builder 1/14] RUN : && apt-get update && apt-get install -y --no-install-rec 85.3s
=> [builder 2/14] RUN : && apt-get update && apt-get install -y --no-install-rec 15.8s
=> [builder 3/14] RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - 6.6s
=> [builder 4/14] RUN curl https://packages.microsoft.com/config/debian/10/prod.list | tee /e 0.6s
=> [builder 5/14] RUN : && apt-get update && apt-get install -y --no-install-rec 28.1s
=> [builder 6/14] COPY dev-requirements.txt dev-requirements.txt 0.0s
=> [builder 7/14] RUN pip install -r dev-requirements.txt 83.4s
=> [builder 8/14] COPY requirements.txt requirements.txt 0.0s
=> [builder 9/14] RUN pip install -r requirements.txt 221.3s
=> [builder 10/14] COPY optional-requirements.txt optional-requirements.txt 0.0s
=> [builder 11/14] RUN pip install -r optional-requirements.txt 90.1s
=> [builder 12/14] COPY . /fides 7.2s
=> [builder 13/14] WORKDIR /fides 0.0s
=> [builder 14/14] RUN mkdir -p src/fidesapi/build/static 0.3s
=> [dev 1/1] RUN pip install -e ".[all,mssql]" 47.4s
=> exporting to image 8.4s
=> => exporting layers 8.4s
=> => writing image sha256:0d214129491d1273ff68d0a4f9843efaa733024cd18f10afa11f8f528a6e05e6 0.0s
=> => naming to docker.io/ethyca/fidesctl:local 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
nox > docker build --target=frontend --tag ethyca/fidesctl:local-ui .
[+] Building 99.5s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/library/node:16 1.0s
=> [frontend 1/5] FROM docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aec 39.0s
=> => resolve docker.io/library/node:16@sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf91188 0.0s
=> => sha256:8951351b7c6a2f8ff9ec25eccc087d37a8aeccf9bf911888ff13c76223467466 1.21kB / 1.21kB 0.0s
=> => sha256:90930c7a6dd7f8531d230a2206f35582a1ac1bfb480063ca55dc4e4899dc7674 7.75kB / 7.75kB 0.0s
=> => sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b65f3a 9.77MB / 9.77MB 2.0s
=> => sha256:407013b47e765f5e900729141d6668a3588b484af4082b232e586041e5bbd036 2.21kB / 2.21kB 0.0s
=> => sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1fd5a2 49.23MB / 49.23MB 12.7s
=> => sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27a5f1 7.72MB / 7.72MB 2.9s
=> => sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6f415d 52.17MB / 52.17MB 7.8s
=> => sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5f3f0 184.07MB / 184.07MB 24.1s
=> => sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42a86e 4.09kB / 4.09kB 7.9s
=> => sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05ba841 34.22MB / 34.22MB 11.9s
=> => sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80185b 2.29MB / 2.29MB 12.3s
=> => sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482 451B / 451B 12.4s
=> => extracting sha256:8d6f1451981514e25c21542f5c7ee9bb90052b8856b484ce47294cbf1fd5a234 4.4s
=> => extracting sha256:8ccc9fb4baf6e7f2e6ee18ed689c8ee1171c6751c8bbd317d580a193da27a5f1 1.2s
=> => extracting sha256:620d55693ed5943ab298346d9ccafefdd6d6f33994e6820a857737df89b65f3a 0.6s
=> => extracting sha256:82dcb0fa2b6020cd95c3972ec0fe03da38862f57574fe03a49360713d6f415d6 4.9s
=> => extracting sha256:d75d85ab89337b32fb8b4e0454a157fa71a765a1daf512255323edd5f3f0afe7 11.8s
=> => extracting sha256:58af0698dc7f636576a444c050fdf327087c17ca5f52bf8691359e902a42a86e 0.1s
=> => extracting sha256:915a9c8840243d713b6c11835a772389bb3277c64fba1bd36cb619b05ba8417a 1.3s
=> => extracting sha256:798bae4139c68cdd5e53b6ff9c601d8cbd88c94cfe9df501df3e7beadd80185b 0.1s
=> => extracting sha256:c76fba79a64114d60394b5c76efbb7460b3fad11363da2089081a49ba6ec1482 0.0s
=> [internal] load build context 35.9s
=> => transferring context: 724.99MB 35.6s
=> [frontend 2/5] WORKDIR /fides/clients/admin-ui 0.7s
=> [frontend 3/5] COPY clients/admin-ui/ . 6.7s
=> [frontend 4/5] RUN npm install 1.8s
=> [frontend 5/5] RUN npm run export 44.9s
=> exporting to image 5.4s
=> => exporting layers 5.4s
=> => writing image sha256:218d35b0efe575e224c435780fe3c34d91828b8f0ce64ac338d3fdafe6c58a34 0.0s
=> => naming to docker.io/ethyca/fidesctl:local-ui 0.0s
I never noticed this before but since we started looking at this I see you can tell if the layer used cache if it starts with CACHED. So in both of your runs the npm install was done without cache, but it is always way faster than both Thomas' and mine. No idea why that would be.
1.8s doesn't seem like enough time to actually install all of those modules? there's some kind of black magic cache going on or something 😂 i don't know how else that can be possible
@allisonking I'm going to defer on this one until you finish up the frontend packaging PR since I don't want to potentially create conflicts