buildkit Cannot give a local image to FROM when using docker-container

I have a Dockerfile which references another image, which is stored locally (overrideable with a build arg):

ARG PARENT=ds_test_base:test
FROM ${PARENT}

The image is there:

$ docker image ls ds_test_base:test
REPOSITORY     TAG           IMAGE ID       CREATED       SIZE
ds_test_base   test   eb69be11f1e5   3 hours ago   10.4GB

When I try to build this Dockerfile with --builder xyz, which is backed by a docker-container driver, I get this:

#3 [internal] load metadata for docker.io/library/ds_test_base:test
#3 ERROR: pull access denied, repository does not exist or may require authorization: authorization status: 401: authorization failed
------
 > [internal] load metadata for docker.io/library/ds_test_base:test:
------
Dockerfile:2
--------------------
   1 |     ARG PARENT=ds_test_base:test
   2 | >>> FROM ${PARENT}
   3 |     
   4 |     MAINTAINER "Openmail"
--------------------
error: failed to solve: ds_test_base:test: pull access denied, repository does not exist or may require authorization: authorization status: 401: authorization failed

If I try to build it with exactly the same parameters, but omitting --builder xyz, it builds just fine.

$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:33 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 27
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-81-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.58GiB
 Name: ip-10-150-29-76
 ID: ZJFE:DJF7:DUGP:WOGK:A66E:IVQS:X6HB:CKP4:SBRG:VROU:ZIBO:GWXF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

$ docker image ls moby/buildkit
REPOSITORY      TAG               IMAGE ID       CREATED       SIZE
moby/buildkit   buildx-stable-1   2b537a02e2d9   6 weeks ago   144MB

Sep 03 '21 12:09 bra-fsn

If I try to build it with exactly the same parameters, but omitting --builder xyz, it builds just fine.

I think this might be expected; if you're using a containerised builder, that builder uses its own cache for images, which is not shared with the dockerd image cache; when building, you'll probably see a warning about this;

WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

If you use the --load option, the image will be loaded into the docker daemon's local image cache (but this cache only supports a single architecture, so no multi-arch images can be stored there); I don't think it's possible currently to read back those images (for use as FROM, but I may be mistaken (@crazy-max ?)).

I wonder if it's possible though to make the container builder itself keep the image (for cases where the second build is running on the same builder: @crazy-max are the volume changes you made related to that?).

Otherwise, it's probably best to push your images to a registry (which would also address the multi-arch use-case, if you happen to be building a multi-arch image).

But, yes, I think the UX / experience could be improved somewhat here to make this easier to use (or more clear "how" to do this)

Sep 29 '21 11:09 thaJeztah

I think the solution we're going with is to do this outside of buildkit like https://github.com/docker/buildx/issues/447

Oct 27 '21 00:10 tonistiigi

Is there any workaround for this? Attempting to build an image on an instance outside of my network (therefore it cannot access my custom registry) and I need to use a base image from that registry. I have copied the tar file for that image, but I cannot find a way to load that image into buildkit.

Mar 14 '22 16:03 AlSummer

https://github.com/docker/buildx/blob/v0.8.0/docs/reference/buildx_build.md#build-context https://github.com/docker/buildx/blob/v0.8.0/docs/reference/buildx_bake.md#defining-additional-build-contexts-and-linking-targets

Mar 14 '22 17:03 tonistiigi

FWIW, this is exactly what we raised in #2210 . builder-container has a regression in functionality compared to docker builder, at least as far as cache is concerned.

I had looked into the containerized builder works, and was surprised that it actually was missing only a few things to make it work. If I recall correctly:

when building store manifests and indexes in blobs (remove the logic to store only binary blobs)
when building store image ref to root blob (manifest or index), either in something like the containerd db or simply in index.json (like we do in linuxkit's cache)
change the FROM lookup to look in local cache before going to remote registry

You could do other things, like load and save equivalents, which would be helpful, but not, I believe, critically necessary.

I still know of a few people willing to help with it a bit, but it has been 9 months since we opened the issue.

Mar 15 '22 20:03 deitch

@tonistiigi how does this help it?

Mar 15 '22 20:03 deitch

As far as I can tell - please do correct me - the buildx bake context stuff allows you to, essentially, "alias" a FROM (or --from=) in a Dockerfile or other builder to one of a local directory, an image in a registry, or possibly the results of a previous stage (based on this).

It looks pretty hesitant to use that target reference:

Please note that in most cases you should just use a single multi-stage Dockerfile with multiple targets for similar behavior. This case is recommended when you have multiple Dockerfiles that can't be easily merged into one.

I don't understand how this manages to store the output of one to the other, especially with containerized builder. Is it just because it is building both at once, so the builder knows about all of the outputs and can "hold onto" them?

More important UX question: Does this mean that, in order to do the simple "use a a local cached image" that we have been used to for docker since day 1, where I do docker build -t somelocalimage:foo -f Dockerfile1 . and then consume it in another with FROM somelocalimage:foo, I now need to:

have a bake.hcl in addition to my Dockerfiles
learn the bake syntax
actually have both of them there
build them both at the same time so the bake file can reference them
call a different command

all to replicate a "docker-simple" (TM) functionality from the existing flow?

Mar 15 '22 21:03 deitch

@deitch As you have been explained multiple times, nothing changes with the existing build behavior. If you want your builds to access Docker images you need to use the Docker driver. The same would be to set the requirement to build in k8s pods, but not do it using k8s driver.

More important UX question: Does this mean that,

Yes, when using Docker images as intermediate storage you needed to run multiple build commands with many flags, in a specific order, and then invent ways how the images created as a side effect of your builds get cleaned up so you don't run out of disk space. If you use named context you only have one command with no extra arguments that takes care of all of this. As the docs explain in most cases this isn't even needed and you should just use multi-stage builds. This old pattern was invented before multi-stage builds support was added and is only needed for cases where you have a requirement to build multiple projects that depend on each other and can't be merged(and you can't access any registry).

Mar 16 '22 01:03 tonistiigi

If you want your builds to access Docker images you need to use the Docker driver

Understood. But isn't the point of buildkit and the various drivers so that we can open new capabilities? buildkit does this phenomenally well, but some of those capabilities are not yet retrofitted into the docker driver. So you have to choose between:

images in the docker cache to use as FROM
new buildkit capabilities (like multi-arch and OCI format storage and remote builders and etc. etc.)

I think we are trying to bridge this gap: if docker driver could do everything buildkit can (which it eventually will, I understand), and buildkit containerized could do everything docker could (again, within reason), then the choice between features Docker and features buildkit (and if you need both, you are out of luck) wouldn't be a problem.

As the docs explain in most cases this isn't even needed and you should just use multi-stage builds. This old pattern was invented before multi-stage builds support was added and is only needed for cases where you have a requirement to build multiple projects that depend on each other and can't be merged(and you can't access any registry).

Sure, and I built tons of images that way in the early days. And when I could move to multistage after it existed, I did.

But is it a fair assumption that every (or even a majority) of builds are simple linear "start at point A, go through some intermediates, get to point Z" and thus multistage candidates?

It is very common to have a base image (A), build some "golden base" (B), then some intermediates (C, D, E, F), then some finals (Q, R, S, T, U, V, W, X, Y, Z). This "tree" isn't a single build that can go in a multistage Dockerfile, and the intermediates aren't stored in a remote registry: they might be private, they might be part of a testing process and aren't valid to be stored until the finals (Q, R, S, T, U, V, W, X, Y, Z) are generated and fully tested (if at all). The local cache holds all of those interim images, and the later steps absolutely are not a single multistage build, simply cannot be. They cannot even be run at the same time (like bake with a bake.hcl).

Build B from A.
Build C, D, E, F from B
Build Q, R, S, T, U, V, W, X, Y, Z from C, D, E, F

These are separate build processes, might be running at separate times. This isn't a single multi-stage Dockerfile (imagine what a nightmare that single file would be), and isn't run as a single step or stage in a CI process.

Let's be practical: someone has been doing this for years with docker build. It works great (it really does, hats off). They see buildkit, love what it can do, need those features that do not yet exist in the docker driver, so they try to go containerized.

And their entire build process breaks. Because step 1 stored output B, and step 2 cannot find B for C,D,E,F (repeat for step 3).

If you use named context you only have one command with no extra arguments that takes care of all of this.

That's just it. The above very common use case describes how "one command" isn't practical here. It has to be multiple commands. Yet docker build combined with FROM and local caches made those multiple commands dead simple to grasp and use. That was a key part of docker's adoption.

The part I am having a really hard time figuring out is, why the strong objection to proper caching inside buildkit container? You already store blobs, you already have the OCI layout, the number of additional steps to get to parity with default docker behaviour is so small (and people have offered to help build it). Why the resistance?

Mar 16 '22 08:03 deitch

I have the same sort of non-linear use case as @deitch describes and have found myself blocked by this, as an additional anecdote.

Mar 19 '22 22:03 msg555

The part I am having a really hard time figuring out is, why the strong objection to proper caching inside buildkit container? You already store blobs, you already have the OCI layout, the number of additional steps to get to parity with default docker behaviour is so small (and people have offered to help build it). Why the resistance?

BuildKit is not supposed to be a reimplementation of all the Docker features. Obviously, Docker has useful features, like an image store, and if you like it, you should continue to use it. BuildKit does not supersede the whole Docker engine but is scoped for a specific job: building from well-defined (immutable) sources to the expected build result.

If your problem statement is that you want to build some images, then take a break and later use the images you built in another build, then you need a place to keep them between your builds. There are plenty of options: docker image store, containerd image store, or any registry implementation. With named context we have made the registry case even simpler as you don't need to do any changes in Dockerfile (or use build args) if you want to switch between release images and local dev/staging registry image.

Defining an image store is fundamentally conflicting with builder design and BuildKit features. For example, BuildKit has an automatic garbage collector or smart pruning filters for managing storage. Users don't need to think about where their build are running, and everything can be moved to different infrastructure. There is support for multi-node build that splits build between multiple machines. All this would be impossible if there was some kind of name-based storage.

The user does not need to think that there is a special machine somewhere that contains their images, how they are named, how much storage they take, what happens if another version with the same name is built, what happens if one of them is deleted, what do you do after you run image ls and get 100 unnamed images as a result. If they want to think in these terms, then they should use a tool that is built for solving the image storage problem, not a builder.

Mar 24 '22 03:03 tonistiigi

This is a pretty good explanation @tonistiigi ; thanks for that. I will try to respond in detail.

BuildKit does not supersede the whole Docker engine but is scoped for a specific job: building from well-defined (immutable) sources to the expected build result.

Definitely. "Do one job and do it well". The challenge is that docker may have some features (local reusable image store), buildkit may others (cross-arch builds, remote contexts, etc.), and if you need to use both, you are stuck.

If your problem statement is that you want to build some images, then take a break and later use the images you built in another build, then you need a place to keep them between your builds

Agreed 100% there. You need somewhere to store them.

There are plenty of options: docker image store, containerd image store, or any registry implementation.

Well, not exactly. docker image store doesn't support anything other than local architecture (and expands the images), and both docker store and containerd store are not accessible to container-driver buildkit.

I think you have narrowed the problem down well; I am going to modify it somewhat to make it more explicit: if you want to build images using buildkit because of its features that docker does not have, and you want to "take a break" between builds, then you need somewhere to store those images that buildkit supports sourcing from. That list is pretty narrow. Actually, it is pretty much: network registry. And if your image does not or cannot go to the registry (e.g. during development lifecycle, when you are not yet ready to push, or an interim image that you never will push), what solution do you habve?

With named context we have made the registry case even simpler as you don't need to do any changes in Dockerfile (or use build args) if you want to switch between release images and local dev/staging registry image.

Yes, I see that. Takes a while to get it, but basically named contexts are "image aliases". Come to think of it, this would have been nice in docker from day 1.

Defining an image store is fundamentally conflicting with builder design and BuildKit features. For example, BuildKit has an automatic garbage collector or smart pruning filters for managing storage. Users don't need to think about where their build are running, and everything can be moved to different infrastructure. There is support for multi-node build that splits build between multiple machines. All this would be impossible if there was some kind of name-based storage.

This is a pretty solid argument. it doesn't eliminate the need to solve the problem, but it makes a solid case for why a local cache may not be the approach.

You basically are saying:

Docker did build and image cache and pulling and running and and and ...
buildkit is focused on building, and doing that well, so let's not mix it up with image storage (even if it does have some blob caching).
If we need to solve image store other than in networked registry, let's solve it.

I can get behind it, but what is that solution?

networked registry doesn't quite work for a bunch of reasons: material user experience change ("anti-docker behaviour", not to mention defeating existing user expectations); use cases do not or cannot push to networked registries; name in the Dockerfile is human and understandable to the engineer, not some odd localhost:8888/something name.
docker image cache is not readable by container-driver buildkit
containerd image cache is not readable by container-driver buildkit

So what is the right answer? Reading what you wrote, you almost want to be able to have a pluggable implementation of image aliasing, so I can use the following Dockerfile:

FROM myimage:1.2.3

and then run docker buildx build --alias myimage:1.2.3=driver://image where driver is different drivers: knows how to talk to networked registry (default), or networked registry while changing the name, or a local OCI file layout, or docker, etc. Something that lets it keep builds inside buildkit and image storage outside buildkit, without being so tightly tied to the same name in the networked registry.

Can named context actually do that? I think maybe some of it?

Mar 24 '22 21:03 deitch

and then run docker buildx build --alias myimage:1.2.3=driver://image where driver is different drivers:

You don't need a new flag. This is already supported. --build-context myimage:1.2.3=docker-image://image. The source can be a local directory, git repository, docker image, URL to tarball, or another build definition in bake for chaining. Adding another source that is path to OCI file layout does not conflict with the buildkit design.

Mar 28 '22 23:03 tonistiigi

You don't need a new flag. This is already supported.

Yeah, I wasn't suggesting a new flag. Just using language that was instantly understandable in our conceptual discussion.

The source can be a local directory, git repository, docker image, URL to tarball, or another build definition in bake for chaining

Yeah, I got that from your descriptions. Makes sense.

Is there anything like the current docker behaviour of, "look in this cache, and if not there, go pull it"?

Adding another source that is path to OCI file layout does not conflict with the buildkit design.

Meaning you would be open to it? I might be willing to submit a PR for that. Point me at the right place in buildkit where to hook it in.

The hard part, I think, still is the "generic idea". When I do a classic docker build, it knows how to parse each FROM line in the dockerfile, and then look in the local cache, going to registry if it fails. I don't need to tell it, "for image X, look here; for image Y, look there; etc.". I am not sure we can get that behaviour out of this, which means that the build command needs to know each and every image in the dockerfile and its source, as opposed to a generic docker-style. The current docker cache almost is a "read-through cache", where it only goes remote if you get a cache miss.

I am up for getting us to have --build-context myimage:1.2.3=oci-layout:///path/to/oci (we would need to figure out if we should point to the root manifest/index or the generic base of the OCI layout), but I would really like to have a --build-context *=oci-layout:///path/to/oci, where it just becomes another read-through cache: try in the OCI layout, then go network.

--build-context as it stands buys us a lot of flexibility for individual images, but now we pass the responsibility for knowing each image to the command-line. This is more complicated than the clean separation of docker build, where the CLI caller does not need to know each image in the dockerfile.

Do you see where I am driving?

Mar 29 '22 07:03 deitch

Sorry, but why not being able to specify a --build-context-like syntax inside of the Dockerfile's FROM itself? Something like this:

FROM ../my-base-image

[...]

Mar 29 '22 19:03 felipecrs

Meaning you would be open to it?

Yes. Take a look at how the type=local cache import is implemented. It also loads OCI layout from client disk so the transfer parts should be reusable. First, it needs to be implemented in LLB, probably as extra property for the llb.Image. Then it can be plugging into Dockerfile and --build-context flag.

--build-context *=

That does not work well. First of all names are needed for the build-context mechanism exposed to frontends. Secondly, OCI has not even decided on a standard for naming multiple images within the OCI layout.

Sorry, but why not being able to specify a --build-context-like syntax inside of the Dockerfile's FROM itself?

I'm not sure what the example you provided is supposed to do but Dockerfile is a secure and portable build mechanism. It does not have permission to randomly start reading files from user's/host's disk.

Mar 29 '22 23:03 tonistiigi

I'm not sure what the example you provided is supposed to do but Dockerfile is a secure and portable build mechanism. It does not have permission to randomly start reading files from user's/host's disk.

It's supposed to do the same as:

FROM my-base-image

[...]

docker build --build-context my-base-image=../my-base-image .

And about the Dockerfile being secure and portable, this means that the buildx bake is the non-portable and non-secure counterpart to address the Dockerfile limitations, is that it?

Mar 29 '22 23:03 felipecrs

@felipecrs Bake has a different security model indeed. Atm. for local bake files it is allowed for them to point to any local path. For remote files, we have disabled parent access. In future releases, the plan is to move to https://github.com/docker/buildx/issues/179 to control what is allowed and what is not.

Mar 30 '22 00:03 tonistiigi

I'm not sure what the example you provided is supposed to do but Dockerfile is a secure and portable build mechanism. It does not have permission to randomly start reading files from user's/host's disk.

Yeah, I get what Tonis is saying here. the FROM line is always an image ref. The ability to turn that ref into something other than the actual registry is outside of the boundaries of the Dockerfile itself, i.e. the build engine that interprets it, otherwise Dockerfile wouldn't be portable and self-contained.

Mar 30 '22 06:03 deitch

Secondly, OCI has not even decided on a standard for naming multiple images within the OCI layout.

Quite. I have generally used index.json (which is part of the OCI layout), but even there is disagreement on how to map the image name to the root blob. containerd uses their db, etc.

That doesn't change the question of specific name vs generic. It removes it entirely from buildkit's scope, actually. It wouldn't know or care; it would just pass it on to the driver.

That does not work well. First of all names are needed for the build-context mechanism exposed to frontends.

That isn't a fundamental product reason as to why it wouldn't be desired; it is a technical engineering reason why it wouldn't work with the current design.

I will try and put it in other terms.

--build-context is an aliasing mechanism. It lets me tell the builder, "alias image X with source Y", where "source Y" might be another image, or source on disk, or git repo, etc.
docker build (legacy, or docker driver buildkit) is a read-through cache mechanism. It lets me tell the builder, "for all images, use the local cache, and only go remote if it doesn't exist." It actually isn't even a cache, it is a proxy. It happens to be a read-through cache, but it need not be. The builder doesn't care; it just sends all requests for images to a proxy and says, "give me X", "give me Y", etc.

While --build-context keeps the resolution of images inside buildkit, just using different aliasing drivers, what I think I am getting at here is blanket proxy.

I am aware that we could (almost) get to that by running a customized local registry as read-through cache, with perhaps some special logic that handles whatever aliasing is desired.

Of course, all of the above is much more complex than just, "I build something using buildkit, then build something else, and buildkit caches the results of the first".

Mar 30 '22 07:03 deitch

While explaining this issue to some people, I realized the most basic use case why it matters, and why multi-staged docker files don't solve it, why bake and build contexts only partially do it.

Let's say that I am developing an image that normally is public, call it foo/myimage. Part of my build and test process is that there are downstream images, completely independent, that depend on foo/myimage, call them bar/image and bar/other.

A normal, sane, build process has me do the following:

Do work on the source of foo/myimage
docker build -t foo/myimage:testout .
Do not push foo/myimage:testout, since I have absolutely no idea if that is the finally version until I work up the downstream dependencies
Modify the Dockerfiles for the source of bar/image and bar/other
docker build -t bar/image:something and docker build -t bar/other:else
If all works, docker push foo/myimage:testout

The above is a great, simple chaining process, very Docker-ish, and 100% depends on foo/myimage:testout being available in some local read-through cache.

Even with build-contexts, how would I do my normal process? What can I give to docker build -t bar/image:something that will point it at local output? buildkit container-drive only understands a networked registry derived from the FROM image ref, and build contexts require that I explicitly give it the name (meaning yet another part of my build process I need to edit and manage, in addition to the Dockerfile), but it also has no "source" that I can point at the output of the first.

In theory, I could do a docker save on my first image, and then untar it, or maybe build the output with -o to a local file, but my process got much more complex, and if my testing succeeds, I need to rebuild it again in order to push it. Caching will make it more efficient, but yet another step to run.

Am I doing a decent enough job explaining how this breaks normal processes?

Apr 01 '22 07:04 deitch

Oh and by the way, I totally get your argument that, "this is an optimized builder, not image manager/cacher; docker does both, this is just half of it." What I am trying to do is understand, "ok, what is the other half that complements it so we can get buildkit-build awesomeness with image caching"?

Apr 01 '22 08:04 deitch

Thinking a bit more about (partial) solutions:

buildkit caches many blobs - but not image refs, for the reasons you described @tonistiigi - but only layers, not json blobs (config, manifest, index). What if it also cached those, which is trivial? The same way that buildkit currently, when it hits "I need blob X", it looks in local cache and, if it cannot find it, goes to the registry for the image currently being looked up, it could do the same for manifests and such.

Then the only thing left to do would be resolve image-> hash

That would allow something like:

build --build-context myimage:1.2.3=docker-image://myimage:1.2.3@sha256:abcdef11222

or even shorthand

build --build-context myimage:1.2.3=docker-image://@sha256:abcdef11222

Buildkit could:

Hit myimage:1.2.3
See that it has a context
See that the context has a sha256 ref
Try to use that (possibly cached) blob as the root manifest/index
If it finds it, follow from there; if not, go to remote registry... just like every other blob it does today

It isn't perfect, but it gets rid of a lot of the headache without breaking the current buildkit model/mindset:

we just cache more very lightweight blobs
the cache clearing that buildkit does is unaffected
the issue of "image to manifest/index" resolution is kept out of buildkit, just like current build-contexts
you can build one, then another, then another, without resorting to multi-stage Dockerfiles, or pushing to a registry when you cannot, or using bake

It still means the command-line needs to know a lot, but it is a big step forward while staying in the same context.

Thoughts @tonistiigi ?

Apr 01 '22 08:04 deitch

It still means the command-line needs to know a lot

I'm not sure if I agree with this part. Because bake would be as simple as:

docker buildx bake image other

To build the two images dependent on myimage, by also building myimage in the process.

Apr 01 '22 12:04 felipecrs

To build the two images dependent on myimage, by also building myimage in the process

Are you suggesting that the dev process becomes one that uses bake? So:

if I am building from an image that is publicly known somewhere, docker buildx build
if I am working locally on an image that will be consumed by another image, docker buildx bake

Still not quite the simplicity of "just have a local read-through cache", but something.

I think the part that throws me is that, I have to use a completely different command-line for the second image, depending on if my first image is locally built or not.

Apr 01 '22 13:04 deitch

I think the part that throws me is that, I have to use a completely different command-line for the second image, depending on if my first image is locally built or not.

I would just rather use bake for both cases, including "building from an image that is publicly knwon".

I have to use a completely different command-line for the second image, depending on if my first image is locally built or not.

Well, that will be defined in your Dockerfile, or at most, in docker-bake.hcl (in case you want to chain two Dockerfiles). So that, the command-line call will be always the simple:

docker buildx build bake my-second-image

Apr 01 '22 15:04 felipecrs

And to support both cases in the same docker-bake.hcl, the command-line call could vary:

# docker-bake.hcl

target "base" {
    dockerfile = "baseapp.Dockerfile"
}

target "app" {
    contexts = {
        baseapp = "target:base"
    }
}

Building app, also building base

docker buildx bake app

Building app, using previously pushed base

docker buildx bake app --set=app.contexts.baseapp=docker-image://my-previously-pushed-base

Apr 01 '22 15:04 felipecrs

@felipecrs I am starting to see where you are going.

I am curious how bake even knows about the first image when it doesn't store things? Ah, I am guessing that it caches the root manifest of the first (but not the image name)? That almost inches towards what I was suggesting above, that we be able to have buildkit always cache json (config, manifests, indexes) in addition to binary blobs, and be able to to --build-context baseapp=@sha256:abcdef123456.

In any case, how do I get from where you are going to an actual flow?

for the majority case, when I am just building downstream apps (2, 3, 4), docker buildx bake app
for the minority case, when I am building the upstream app 1, and also need to build and test downstream 2,3,4 before pushing 1, docker buildx bake app --set=app.contexts.baseapp=<something>

It kind of reverses what you did. Is there a way to use the CLI to set the context with --set to the output of a previous build step, or is that only in the bake file?

For that matter, I could even do something like:

for the majority case, when I am just building downstream apps (2, 3, 4), docker buildx build
for the minority case, when I am building the upstream app 1, and also need to build and test downstream 2,3,4 before pushing 1, docker buildx bake app

It still grates on me a bit that I cannot give me users the seamless experience they were used to. Well, I do wrap the build, so maybe there is room there.

Apr 03 '22 10:04 deitch

One other question: does that caching work with output formats? When I build, I need the image in OCI format, so I run build -o type=oci

Apr 03 '22 10:04 deitch

Oh wait, I am not so sure it will work.

How would bake handle it if they are in 2 completely different directories?

|- A
    |- Dockerfile
    |- srcs/
|- B
    |- Dockerfile
    |- srcs/

Apr 03 '22 10:04 deitch