BuildKit in local cluster with Minikube and vcluster

[johnnypea]

I run vcluster in Minikube localy for testing purposes and I can't get local image building working.

Please, could you check my config? I'm not sure about "buildKit" and "docker" parts in images section.

vars:
  - name: REGISTRY
    value: "myapp.local" 

images:
  php:
    image: ${REGISTRY}/php
    dockerfile: api/Dockerfile
    context: api/
    build:
      buildKit: 
        inCluster: {}
        skipPush: true
      docker:
        options:
          target: symfony_php
          
deployments:
- name: app
  helm:
    componentChart: true
    values:
      containers:
      - name: php-container
        image: ${REGISTRY}/php

Tags are the same but it doesn't use local image, why?

[php] Info: Building image 'myapp.local/php:WdljxgD' with engine 'buildkit'

[info]   Execute hook 'wait for selector app.kubernetes.io/component=app' at after:deploy:app
[warn]   Pod intraspace/app-0 has critical status: ErrImagePull. DevSpace will continue waiting, but this operation might timeout
[fatal]  error deploying: in hook 'wait for selector app.kubernetes.io/component=app': timed out waiting for the condition

  - image: myapp.local/php:WdljxgD
    imageID: ""
    lastState: {}
    name: php-container
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'rpc error: code = Unknown desc = Error response from daemon: Get
          "https://myapp.local/v2/": dial tcp: lookup myapp.local: Temporary failure
          in name resolution'
        reason: ErrImagePull

It looks like the images are build with local docker daemon on my host not in the actual cluster.

❯ docker images
REPOSITORY                TAG         IMAGE ID       CREATED             SIZE
myapp.local/php           WdljxgD     656d1af96fb0   14 minutes ago      191MB

This config works with Minikube without BuildKit and vcluster.

Thank you. 🙏

[pratikjagrut]

@johnnypea Thank you for the issue. We'll look into this.

[pratikjagrut]

We made a mistake in doc. I think the correct config should be like below:

 php:
    image: ${REGISTRY}/php
    dockerfile: api/Dockerfile
    context: api/
    build:
      buildKit: 
        inCluster: {}
        skipPush: true
        options:
          target: symfony_php

Could you please try this out?

[johnnypea]

@pratikjagrut thanks, but still the same problem

  - image: myapp.local/php:YPbQNky
    imageID: ""
    lastState: {}
    name: php-container
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'rpc error: code = Unknown desc = Error response from daemon: Get
          "https://myapp.local/v2/": dial tcp: lookup myapp.local: Temporary failure
          in name resolution'
        reason: ErrImagePull

[warn]   Pod intraspace/app-0 has critical status: ErrImagePull. DevSpace will continue waiting, but this operation might timeout
[fatal]  error deploying: in hook 'wait for selector app.kubernetes.io/component=app': timed out waiting for the condition

It looks like it doesn't or can't use local image.

[FabianKramm]

@johnnypea maybe try to use imagePullPolicy: Never as this might help

[johnnypea]

@FabianKramm didn't help :(

  - image: myapp.local/php:fMyzTrj
    imageID: ""
    lastState: {}
    name: php-container
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: Container image "myapp.local/php:fMyzTrj" is not present
          with pull policy of Never
        reason: ErrImageNeverPull

[johnnypea]

@FabianKramm @pratikjagrut images are created for my local Docker not one in Minikube when used with vcluster. It works as expected with "minikube" context". Is there some configuration I should set?

[johnnypea]

And I don't know if this could be related but I noticed this kind of errors in my vcluster console:

I0315 11:58:31.819918   28173 portforward.go:391] error copying from remote stream to local connection: readfrom tcp6 [::1]:11857->[::1]:51561: write tcp6 [::1]:11857->[::1]:51561: write: broken pipe
I0315 11:58:32.677832   28173 portforward.go:391] error copying from remote stream to local connection: readfrom tcp6 [::1]:11857->[::1]:51581: write tcp6 [::1]:11857->[::1]:51581: write: broken pipe
I0315 11:58:34.366057   28173 portforward.go:391] error copying from remote stream to local connection: readfrom tcp6 [::1]:11857->[::1]:51599: write tcp6 [::1]:11857->[::1]:51599: write: broken pipe
I0315 11:58:35.121331   28173 portforward.go:391] error copying from remote stream to local connection: readfrom tcp6 [::1]:11857->[::1]:51617: write tcp6 [::1]:11857->[::1]:51617: write: broken pipe
I0315 11:58:35.173762   28173 portforward.go:391] error copying from remote stream to local connection: readfrom tcp6 [::1]:11857->[::1]:51622: write tcp6 [::1]:11857->[::1]:51622: write: broken pipe

[FabianKramm]

Ah yeah, since minikube has its own docker daemon, devspace pushes the image to docker instead of the minikube daemon, which is why the image cannot be pulled. A workaround currently would be to use minikube as context name or expose the docker daemon before running devspace with:

minikube docker-env
devspace dev

[johnnypea]

@FabianKramm but shouldn't Buildkit config do the trick?

      buildKit: 
        inCluster: {}

[FabianKramm]

@johnnypea not sure but I believe pushing the image still doesn't reach the correct docker daemon

[madhavajay]

I am having a related problem. I want to deploy multiple clusters to do some testing between clusters however when I give my minikube cluster a unique name it does not get picked up by devspace.

$ minikube profile list

|----------------|-----------|---------|---------------|------|---------|---------|-------|
|    Profile     | VM Driver | Runtime |      IP       | Port | Version | Status  | Nodes |
|----------------|-----------|---------|---------------|------|---------|---------|-------|
| test-domain-1  | hyperkit  | docker  | 192.168.64.12 | 8443 | v1.23.3 | Running |     1 |
| test-network-1 | hyperkit  | docker  | 192.168.64.11 | 8443 | v1.23.3 | Running |     1 |
|----------------|-----------|---------|---------------|------|---------|---------|-------|

I tried manually sourcing the docker-env but it doesn't seem to matter.

$ eval $(minikube -p test-network-1 docker-env)
$ devspace deploy -p network -n test-network-1

....
[tailscale] ------
[tailscale] error: denied: requested access to the resource is denied

I also tried inCluster: {} but I got a different error about needing to setup a builder or something?

I have also tried to use k3s / k3d but I don't know how to set the private docker registry settings for it either, so I get the push errors. When I do a docker login it pushes to live docker hub which is not what I wanted.

It would be great if there was a way to use more than 1 local cluster for development somehow.