The update-codegen script does not automatically generate files & directories

[max-rocket-internet]

To quote the README:

The update-codegen script will automatically generate the following files & directories:

• pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go
• pkg/generated/

But:

$ git clone https://github.com/kubernetes/sample-controller.git
$ cd sample-controller
$ rm -f pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go
$ ./hack/update-codegen.sh
Generating deepcopy funcs
Generating clientset for samplecontroller:v1alpha1 at k8s.io/sample-controller/pkg/generated/clientset
Generating listers for samplecontroller:v1alpha1 at k8s.io/sample-controller/pkg/generated/listers
Generating informers for samplecontroller:v1alpha1 at k8s.io/sample-controller/pkg/generated/informers
$ ls pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go
ls: pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go: No such file or directory

Am I misunderstanding how this is supposed to work?

[farah]

@dims @sttts Hey guys, any idea on how to solve this? It's a bit confusing for beginners.

k8s.io$ git clone https://github.com/kubernetes/sample-controller
k8s.io$ cd sample-controller
sample-controller$ ./hack/update-codegen.sh
bash: ../code-generator/generate-groups.sh: No such file or directory

thanks in advance.

[max-rocket-internet]

TBH I think this code generation thing adds more headache/complexity than it solves, especially when you're just beginning to write a controller using this repo as a start.

[farah]

hey @max-rocket-internet, just realised that the fix is actually in the read me. Run go mod vendor and you should be good.

[caboteria]

Here's what I've figured out from a couple of hours of faffing around with this:

The first problem is "how do you run generate-groups.sh"? There are two answers. One was pointed out by @farah : populate the vendor directory. The other is to clone the code-generator project so that it's a sibling of sample-controller (which is where it lives in the kubernetes/kubernetes repo). You can look in hack/update-codegen.sh and you'll see that the script looks in the vendor directory and then falls back to ../code-generator.

The second problem is "where does generate-groups.sh put the files that it generates"? Remember how this repo is a mirror of a small part of kubernetes/kubernetes? Well, generate-groups.sh assumes and requires that everything is laid out exactly like it is in the kubernetes project. In that project, sample-controller is in k8s.io/sample-controller, i.e. {parent}/{project}. You'll see in generate-groups.sh that it goes up three directories, then uses the module path as a directory path which means that it doesn't work properly if your project's module isn't {parent}/{project}. You can hack update-codegen.sh to use a different output-base but you might still end up having to move the files after they're generated. I did.

This smells like it was written before Go modules and never updated to work with modules since it just so happens to work with the directory structure in the kubernetes/kubernetes project.

HTH

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[ferama]

Did you find any solution? The go mod vendor doesn't work for me. This is what I did:

$ git clone https://github.com/kubernetes/sample-controller
$ cd sample-controller
$ go mod vendor
$ ./hack/update-codegen.sh

The script generates a bunch of dirs and filess inside my home

k8s.io/
k8s.io/sample-controller
k8s.io/sample-controller/pkg
k8s.io/sample-controller/pkg/generated
k8s.io/sample-controller/pkg/generated/clientset
k8s.io/sample-controller/pkg/generated/clientset/versioned
k8s.io/sample-controller/pkg/generated/clientset/versioned/scheme
k8s.io/sample-controller/pkg/generated/clientset/versioned/scheme/doc.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/scheme/register.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1/generated_expansion.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1/samplecontroller_client.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1/fake
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1/fake/doc.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1/fake/fake_samplecontroller_client.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1/doc.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/fake
k8s.io/sample-controller/pkg/generated/clientset/versioned/fake/clientset_generated.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/fake/doc.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/fake/register.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/clientset.go
k8s.io/sample-controller/pkg/generated/clientset/versioned/doc.go

but all the files there are empty

[max-rocket-internet]

I switched to kubebuilder, I think it's better.

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vadasambar]

I edited my update-codegen script like this:

bash "${CODEGEN_PKG}"/generate-groups.sh "deepcopy,client,informer,lister" \
  pkg/generated pkg/apis \
  mycontroller:v1alpha1 \
  --output-base "$(dirname "${BASH_SOURCE[0]}")/../" \
  --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt

This generates the files under the parent directory's pkg folder but all of the files are empty as mentioned in this comment. Wonder if anyone found a solution for the empty files.

[vadasambar]

/remove-lifecycle rotten

[vadasambar]

/reopen

[k8s-ci-robot]

@vadasambar: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[max-rocket-internet]

/reopen

[k8s-ci-robot]

@max-rocket-internet: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[max-rocket-internet]

Reopened for @vadasambar but I moved on from this topic a long time ago 🙂

[vadasambar]

I can reproduce the same problem when I try running the update-codegen.sh script against the original sample-controller inside the kubernetes repo.

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ tree pkg

pkg
├── apis
│   └── samplecontroller
│       ├── register.go
│       └── v1alpha1
│           ├── doc.go
│           ├── register.go
│           ├── types.go
│           └── zz_generated.deepcopy.go
├── generated
│   ├── clientset
│   │   └── versioned
│   │       ├── clientset.go
│   │       ├── doc.go
│   │       ├── fake
│   │       │   ├── clientset_generated.go
│   │       │   ├── doc.go
│   │       │   └── register.go
│   │       ├── scheme
│   │       │   ├── doc.go
│   │       │   └── register.go
│   │       └── typed
│   │           └── samplecontroller
│   │               └── v1alpha1
│   │                   ├── doc.go
│   │                   ├── fake
│   │                   │   ├── doc.go
│   │                   │   ├── fake_foo.go
│   │                   │   └── fake_samplecontroller_client.go
│   │                   ├── foo.go
│   │                   ├── generated_expansion.go
│   │                   └── samplecontroller_client.go
│   ├── informers
│   │   └── externalversions
│   │       ├── factory.go
│   │       ├── generic.go
│   │       ├── internalinterfaces
│   │       │   └── factory_interfaces.go
│   │       └── samplecontroller
│   │           ├── interface.go
│   │           └── v1alpha1
│   │               ├── foo.go
│   │               └── interface.go
│   └── listers
│       └── samplecontroller
│           └── v1alpha1
│               ├── expansion_generated.go
│               └── foo.go
└── signals
    ├── signal.go
    ├── signal_posix.go
    └── signal_windows.go

21 directories, 30 files

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ cat pkg/generated/clientset/versioned/clientset.go

/*
Copyright The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// Code generated by client-gen. DO NOT EDIT.

package versioned

import (
        "fmt"
        "net/http"

        discovery "k8s.io/client-go/discovery"
        rest "k8s.io/client-go/rest"
        flowcontrol "k8s.io/client-go/util/flowcontrol"
        samplecontrollerv1alpha1 "k8s.io/sample-controller/pkg/generated/clientset/versioned/typed/samplecontroller/v1alpha1"
)

type Interface interface {
        Discovery() discovery.DiscoveryInterface
        SamplecontrollerV1alpha1() samplecontrollerv1alpha1.SamplecontrollerV1alpha1Interface
}

// Clientset contains the clients for groups. Each group has exactly one
// version included in a Clientset.
type Clientset struct {
        *discovery.DiscoveryClient
        samplecontrollerV1alpha1 *samplecontrollerv1alpha1.SamplecontrollerV1alpha1Client
}

// SamplecontrollerV1alpha1 retrieves the SamplecontrollerV1alpha1Client
func (c *Clientset) SamplecontrollerV1alpha1() samplecontrollerv1alpha1.SamplecontrollerV1alpha1Interface {
        return c.samplecontrollerV1alpha1
}

// Discovery retrieves the DiscoveryClient
func (c *Clientset) Discovery() discovery.DiscoveryInterface {
        if c == nil {
                return nil
        }
        return c.DiscoveryClient
}

// NewForConfig creates a new Clientset for the given config.
// If config's RateLimiter is not set and QPS and Burst are acceptable,
// NewForConfig will generate a rate-limiter in configShallowCopy.
// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
// where httpClient was generated with rest.HTTPClientFor(c).
func NewForConfig(c *rest.Config) (*Clientset, error) {
        configShallowCopy := *c

        if configShallowCopy.UserAgent == "" {
                configShallowCopy.UserAgent = rest.DefaultKubernetesUserAgent()
        }

        // share the transport between all clients
        httpClient, err := rest.HTTPClientFor(&configShallowCopy)
        if err != nil {
                return nil, err
        }

        return NewForConfigAndClient(&configShallowCopy, httpClient)
}

// NewForConfigAndClient creates a new Clientset for the given config and http client.
// Note the http client provided takes precedence over the configured transport values.
// If config's RateLimiter is not set and QPS and Burst are acceptable,
// NewForConfigAndClient will generate a rate-limiter in configShallowCopy.
func NewForConfigAndClient(c *rest.Config, httpClient *http.Client) (*Clientset, error) {
        configShallowCopy := *c
        if configShallowCopy.RateLimiter == nil && configShallowCopy.QPS > 0 {
                if configShallowCopy.Burst <= 0 {
                        return nil, fmt.Errorf("burst is required to be greater than 0 when RateLimiter is not set and QPS is set to greater than 0")
                }
                configShallowCopy.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(configShallowCopy.QPS, configShallowCopy.Burst)
        }

        var cs Clientset
        var err error
        cs.samplecontrollerV1alpha1, err = samplecontrollerv1alpha1.NewForConfigAndClient(&configShallowCopy, httpClient)
        if err != nil {
                return nil, err
        }

        cs.DiscoveryClient, err = discovery.NewDiscoveryClientForConfigAndClient(&configShallowCopy, httpClient)
        if err != nil {
                return nil, err
        }
        return &cs, nil
}

// NewForConfigOrDie creates a new Clientset for the given config and
// panics if there is an error in the config.
func NewForConfigOrDie(c *rest.Config) *Clientset {
        cs, err := NewForConfig(c)
        if err != nil {
                panic(err)
        }
        return cs
}

// New creates a new Clientset for the given RESTClient.
func New(c rest.Interface) *Clientset {
        var cs Clientset
        cs.samplecontrollerV1alpha1 = samplecontrollerv1alpha1.New(c)

        cs.DiscoveryClient = discovery.NewDiscoveryClient(c)
        return &cs
}

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ sudo rm -R pkg/generated pkg/apis

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ tree pkg

pkg
└── signals
    ├── signal.go
    ├── signal_posix.go
    └── signal_windows.go

1 directory, 3 files

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ ./hack/update-codegen.sh 
Generating deepcopy funcs
Generating clientset for samplecontroller:v1alpha1 at k8s.io/sample-controller/pkg/generated/clientset
Generating listers for samplecontroller:v1alpha1 at k8s.io/sample-controller/pkg/generated/listers
Generating informers for samplecontroller:v1alpha1 at k8s.io/sample-controller/pkg/generated/informers

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ tree pkg

pkg
├── generated
│   └── clientset
│       └── versioned
│           ├── clientset.go
│           ├── doc.go
│           ├── fake
│           │   ├── clientset_generated.go
│           │   ├── doc.go
│           │   └── register.go
│           ├── scheme
│           │   ├── doc.go
│           │   └── register.go
│           └── typed
│               └── samplecontroller
│                   └── v1alpha1
│                       ├── doc.go
│                       ├── fake
│                       │   ├── doc.go
│                       │   └── fake_samplecontroller_client.go
│                       ├── generated_expansion.go
│                       └── samplecontroller_client.go
└── signals
    ├── signal.go
    ├── signal_posix.go
    └── signal_windows.go

10 directories, 15 files

suraj@suraj:~/sandbox/kubernetes/staging/src/k8s.io/sample-controller$ cat pkg/generated/clientset/versioned/clientset.go

/*
Copyright The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// Code generated by client-gen. DO NOT EDIT.

package versioned

[vadasambar]

Started a thread around the problem in Kubernetes slack here

[vadasambar]

This works for me:

#!/usr/bin/env bash

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -o errexit
set -o nounset
set -o pipefail

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}

# Pre-requisites
# 1. this script needs ./pkg/apis/samplecontroller/v1alpha1 to generate everything else (generated files will be empty otherwise)
# 2. changing --output-base to something else doesn't work
# 
# Only the following things can be auto-generated:
# 1. ./pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go
# 2. everything under ./pkg/apis/generated
bash "${CODEGEN_PKG}"/generate-groups.sh "deepcopy,client,informer,lister" \
  k8s.io/sample-controller/pkg/generated k8s.io/sample-controller/pkg/apis \
  samplecontroller:v1alpha1 \
  --output-base "$(dirname "${BASH_SOURCE[0]}")/../../.." \
  --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt

# This block:
# output_base: the directory outside the repo where auto-generated files are created
# 1. removes existing auto-generated files
# 2. copies over the auto-generated files from the output_base to pkg/apis and pkg/generated
# 3. removes output_base
rm -R ./pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go ./pkg/generated -f
mv "$(dirname "${BASH_SOURCE[0]}")/../../../k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1/zz_generated.deepcopy.go" $SCRIPT_ROOT/pkg/apis/samplecontroller/v1alpha1 -f
mv "$(dirname "${BASH_SOURCE[0]}")/../../../k8s.io/sample-controller/pkg/generated" $SCRIPT_ROOT/pkg
rm -R "$(dirname "${BASH_SOURCE[0]}")/../../../k8s.io"

# To use your own boilerplate text append:
#   --go-header-file "${SCRIPT_ROOT}"/hack/custom-boilerplate.go.txt

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.