feat: codegen with portable APIs

[idlsoft]

Go modules are generating dagger APIs under an internal package.

This means that

• code written in a given module cannot be used directly anywhere else
• you can't use go libraries with dagger.io/dagger imports.

This PR adds an option to import APIs as dagger.io/dagger instead of module/internal/dagger (see #11417)

There is a simple unittest, but it can also be tested locally using dagger -i call engine-dev playground terminal:

mkdir mod1; cd mod1
dagger init --sdk=go --portable-api
dagger call container-echo --string-arg "Hello, world\!" stdout

Note the import in main.go, the replace in go.mod and an extra go.mod under internal/dagger.

Changes are grouped by commit to make it easier to follow.

[TomChv]

@idlsoft Hey!

What is the difference between your proposal and the generated client feature? #9582

go mod init example.com
dagger init # No SDK, this can be used outside module
dagger install xxx # Install any module you want to call directly from your Golang project
dagger client install go

This will generate a client that you can use to call your module from an outside.

[idlsoft]

What is the difference between your proposal and the generated client feature? #9582

This serves a different purpose, and gives extra flexibility on how you write your module code. In your module you get to import dagger.io/dagger instead of module/internal/dagger, which then allows you to have a go library outside your modules with the same dagger.io/dagger imports and use it as a regular go dependency in all your modules.

Unlike a dagger client, which only lets you call dagger functions, this will allow you to reuse any code, with any types not serializable by the dagger engine. It's not cross-language but that's not its purpose. And it's not a replacement, it doesn't affect how modules talk to each other.

Btw this started with me asking a question here.

[TomChv]

This serves a different purpose, and gives extra flexibility on how you write your module code. In your module you get to import dagger.io/dagger instead of module/internal/dagger, which then allows you to have a go library outside your modules with the same dagger.io/dagger imports and use it as a regular go dependency in all your modules.

So do that means you can technically go install your module if it's only using the core bindings?

Btw this started with me asking a question here.

Okay yeah I understand the use case

[idlsoft]

Code looks ok, I left 2 questions. I need to test that PR.

FWIW I wrote a basic unittest, which was surprisingly easy to do.

I'm also curious, what would be the diff between your PR and if we supported using the library dagger.io/dagger in the module code instead of redeclaring all the core bindings.

That would be ideal, but I assumed there was a good reason for that redeclaration. Maybe because it's guaranteed to be tied to the engine? But yeah, that would have been neat.

[TomChv]

That would be ideal, but I assumed there was a good reason for that redeclaration. Maybe because it's guaranteed to be tied to the engine? But yeah, that would have been neat.

There are reasons but that's a topic I plan to work on in early January because I might have an idea on how to workaround it. If that's ok for you, I'll put this PR on hold, try to find a way to use the library inside module and if I can't make it work I'll merge your PR.

[idlsoft]

There are reasons but that's a topic I plan to work on in early January because I might have an idea on how to workaround it. If that's ok for you, I'll put this PR on hold, try to find a way to use the library inside module and if I can't make it work I'll merge your PR.

Sounds good, thank you. The 1st commit with PendingExperimental is independent though, and will solve the dagger develop --sdk=go --with-self-calls issue, so could be useful.

[TomChv]

The 1st commit with PendingExperimental is independent though, and will solve the dagger develop --sdk=go --with-self-calls issue, so could be useful.

/cc @eunomie on that one

[idlsoft]

There are reasons but that's a topic I plan to work on in early January because I might have an idea on how to workaround it.

I see discussions about committing generated code to the repo, wonder if what you're doing may help make it cleaner.

[TomChv]

I see discussions about committing generated code to the repo, wonder if what you're doing may help make it cleaner.

Yeah it should make the generated code much lighter, ideally we should only commit the generated bindings and that's it. I'm planning to work on that this week, I'll give update once I have done some progress;