How to use CustomResourceOptions?

[LiHRaM]

Hi,

I've been looking through the code and am really struggling to understand how to use CustomResourceOptions (e.g. to mark a resource as protected when importing). Any pointers on how this can be done?

Happy to submit a PR with docs if you can point me in the right direction.

[UnoSD]

hi @LiHRaM

I have not added full support for CustomResourceOptions (PRs are welcome), but I have added a custom operation for dependsOn as it was needed in one of my projects, you can use that operation on any resource.

Have a glance at the dependsOn implementation; from a design perspective, I believe it may be good to keep it simple and just add more operations from CRO instead of a nested CE, but I have not give it so much thoughts so far, so happy to discuss. Also, let me know if you attempt and get stuck, happy to support.

[LiHRaM]

Hey, thanks for the response! I'm sorry for taking so long to respond, I've been thinking this over.

My main concern is how up-to-date this project will be in the long run. To stay relevant, it would probably need some automation that catches when packages are released upstream, generate code based on the updates and release to NuGet.

This brings me to the issue of having a flatter approach (where we flatten arguments and options into a single CE) - if we don't mirror the structure of the SDKs, aren't we designing a solution that forces us to manually capture a lot of breaking changes?

I noticed, for example, that there seemed to be no CE equivalent for the Provider at the root of the Kubernetes package. I haven't looked through the code to see why it hasn't been caught, but I was hoping maybe we could come up with something that can generate the computation expressions more thoroughly, even if that means we'll have to nest it one layer deeper.

This may go against your design ideas for the project, I realize, and am happy to take my ideas elsewhere as well, but I was hoping that we could figure something out where the maintenance burden is minimal going forward.

I also recognize that you may have some design ideas that fit with the vision of automating as much as possible, and would be interested in hearing what your thoughts are on this in general?

[UnoSD]

No worries, I'm myself guilty of abandoning the project every once in a while when my life gets more complicated :)

Ideally we would trigger the build on NuGet updates, but given I do not have time to work on this, a quick win could be to run the pipeline on a schedule (once a week?) so it would stay current.

This brings me to the issue of having a flatter approach (where we flatten arguments and options into a single CE) - if we don't mirror the structure of the SDKs, aren't we designing a solution that forces us to manually capture a lot of breaking changes?

Can you expand on that? Is your concern around clashing names?

I noticed, for example, that there seemed to be no CE equivalent for the Provider at the root of the Kubernetes package.

I have not used that package much, can you describe the problem? We can try and find a solution here.

[LiHRaM]

No worries, I'm myself guilty of abandoning the project every once in a while when my life gets more complicated :)

Haha yeah, that's often the nature of open source I'm afraid. :smile: That said, I really enjoy this project and want to thank you for the work you have put in already. It's a huge convenience when working with Pulumi.

Ideally we would trigger the build on NuGet updates, but given I do not have time to work on this, a quick win could be to run the pipeline on a schedule (once a week?) so it would stay current.

That sounds great actually!

This brings me to the issue of having a flatter approach (where we flatten arguments and options into a single CE) - if we don't mirror the structure of the SDKs, aren't we designing a solution that forces us to manually capture a lot of breaking changes?

Can you expand on that? Is your concern around clashing names?

I noticed, for example, that there seemed to be no CE equivalent for the Provider at the root of the Kubernetes package.

I have not used that package much, can you describe the problem? We can try and find a solution here.

I want to expand on this for both questions you have here.

I've made a minimal example here: https://github.com/LiHRaM/pulumi-fsharp-playground/blob/main/Program.fs.

The CE provider is not found where I would expect to find it:

And when just writing provider and trying to use intellisense to suggest the location yields nothing.

---

What you'll notice (and I'm sure you're familiar with this) - is that most of the resources accept name, args and options. Since you're using a CE to sort of accept the args and options at the same level as the name, the way we generate these things becomes less flexible.

For example, it's impossible to use the normal constructor to create the object but use a CE to submit the args. I've wanted this so I could mark a resource as protected or to submit a custom Kubernetes provider to augment some resources. If we generated a CE for each Args instead, it would be more nested, but also, I think, easier to compose and generate for.

[UnoSD]

I really enjoy this project and want to thank you for the work you have put in already

Thank you. Glad it was helpful.

Ideally we would trigger the build on NuGet updates, but given I do not have time to work on this, a quick win could be to run the pipeline on a schedule (once a week?) so it would stay current.

That sounds great actually!

I just realised that I need to update also the package version on the fsproj; I wonder if I could use a wildcard to always get the latest, my proj file knowledge is a bit outdated.

The CE provider is not found where I would expect to find it

I suspected you were referring to that; I am using the Pulumi JSON schema (see loadSchema' in Schema.fs):

let private loadSchema' version provider =
    $"https://raw.githubusercontent.com/pulumi/pulumi-{provider}/v{version}" +
    $"/provider/cmd/pulumi-resource-{provider}/schema.json" |>
    getSchemaFromFileCacheOrDownload provider version |>
    JsonValue.Parse

I believe it does not contain type information for Provider (happy to be proven wrong, haven't checked the JSON again); however, if that's a common class with the same properties for all providers, it should be fairly easy to hardcode it's structure.

the way we generate these things becomes less flexible it's impossible to use the normal constructor to create the object but use a CE to submit the args

at the current stage, it's true; however, if we were to add all the CustomReourceOptions in the main CE, would that be ever needed? it feels like it's a workaround for the current limitation, but once all the properties are there it may be unlikely that someone wants to mix up classic and extensions syntax.

that said, another quick win to support this would just be to support CustomReourceOptions in a Yield as a workaround; I don't like introducing breaking changes, but I may not need to remove it at all later and leave it for compatibility and to share the same CRO across resources (hope this makes sense)

For example, it's impossible to use the normal constructor to create the object but use a CE to submit the args. I've wanted this so I could mark a resource as protected or to submit a custom Kubernetes provider to augment some resources. If we generated a CE for each Args instead, it would be more nested, but also, I think, easier to compose and generate for.

that's exactly what I meant, it could be helpful to compose and share the same CRO across resources; I'm starting to like this idea. That can also be split easily into two tasks: 1) Add Yield to support CRO object 2) Create CRO CE

[LiHRaM]

I suspected you were referring to that; I am using the Pulumi JSON schema (see loadSchema' in Schema.fs):

let private loadSchema' version provider =
    $"https://raw.githubusercontent.com/pulumi/pulumi-{provider}/v{version}" +
    $"/provider/cmd/pulumi-resource-{provider}/schema.json" |>
    getSchemaFromFileCacheOrDownload provider version |>
    JsonValue.Parse

I believe it does not contain type information for Provider (happy to be proven wrong, haven't checked the JSON again); however, if that's a common class with the same properties for all providers, it should be fairly easy to hardcode it's structure.

Ah, I see! Well, it is there, but it is at the root level. Here's some output from my shell to show what I mean - my prompt is ;, so the commands are on those lines and the output is structured data, shell is nushell, anyway:

; let data = (fetch https://raw.githubusercontent.com/pulumi/pulumi-kubernetes/v3.18.2/provider/cmd/pulumi-resource-kubernetes/schema.json)
; $data
╭─────────────┬──────────────────────────────────────────────────────────────────╮
│ name        │ kubernetes                                                       │
│ displayName │ Kubernetes                                                       │
│ description │ A Pulumi package for creating and managing Kubernetes resources. │
│ keywords    │ [list 4 items]                                                   │
│ homepage    │ https://pulumi.com                                               │
│ license     │ Apache-2.0                                                       │
│ repository  │ https://github.com/pulumi/pulumi-kubernetes                      │
│ publisher   │ Pulumi                                                           │
│ config      │ {record 1 field}                                                 │
│ types       │ {record 823 fields}                                              │
│ provider    │ {record 3 fields}                                                │
│ resources   │ {record 226 fields}                                              │
│ language    │ {record 4 fields}                                                │
╰─────────────┴──────────────────────────────────────────────────────────────────╯
; $data.provider
╭─────────────────┬───────────────────────────────────────────────╮
│ description     │ The provider type for the kubernetes package. │
│ type            │ object                                        │
│ inputProperties │ {record 12 fields}                            │
╰─────────────────┴───────────────────────────────────────────────╯
; $data.provider.inputProperties
╭─────────────────────────────┬───────────────────╮
│ cluster                     │ {record 2 fields} │
│ context                     │ {record 2 fields} │
│ enableConfigMapMutable      │ {record 3 fields} │
│ enableDryRun                │ {record 3 fields} │
│ enableReplaceCRD            │ {record 3 fields} │
│ helmReleaseSettings         │ {record 2 fields} │
│ kubeClientSettings          │ {record 2 fields} │
│ kubeconfig                  │ {record 4 fields} │
│ namespace                   │ {record 2 fields} │
│ renderYamlToDirectory       │ {record 2 fields} │
│ suppressDeprecationWarnings │ {record 3 fields} │
│ suppressHelmHookWarnings    │ {record 3 fields} │
╰─────────────────────────────┴───────────────────╯
; 

I'm sure you'll have a better idea than I on how this might impact the code generation. :smile:

that said, another quick win to support this would just be to support CustomResourceOptions in a Yield as a workaround; I don't like introducing breaking changes, but I may not need to remove it at all later and leave it for compatibility and to share the same CRO across resources (hope this makes sense)

I'm starting to like this idea. That can also be split easily into two tasks: 1) Add Yield to support CRO object 2) Create CRO CE

Yep, that sounds great to me. :smile: I'd be happy to submit PRs for these features.

[UnoSD]

thank you for nushell, seems awesome! I've been meaning to try and build an F# shell, but that seems to achieve similar functionality!

happy to review the PR, please let me know if you need help as I appreciate the AST is a bit of an undocumented nightmare

[nodakai]

So, a consensus was reached regarding the API design?

This is just a user's voice... from my previous experiences with Terraform, supporting https://www.pulumi.com/docs/intro/concepts/resources/options/ignorechanges/ (as part of the general CRO) will be quite useful in many cases. (I'm dreaming of string typing of the ignore specifier but realistically the required amount of work might be too much compared to the benefit brought by strong typing)

[LiHRaM]

Hey, yeah, we agreed on an initial WIP at least, but I simply haven't had time to work on it.

[mvsmal]

Hi @UnoSD , I believe I have time to implement CRO CE in some way. I already started playing around with it. I have a few questions.

1. Which option do you prefer: a generated CE with the help of Reflection or just hard-coded version? With generated, it will be easier to keep up with the updates, however I don't expect them to be frequent, as this is a pretty stable Pulumi type. Also it will require much more work, as it will probably need a different Myriad Generator. I am not sure if Pulumi.FSharp.Myriad project can be used for that. But it has lots of useful staff, which could be reused.
2. Where would you like to place that CE? In Pulumi.FSharp.Core root namespace?

[UnoSD]

hi @mvsmal, as you pointed out, this is going to be a pretty stable API, hence I'd be happy to hardcode it to reduce complexity and compile time.

for 2., I believe I already added a dependsOn, I can't remember if I just dropped it in there or if it was a well thought design; if you believe it does not fit in the same place, we can discuss moving both to a new namespace/module/project.

[mvsmal]

Here is a PR for CustomResourceOptions as a nested CE: https://github.com/UnoSD/Pulumi.FSharp.Extensions/pull/29