[Issue] Figure out flow for custom domain names

[pamelafox]

• [X] Make sure you've installed the latest version using instructions in the wiki

Output from azd version

azd version 0.7.0-beta.1 (commit 9ce71659f7688d0dc3dda8b84e5accedca58cf01)

Describe the bug

Everything is working as currently designed, so this is more a bug in the process than the code.

I would like to be able to use azd to deploy a website that has a custom domain attached (on Static Web Apps, App Service, or Container Apps). However, if I specify a domain name in Bicep using 'Microsoft.Web/staticSites/customDomains@2022-03-01' that has not yet been verified, then I get an error during the provision stage. I cannot verify a domain name until I know the website endpoint, however, since the DNS record is a CNAME for 'www' with the value of the website endpoint.

This is how I ended up doing it for my personal website, www.pamelafox.org:

1. I set up the infra with domainName as a parameter defaulting to empty string, and only conditionally create the custom domain if the parameter is non-empty: https://github.com/pamelafox/pamelafox-site/blob/main/infra/swa.bicep#L24
2. I then run azd up with domainName still empty in main.parameters.json
3. When I see the deployed endpoint URI, I go to my registrar and add a CNAME of www->endpoint and wait a few minutes
4. I set domainName in main.parameters.json and re-deploy.

That seemed to work, but it's possible that I also intervened in the Portal in between those steps. Maybe there'd be a more straightforward approach if I used Azure's registrar? I already have a registrar I pay for, so I set it up there.

[weikanglim]

Not sure if it helps, but I usually think of setting up DNS for an application as two separate steps:

1. Get the app up-and-running with a private endpoint (run provision)
2. At some point when the code is deployed, and everything is healthy, use DNS records to point to the private endpoint. You could choose to use the Azure offering here. It can either be done through the Portal manually, or via a separate bicep deployment to be automated.

I don't think you'd want to set up DNS immediately as part of the app provisioning, until you have everything confirmed working.

[rajeshkamal5050]

We need to provide some guidance around this before GA.

[rajeshkamal5050]

@weikanglim @pamelafox is there any existing Azure documentation which already exists which we can point to as part of Azd FAQ?

cc @savannahostrowski @ellismg

[savannahostrowski]

I managed to setup a custom domain via Bicep for an ACA template but it's not perfect and it's also a bit of a chicken and an egg problem.

I have my certificate as a managed certificate in the Container App Environment (added after first provision). Then I just use the certificateId as part of the customDomain ingress property in the Container App. It's probably harmless but less than ideal as I'd rather have no one know anything about my deployments but I hardcoded the managed certificate name. This was just the simplest thing I could think of.

So at the end of the day, the UX flow for someone consuming this template would be:

• Comment out the custom domain and managed certificate resource bits for first provision
• Add a cert as a managed certificate in the CAE
• Grab certificate name
• Uncomment the custom domain Bicep and pop in the cert name
• Re-provision

https://github.com/savannahostrowski/terminal-personal-site/blob/07e32659fde54dc82ac45acc9544931d33fd0be5/infra/core/host/container-app.bicep#L31

[pamelafox]

It may be possible to do a slightly nicer workflow by using conditions in the Bicep? i.e. customDomain is in parameters.json, only create the resources in the Bicep if customDomain is !empty. Or is there a reason commenting out is necessary vs. Bicep conditionals?

[savannahostrowski]

I'm no Bicep expert so perhaps a conditional statement will suffice! I will investigate this a bit further and circle back!

[weikanglim]

I'd love for the flow to look sequential:

1. Provision infrastructure provision
2. Provision domain names provision public-dns

But if we find the single IaC easier to reason about, that'd be fine too.

From an advanced IaC authoring perspective, creating bicep conditionals based on Azure state may make it harder to test and maintain over time.

[mip1983]

At the moment to do this, I have to use 'azd infra synth'. In the containerApp.tmpl.yaml files for my web projects I add lines for custom domain under 'ingress':

    ingress:
      external: true
      targetPort: {{ targetPortOrDefault 8080 }}
      transport: auto
      allowInsecure: false
      customDomains: 
      - name: {{ .Env.WEB_URL }}
        bindingType: SniEnabled
        certificateId: {{ .Env.SSL_CERT_ID }}

I then add those environment variables as input and output in main.bicep and main.parameters.json so they can be fed by variables in my CI/CD environment.

To deploy, I need to do the 'azd provision' step first. Once the container app environment is up, I can manually upload my cert and configure DNS as required. Then azd deploy after that. (My cert and domain aren't Azure managed at the moment but that's the direction I'd like to go eventually.)

If I don't put those lines in the yaml, the SSL cert binding gets removed by azd deploy, so it's the only way I've found so far that I can deploy with this can keep the custom domain in place.

It would be great if in Aspire you could specify that a project will have a custom domain, something like:

pseudocode:

var web= builder.AddProject<Projects.Web>("web")
	.WithExternalHttpEndpoints(hasCustomDomain: true, userProvidedCertificate: true)

If 'userProvidedCertificate' is true: Then it could put the 'customDomains' part into the yaml automatically and generate environment variables for the user to provide for url and certificate id. Then azd deploy could prompt you to provide values for these environment variables as part of the deploy process if you're running locally, or pick up from your CI/CD env.

If 'userProvidedCertificate' is false: Generate an azure managed certificate (and domain?) as part of the provisioning/deploy process.

I'd love to get away from needing 'azd infra synth' so that everything is driven by code from my aspire project and this is one of the main puzzle pieces preventing that at the moment for me.

[ellismg]

I'd love to get away from needing 'azd infra synth' so that everything is driven by code from my aspire project and this is one of the main puzzle pieces preventing that at the moment for me.

The medium-term plan (post the initial aspire GA, but high priority after that) is to provide a way to allow full customization of the ContainerApp object (i.e. what you can control via the .yaml file today with azd infra synth, but via a strongly typed object in C#). @davidfowl has been prototyping some ideas there. Once we have that, I think we'd be able to build what you want. We already have support for adding parameters to infrastructure via the app host, and I think that combined with the ability to modify the ContainerApp in the app host would allow you to wire these things up in a way that azd deploy would be able to handle

[mip1983]

@ellismg That's great to know the vision and this pain point will go away, thanks for sharing that. That'll really simplify things, looking forward to it.

[softwaretirol]

This is a blocker for us for adapting aspire.

[relounge]

@mip1983 how did you manage customDomains to work? I added the same to my service yaml file inside ingress settings, but azd deply seems to ignore this lines and drops network settings again and again. So annoying

[mip1983]

@mip1983 how did you manage customDomains to work? I added the same to my service yaml file inside ingress settings, but azd deply seems to ignore this lines and drops network settings again and again. So annoying

There's now a command in azd to get it to preserve your domain settings:

azd config set alpha.aca.persistDomains on

So in my build pipeline (Azure DevOps):

- pwsh: |
     azd config set alpha.aca.persistDomains on
     displayName: Configure AZD to persist domain name configuration on container app environment.

The domain and cert are manually set up in the Azure portal after deploy, but this command ensures they're preserved.

I'm using an externally managed domain and certificate at the moment, eventually would like this to all be azure managed and generated via aspire.

[relounge]

I use GutHub actions for deployment. I added a stet with this setting but nothing has changed. ` - name: Log in with Azure (Client Credentials) if: ${{ env.AZURE_CREDENTIALS != '' }} run: | $info = $Env:AZURE_CREDENTIALS | ConvertFrom-Json -AsHashtable; Write-Host "::add-mask::$($info.clientSecret)"

      azd auth login `
        --client-id "$($info.clientId)" `
        --client-secret "$($info.clientSecret)" `
        --tenant-id "$($info.tenantId)"
    shell: pwsh
    env:
      AZURE_CREDENTIALS: ${{ secrets.AZURE_CREDENTIALS }}

  - name: Setup azd
    run: azd config set aca.persistDomains on

  - name: Provision Infrastructure
    run: azd provision --no-prompt
    env:
      AZD_INITIAL_ENVIRONMENT_CONFIG: ${{ secrets.AZD_INITIAL_ENVIRONMENT_CONFIG }}

  - name: Deploy Application
    run: azd deploy --no-prompt`

Does anyone know what is wrong here?

I did the same locally from the dev machine and used "azd up" and it seem to take the customDomain section but it end up with error "Invalid certificateId". @mip1983 please, can you tell me what is this certificate Id? I use azure managed certificated and in the portal I see only certificate names. I tried to use them but no luck

[mip1983]

@relounge If you're using 'azd config set alpha.aca.persistDomains on' you can remove the custom domains bit from your YAML file. That's only something I used along with output variables from bicep before this alpha.aca.persistDomains existed as a work around to stop it clearing the domain config each time.

Unless you're doing any other customization, you can just take the YAML that infra synth generates (or not use it at all if this is your only customization) along with this azd config command.

[relounge]

Thanks it worked. Added this to my workflow file.

  - name: Setup custom domains
    run: azd config set alpha.aca.persistDomains on

The only one thing left is to understand how to stop dropping ingress settings all the time, I mean allow an endpoint to be external after deployment. But this is out of this topic

[mip1983]

Thanks it worked. Added this to my workflow file.

  - name: Setup custom domains
    run: azd config set alpha.aca.persistDomains on

The only one thing left is to understand how to stop dropping ingress settings all the time, I mean allow an endpoint to be external after deployment. But this is out of this topic

Hopefully should be a simple one liner 'WithExternalHttpEndpoints' in Program.cs of your Aspire.AppHost project e.g.:

var webApp = builder.AddProject<Projects.web>("<your-web-project>")
	.WithExternalHttpEndpoints();

[mip1983]

@rajeshkamal5050

Is there any guidance in the works for this area? Would be good to switch over to an aspire/azure based dns and certificate solution

[rajeshkamal5050]

@mip1983 this depends on the work being done as part of https://github.com/Azure/azure-dev/issues/3292 which is currently being worked on.

cc @vhvb1989 @ellismg @davidfowl @mitchdenny

[mitchdenny]

We are just about to backport a change into release/9.0 of Aspire to allow custom domains to be set when using PublishAsAzureContainerApp. We still need to support the current behavior for people that aren't using this however.

[gabsamples6]

@mitchdenny @pamelafox Hello, wondering if someone can point to a documentation on how to implement "Custom Domains" in aspire, we usually use Azure pipeline for deployment for all our stuff, and being new to Aspire I noticed that azcmd is the only way is documented..

My questions

1. I would like to use an Azure Pipeline for CI/CD and redeploy to the same Custom Domain, currently aspire generate a random one all the time
2. I have jumped from one issue to another without finding a concrete example or documentation on how to deploy using a customDomain or to a url that does not change everytime I publish an update.
3. or how can I extract the yaml generated so that I can put in a dedicate pipeline that runs whenver a change takes place.

Many thanks

[mip1983]

@mitchdenny @pamelafox Hello, wondering if someone can point to a documentation on how to implement "Custom Domains" in aspire, we usually use Azure pipeline for deployment for all our stuff, and being new to Aspire I noticed that azcmd is the only way is documented..

My questions

1. I would like to use an Azure Pipeline for CI/CD and redeploy to the same Custom Domain, currently aspire generate a random one all the time
2. I have jumped from one issue to another without finding a concrete example or documentation on how to deploy using a customDomain or to a url that does not change everytime I publish an update.
3. or how can I extract the yaml generated so that I can put in a dedicate pipeline that runs whenver a change takes place.

Many thanks

It's not finalized in Aspire at the moment, so you have to use a '#pragma warning disable ASPIREACADOMAINS001' in your code to do it, but an example is:

var domainParam = builder.AddParameter("customdomain");
var certParam= builder.AddParameter("cert");

builder.AddProject<ProjectsMyProject_Web>(Applications.IdentityWeb)
    .PublishAsAzureContainerApp((module, app) =>
    {
        app.ConfigureCustomDomain(domainParam , certParam);
    });

This will be a two pass thing as you don't know the certificate id before it's created in Azure, so you'll only be able to set up a value for that after the first deploy.

In terms of configuring those param values in CI/CD, I'm not sure if you can use whatever form of config you want e.g. params, or if it needs to be in 'AZD_INITIAL_ENVIRONMENT_CONFIG' (where I have it), which is the current less than pretty way of passing infra params in the pipeline, for example in one of my build steps in azure devops:

          - task: AzureCLI@2
            displayName: Provision Infrastructure
            inputs:
              azureSubscription: '${{ parameters.azureConnection }}'
              keepAzSessionActive: true
              scriptType: 'pscore'
              scriptLocation: inlineScript
              inlineScript: |
                azd provision --no-prompt
            env:
              AZURE_SUBSCRIPTION_ID: $(Azure.SubscriptionId)
              AZURE_ENV_NAME: $(Azure.EnvName)
              AZURE_LOCATION: $(Azure.Location)
              AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

Where the value of that variable is a chunk of json squashed into a string in a variable library (bit yuk), e.g.

{
    "infra": {
        "parameters": {
            "customdomain": "my.example.com",
            "cert": "certificateId.once.you.have.it.from.azure",
        }
    }
}

I'm hoping there are nicer ways of doing these things in releases to come.

[gabsamples6]

@mip1983 thanks for your reply I have done that but did not make a change, also do you know if in the azure pipeline you can do _"azd config set alpha.aca.persistDomains on" I have added it but the domain do dissapear, but works in the cmd prompt.

many thanks for your time and help

[gabsamples6]

Just tried below in an azure pipeline and did not persist the domain - Any help anyone?

  - pwsh: |
      azd config set alpha.aca.persistDomains on
    displayName: Configure AZD to persist domain name configuration on container app environment.

[mip1983]

I just wanted to ask here if there is anything in the pipeline on maturing the custom domain story in Azure?

I've recently needed to deploy a new instance/container apps environment of my project (through Azure Devops pipelines), and the domain/cert part was a bit of a challenging part. It was a bit of a struggle to simply pass an empty string as the certificate parameter in the pipeline environment, I'd just hit on errors that the parameter was missing.

I eventually took my own copy of the 'ConfigureCustomDomain' extension and got it accept 'none' instead of an empty string as the literal it was testing for so it could do the first pass and create the domains without certificates.

On an instance where I have multiple domain bindings, I tried multiple calls to 'ConfigureCustomDomain' with separate parameters, but I just hit errors like this on publish:

info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 9.3.0+f76a033601ade4a17a422d5d1e4d004ab85e5179
fail: Aspire.Hosting.DistributedApplicationRunner[0]
      Failed to publish the distributed application.
      System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
         at System.Collections.Generic.List`1.Enumerator.MoveNext()
         at Aspire.Hosting.Publishing.ManifestPublishingContext.WriteModel(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/ManifestPublishingContext.cs:line 77
         at Aspire.Hosting.Publishing.ManifestPublisher.WriteManifestAsync(DistributedApplicationModel model, Utf8JsonWriter jsonWriter, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/ManifestPublisher.cs:line 67
         at Aspire.Hosting.Publishing.ManifestPublisher.PublishInternalAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/ManifestPublisher.cs:line 55
         at Aspire.Hosting.Publishing.ManifestPublisher.PublishAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/ManifestPublisher.cs:line 23
         at Aspire.Hosting.DistributedApplicationRunner.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/DistributedApplicationRunner.cs:line 46
fail: Microsoft.Extensions.Hosting.Internal.Host[11]

So I have to keep that out and remember to manually configure the extra bindings post deploy.

The ideal I imagine would be if you didn't need to pass this certificate parameter, instead just a configuration object that has a 'AzureManaged' flag (it could then be an eventual consistency thing, on the Azure side if it see's this flag and there isn't already a binding, it could go off and do it's thing post deploy). Not easy I'm sure as it would take changes on the Azure side I'd have thought, but avoiding this parameter and needing the second pass would be a headache removed. Better still would be an Azure managed DNS integration that also got configured.

I'm sure on the Aspire side you're keen to get this key bit of the puzzle in place as well, this has been hanging around as a pain point for a long while now, is there anything like the above on the horizon?

[vhvb1989]

Removing aspire tag. Custom domain is handled from the AppHost model. So this should not be affecting Aspire anymore