TerraformPluginSDKAsyncConnector goroutine may share context with reconciliation goroutine

[sjiekak]

What happened?

If https://github.com/hashicorp/terraform-provider-google is used in a terraform.SetupFn, and the context.Context of given to the terraform.SetupFn is used as is, then it is likely the goroutine effectively performing the requests async rely on a context from the reconciliation loop goroutine. I believe this is the source of issues https://github.com/crossplane-contrib/provider-upjet-gcp/issues/538 or https://github.com/crossplane-contrib/provider-upjet-gcp/issues/611

• whichever context is given to terraform.SetupFn is stored in the meta field of terraform.Setup resulting from the configuration of the terraform provider
• terraform.SetupFn is called with goroutine Controller Context to synchronously to connect, ie during each reconciliation. The result is stored to be reused
• the result terraform.Setup is later reused in asynchronous function create , Update or Delete, with a total async timeout of one hour. When used, it relies on the stored context from goroutine Controller Context for all API calls to gcp.
• Currently, projects such as https://github.com/crossplane-contrib/provider-upjet-gcp need to strip any cancellation from the context otherwise resources may never be created

How can we reproduce it?

It is more likely to appear when the resource creation takes enough time for the reconciliation context to expire. Which I believe is 3 minutes. It has been reproduced with gcp project operations

This behavior can be reproduced with an example such as

type example struct {
	ctx context.Context
}

func (e *example) buildSetupFn() terraform.SetupFn {
	return func(ctx context.Context, client client.Client, mg resource.Managed) (terraform.Setup, error) {
		e.ctx = ctx
		return terraform.Setup{
			Meta: e,
		}, nil
	}
}

func (e *example) resource() *config.Resource {
	return &config.Resource{
		TerraformResource: &schema.Resource{
			Create: func(rd *schema.ResourceData, i interface{}) error {
				retrieved := i.(*example)
				select {
				case <-retrieved.ctx.Done():
					return fmt.Errorf("done")
				default:
					return nil
				}
			},
                       // more fields...
          }
}

[sjiekak]

Suggestion: TerraformPluginSDKAsyncConnector should connect with its own context