Unable to add search attribute when upgrading from v1.19 to v1.22

[ericvg97]

When use our own temporal cluster hosted in Aurora and when running temporal operator search-attribute create --name ReverStatus --type Keyword we get this error unable to add search attributes: Unable to create search attributes: cannot have more than 0 search attribute of type Keyword.

We just upgraded from v1.19 to v1.22.

We compared the db with the one in a fresh install and we found that the cluster_metadata_info data (binary) is quite different. In the fresh install, there are mentions to Custom Attributes, but in our production version there aren't. We don't know if this means we had a problem when upgrading.

cluster_metadata_info_fresh_install.txt cluster_metadata_info_prod.txt

Other relevant info:

1. When running temporal operator cluster describe -o json we get

"persistence_store": "postgres12",
"visibility_store": "postgres12"

2. With tctl we get the same issue
3. Our config in /etc/temporal/config/docker.yaml is

log:
    stdout: true
    level: info

persistence:
    numHistoryShards: 64
    defaultStore: default
    visibilityStore: visibility
    datastores:
        default:
            sql:
                pluginName: "postgres12"
                databaseName: "temporal"
                connectAddr: "XXXX"
                connectProtocol: "tcp"
                user: "XXXX"
                password: "XXXX"
                maxConns: 20
                maxIdleConns: 20
                maxConnLifetime: 1h
                tls:
                    enabled: false
                    caFile:
                    certFile:
                    keyFile:
                    enableHostVerification: false
                    serverName:
        visibility:
            sql:
                pluginName: "postgres12"
                databaseName: "temporal_visibility"
                connectAddr: "XXXX"
                connectProtocol: "tcp"
                user: "XXXX"
                password: "XXXX"
                maxConns: 10
                maxIdleConns: 10
                maxConnLifetime: 1h
                tls:
                    enabled: false
                    caFile:
                    certFile:
                    keyFile:
                    enableHostVerification: false
                    serverName:

global:
    membership:
        maxJoinDuration: 30s
        broadcastAddress: ""
    pprof:
        port: 0
    tls:
        refreshInterval: 0s
        expirationChecks:
            warningWindow: 0s
            errorWindow: 0s
            checkInterval: 0s
        internode:
            # This server section configures the TLS certificate that internal temporal
            # cluster nodes (history, matching, and internal-frontend) present to other
            # clients within the Temporal Cluster.
            server:
                requireClientAuth: false
                certFile:
                keyFile:
                certData:
                keyData:

            # This client section is used to configure the TLS clients within
            # the Temporal Cluster that connect to an Internode (history, matching, or
            # internal-frontend)
            client:
                serverName:
                disableHostVerification: false
        frontend:
            # This server section configures the TLS certificate that the Frontend
            # server presents to external clients.
            server:
                requireClientAuth: false
                certFile:
                keyFile:
                certData:
                keyData:

            # This client section is used to configure the TLS clients within
            # the Temporal Cluster (specifically the Worker role) that connect to the Frontend service
            client:
                serverName:
                disableHostVerification: false
    authorization:
        jwtKeyProvider:
            keySourceURIs:
            refreshInterval: 1m
        permissionsClaimName: permissions
        authorizer:
        claimMapper:
services:
    frontend:
        rpc:
            grpcPort: 7233
            membershipPort: 6933
            bindOnIP: XXXX
            httpPort: 7243

    matching:
        rpc:
            grpcPort: 7235
            membershipPort: 6935
            bindOnIP: XXXX

    history:
        rpc:
            grpcPort: 7234
            membershipPort: 6934
            bindOnIP: XXXX

    worker:
        rpc:
            grpcPort: 7239
            membershipPort: 6939
            bindOnIP: XXXX

clusterMetadata:
    enableGlobalNamespace: false
    failoverVersionIncrement: 10
    masterClusterName: "active"
    currentClusterName: "active"
    clusterInformation:
        active:
            enabled: true
            initialFailoverVersion: 1
            rpcName: "frontend"
            rpcAddress: XXXX

dcRedirectionPolicy:
    policy: "noop"

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"
  visibility:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"

namespaceDefaults:
  archival:
    history:
      state: "disabled"
      URI: "file:///tmp/temporal_archival/development"
    visibility:
      state: "disabled"
      URI: "file:///tmp/temporal_vis_archival/development"


dynamicConfigClient:
    filepath: "/etc/temporal/config/dynamicconfig/docker.yaml"
    pollInterval: "60s"

[rodrigozhou]

How did you do the upgrade? Here's the general guide about how to do upgrades: https://docs.temporal.io/cluster-deployment-guide#upgrade-server

Also, you want to take a look this release notes: https://github.com/temporalio/temporal/releases/tag/v1.20.0#:~:text=Advanced%20visibility%20on%20SQL%20DB%20(beta%20testing)

[jaredLunde]

This happens with Aurora PostgreSQL clusters even with fresh Temporal installs. Here's another occurrence from the Community Support forum: https://community.temporal.io/t/unable-to-add-search-attributes/9946

As with the OP in the forum post, I was able to fix it by using pure Postgres.