oracle-database-operator icon indicating copy to clipboard operation
oracle-database-operator copied to clipboard

OraOperator v1.2.0 automatic failover issues when reinstating new standby

Open andbos opened this issue 7 months ago • 2 comments

I'm trying out the new Automatic failover feature. Could you perhaps suggest a good way of testing it? I deleted the primary SIDB pod and this triggered automatic failover alright but reinstation of the new standby fails with error:

2025-07-03T13:04:37.906+00:00
Initiating reinstatement for database "ncnfdb11"...
Reinstating database "ncnfdb11", please wait...
Error: ORA-16653: failed to reinstate database

Failed.
Reinstatement of database "ncnfdb11" failed
2025-07-03T13:04:37.965+00:00
[W000 2025-07-03T13:04:38.911+00:00] Failed to reinstate database ncnfdb11. Will retry later.
[W000 2025-07-03T13:04:54.936+00:00] Try to connect to the new standby ncnfdb11.
[W000 2025-07-03T13:04:55.936+00:00] Connection to the new standby restored!
[W000 2025-07-03T13:04:57.941+00:00] Try to connect to the primary ncnfdb12:1521/ncnfdb12.
[W000 2025-07-03T13:04:58.942+00:00] Connection to the primary restored!
[W000 2025-07-03T13:04:58.942+00:00] Wait for new primary to be ready to reinstate.
[W000 2025-07-03T13:04:58.946+00:00] New primary is now ready to reinstate.
[W000 2025-07-03T13:04:59.946+00:00] Issuing REINSTATE command.

2025-07-03T13:04:59.946+00:00
Initiating reinstatement for database "ncnfdb11"...
Reinstating database "ncnfdb11", please wait...
Error: ORA-16653: failed to reinstate database

Failed.
Reinstatement of database "ncnfdb11" failed
2025-07-03T13:05:00.017+00:00
[W000 2025-07-03T13:05:00.946+00:00] Failed to reinstate database ncnfdb11. Will retry later.
$ kubectl -n oracle-database get pods
NAME                         READY   STATUS    RESTARTS   AGE
cnfdb-dgbroker-rb0po   0/1     Running   0          23m
cnfdb11-ck99a          0/1     Running   0          8m41s
cnfdb12-e8a8l          1/1     Running   0          45m
$ kubectl -n oracle-database logs -f cnfdb11-ck99a
SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted


SQL>    alter database open
*
ERROR at line 1:
ORA-16649: possible failover to another database prevents this database from
being opened


SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 10 sec(s) before restarting Oracle processes and opening the database
The Oracle base remains unchanged with value /opt/oracle
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
   alter database mount
ORA-1100 signalled during:    alter database mount...
   alter database open
2025-07-03T13:00:34.231335+00:00
Data Guard Broker initializing...
Data Guard Broker initialization complete
Data Guard: verifying database primary role...
Data Guard: broker startup completed
Data Guard determines a failover has occurred - instance will not be opened
ORA-16649 signalled during:    alter database open...

andbos avatar Jul 03 '25 16:07 andbos