Oracle Data Guard per Pluggable Database hands-on

Posted on , updated on by Yannick Jaquier
Oracle ,

Table of contents

Preamble

Oracle Data Guard per Pluggable Database has been made available since 21c (21.7). In a pluggable configuration the traditional configuration was below one with Data Guard working at container level:

dgcdb
dgcdb

Means that all pluggable database are Data Guard replicated to the target container. Any switchover and failover, at container level, will be for the all pluggable database at same time.

The new feature allow you to configure Data Guard per pluggable database and so give the choice to protect only a part of your pluggable databases in a bi-directionnal way as each container database is now a primary. It also bring more granularity in switching or failover only one pluggable database at a time (do you really break only one pluggable in real life ?):

dgpdb
dgpdb

In 21c the feature comes with a long list of restrictions, refer to Ludovico’s post for the complete list. The one that would have been interesting in first shot is the active Data Guard one as well as the capability to change the protection mode (MaxPerformance only for now). If Oracle invest in this feature 23ai will probably brings its bunch of removed limitations…

If you try to change the protection mode, even changing the redo transport is not *really* possible:

DGMGRL> edit DATABASE orcl2 SET property 'LogXptMode'='SYNC';
Property "LogXptMode" updated
DGMGRL> show DATABASE orcl2 LogXptMode
  LogXptMode = 'SYNC'
DGMGRL> show DATABASE verbose orcl2
 
DATABASE - orcl2
 
  ROLE:                PRIMARY
  Intended State:      TRANSPORT-ON
  PDB Data Guard ROLE: TARGET
  Data Guard Target PDB(s): 1
  Instance(s):
    orcl2
 
  Properties:
    DGConnectIdentifier             = 'orcl2'
    ObserverConnectIdentifier       = ''
    FastStartFailoverTarget         = ''
    PreferredObserverHosts          = ''
    LogShipping                     = 'ON'
    RedoRoutes                      = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ArchiveLocation                 = ''
    AlternateLocation               = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    LogXptStatus                    = '(monitor)'
    SendQEntries                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'server01'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server01.domain.com)(PORT=1541))(CONNECT_DATA=(SERVICE_NAME=orcl2_DGMGRL)(INSTANCE_NAME=orcl2)(SERVER=DEDICATED)))'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'
 
  LOG FILE locations:
    Alert LOG               : /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/alert_orcl2.LOG
    Data Guard Broker LOG   : /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/drcorcl2.LOG
 
DATABASE Status:
SUCCESS

Then going to MAXAVAILABILITY is generating errors…

Testing has been done with Oracle Database 21c Enterprise Edition Release 21.15 (latest available RU at the time of writing this blog post). My two Oracle database containers (orcl1 and orcl2) are running on the same physical server that has Red Hat Enterprise Linux release 8.7 (Ootpa) with 12 cores and 64GB of memory.

Databases Data Guard preparation

Container databases preparation

To prepare a database for Data Guard the usal steps are:

  • Activate ARCHIVELOG and FOR LOGGING. Ofter you will set the setup of Fast Recovery Area (FRA) to store the archived loig files
  • Add the standby redo log file with alter database add standby logfile command
  • Enable flashback database with alter database flashback on command
  • Set standby_file_management to auto and set dg_broker_start to true

Starting with 21c this can be done in one single command with Data Guard Broker (dgmgrl). So on each primary CDB issue the below command, customize it to change the database name and Data Guard Broker configuration files:

[oracle@server01 ~]$ dgmgrl / as sysdba
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 11:39:31 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> prepare database for data guard
        with db_unique_name is orcl1
        db_recovery_file_dest is /u01/fra
        db_recovery_file_dest_size is 5g
       broker_config_file_1 is '/u01/app/oracle/dbs/dgmgrl1_orcl1.dat'
       broker_config_file_2 is '/u01/app/oracle/dbs/dgmgrl2_orcl1.dat';
Preparing database "orcl1" for Data Guard.
Initialization parameter DB_FILES set to 1024.
Initialization parameter LOG_BUFFER set to 268435456.
Primary database must be restarted after setting static initialization parameters.
Primary database must be restarted to enable archivelog mode.
Shutting down database "orcl1".
Database closed.
Database dismounted.
ORACLE instance shut down.
Starting database "orcl1" to mounted mode.
ORACLE instance started.
Database mounted.
Initialization parameter DB_FLASHBACK_RETENTION_TARGET set to 120.
Initialization parameter DB_LOST_WRITE_PROTECT set to 'TYPICAL'.
RMAN configuration archivelog deletion policy set to SHIPPED TO ALL STANDBY.
Initialization parameter DB_RECOVERY_FILE_DEST_SIZE set to '5g'.
Initialization parameter DB_RECOVERY_FILE_DEST set to '/u01/fra'.
Initialization parameter DG_BROKER_CONFIG_FILE1 set to '/u01/app/oracle/dbs/dgmgrl1_orcl1.dat'.
Initialization parameter DG_BROKER_CONFIG_FILE2 set to '/u01/app/oracle/dbs/dgmgrl2_orcl1.dat'.
LOG_ARCHIVE_DEST_n initialization parameter already set for local archival.
Initialization parameter LOG_ARCHIVE_DEST_2 set to 'location=use_db_recovery_file_dest valid_for=(all_logfiles, all_roles)'.
Initialization parameter LOG_ARCHIVE_DEST_STATE_2 set to 'Enable'.
Adding standby log group size 209715200 and assigning it to thread 1.
Adding standby log group size 209715200 and assigning it to thread 1.
Adding standby log group size 209715200 and assigning it to thread 1.
Initialization parameter STANDBY_FILE_MANAGEMENT set to 'AUTO'.
Initialization parameter DG_BROKER_START set to TRUE.
Database set to ARCHIVELOG.
Database set to FLASHBACK ON.
Database opened.

This has automatically added the standby redo log file in the FRA:

SQL> SET lines 200 pages 1000
SQL> col member FOR a50
SQL> SELECT * FROM v$logfile;
 
    GROUP# STATUS  TYPE    MEMBER                                             IS_     CON_ID
---------- ------- ------- -------------------------------------------------- --- ----------
         3         ONLINE  /u01/app/oracle/oradata/ORCL1/redo03.LOG           NO           0
         2         ONLINE  /u01/app/oracle/oradata/ORCL1/redo02.LOG           NO           0
         1         ONLINE  /u01/app/oracle/oradata/ORCL1/redo01.LOG           NO           0
         4         STANDBY /u01/fra/ORCL1/onlinelog/o1_mf_4_mc4dqsdl_.LOG     YES          0
         5         STANDBY /u01/fra/ORCL1/onlinelog/o1_mf_5_mc4dqt4k_.LOG     YES          0
         6         STANDBY /u01/fra/ORCL1/onlinelog/o1_mf_6_mc4dqtsh_.LOG     YES          0
 
6 ROWS selected.

On orcl2 I have issued:

prepare DATABASE FOR data guard
WITH db_unique_name IS orcl2
db_recovery_file_dest IS /u01/fra
db_recovery_file_dest_size IS 5g
broker_config_file_1 IS '/u01/app/oracle/dbs/dgmgrl1_orcl2.dat'
broker_config_file_2 IS '/u01/app/oracle/dbs/dgmgrl2_orcl2.dat';

Change db_file_name_convert and pdb_file_name_convert to have datafiles automatically renamed:

SQL> ALTER SYSTEM SET db_file_name_convert = '/u01/app/oracle/oradata/ORCL1','/u01/app/oracle/oradata/ORCL2' scope=both;
 
SYSTEM altered.
 
SQL> ALTER SYSTEM SET pdb_file_name_convert = '/u01/app/oracle/oradata/ORCL1','/u01/app/oracle/oradata/ORCL2' scope=both;
 
SYSTEM altered.
 
SQL> show parameter db_file_name_convert
 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_file_name_convert                 string      /u01/app/oracle/oradata/ORCL1,
                                                  /u01/app/oracle/oradata/ORCL2
pdb_file_name_convert                string      /u01/app/oracle/oradata/ORCL1,
                                                  /u01/app/oracle/oradata/ORCL2

On orcl2 this is the opposite string replacement:

ALTER SYSTEM SET db_file_name_convert = '/u01/app/oracle/oradata/ORCL2','/u01/app/oracle/oradata/ORCL1' scope=both;
ALTER SYSTEM SET pdb_file_name_convert = '/u01/app/oracle/oradata/ORCL2','/u01/app/oracle/oradata/ORCL1' scope=both;

Establish a passwordless connection by creating a wallet with mkstore command line:

[oracle@server01 ~]$ mkdir /u01/app/oracle/admin/orcl1/wallet
[oracle@server01 ~]$ mkstore -wrl $ORACLE_BASE/admin/orcl1/wallet/dgpdb -create
Oracle Secret Store Tool Release 21.0.0.0.0 - Production
Version 21.0.0.0.0
Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
 
Enter password:
Enter password again:

Add in this wallet the credentials for SYS account on both databases:

[oracle@server01 ~]$ mkstore -wrl $ORACLE_BASE/admin/orcl1/wallet/dgpdb -createCredential orcl1 'sys'
Oracle Secret Store Tool Release 21.0.0.0.0 - Production
Version 21.0.0.0.0
Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
 
Your secret/Password is missing in the command line
Enter your secret/Password:
Re-enter your secret/Password:
Enter wallet password:
[oracle@server01 ~]$ mkstore -wrl $ORACLE_BASE/admin/orcl1/wallet/dgpdb -createCredential orcl2 'sys'
Oracle Secret Store Tool Release 21.0.0.0.0 - Production
Version 21.0.0.0.0
Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
 
Your secret/Password is missing in the command line
Enter your secret/Password:
Re-enter your secret/Password:
Enter wallet password:
[oracle@server01 ~]$ mkstore -wrl $ORACLE_BASE/admin/orcl1/wallet/dgpdb -listCredential
Oracle Secret Store Tool Release 21.0.0.0.0 - Production
Version 21.0.0.0.0
Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
 
Enter wallet password:
List credential (index: connect_string username)
2: orcl2 sys
1: orcl1 sys

Modify sqlnet.ora file by adding below content to take into account your newly created wallet (file location is for me /u01/app/oracle/homes/OraDB21Home1/network/admin):

WALLET_LOCATION =
    (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
        (DIRECTORY = /u01/app/oracle/admin/orcl1/wallet/dgpdb)
    )
)
SQLNET.WALLET_OVERRIDE = TRUE

Copy the wallet to your other node and modify accordingly the sqlnet.ora file… Last reload the listener on all nodes with lsnrctl reload command…

Ensure all is working fine with:

[oracle@server01 ~]$ sqlplus /@orcl1 as sysdba
 
SQL*Plus: Release 21.0.0.0.0 - Production on Fri Aug 9 11:46:30 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Last Successful login time: Fri Aug 09 2024 11:27:35 +02:00
 
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.15.0.0.0
 
SQL> Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.15.0.0.0
[oracle@server01 ~]$ sqlplus /@orcl2 as sysdba
 
SQL*Plus: Release 21.0.0.0.0 - Production on Fri Aug 9 11:46:35 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Last Successful login time: Fri Aug 09 2024 11:27:41 +02:00
 
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.15.0.0.0
 
SQL>

Create a new pluggable database for replication

To start fresh drop and recreate a new pluggable database on first instance (orcl1) with:

SQL> ALTER pluggable DATABASE pdb1 CLOSE IMMEDIATE;
 
Pluggable DATABASE altered.
 
SQL> DROP pluggable DATABASE pdb1 including datafiles;
 
Pluggable DATABASE dropped.
 
SQL> CREATE pluggable DATABASE pdb1
     admin USER yjaquier IDENTIFIED BY secure_password
     roles = (dba)
     DEFAULT TABLESPACE users datafile '/u01/app/oracle/oradata/ORCL1/pdb1/users01.dbf' SIZE 5m autoextend ON next 5m maxsize 100m
     file_name_convert = ('/u01/app/oracle/oradata/ORCL1/pdbseed/', '/u01/app/oracle/oradata/ORCL1/pdb1/')
     path_prefix = '/u01/app/oracle/oradata/ORCL1/pdb1/';
 
Pluggable DATABASE created.
 
SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDB1                           MOUNTED
SQL> ALTER pluggable DATABASE pdb1 OPEN read WRITE;
 
Pluggable DATABASE altered.
 
SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDB1                           READ WRITE NO
SQL> ALTER SESSION SET container=pdb1;
 
SESSION altered.
 
SQL> GRANT unlimited TABLESPACE TO yjaquier;
 
GRANT succeeded.

Data Guard Broker configuration

Data Guard Broker configuration creation

Create Data Guard Broker configuration:

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Thu Aug 8 12:50:44 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> create configuration "orcl1" as primary database is "orcl1" connect identifier is "orcl1";
Connected to "orcl1"
Configuration "orcl1" created with primary database "orcl1"
DGMGRL> show configuration verbose;
 
Configuration - orcl1
 
  Protection Mode: MaxPerformance
  Members:
  orcl1 - Primary database
 
  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    TraceLevel                      = 'USER'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    ObserverPingInterval            = '0'
    ObserverPingRetry               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'
    ConfigurationWideServiceName    = 'orcl1_CFG'
    ConfigurationSimpleName         = 'orcl1'
    DrainTimeout                    = '0'
 
Fast-Start Failover:  Disabled
 
Configuration Status:
DISABLED

Repeat same operation on target container instance (orcl2), obviously replacing orcl1 by orcl2 !

Add the orcl2 target instance on orcl1 Data Guard Broker configuration:

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 11:52:36 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> add configuration orcl2 connect identifier is "orcl2";
Configuration orcl2 added.
DGMGRL> enable configuration all;
Succeeded.
DGMGRL> show configuration verbose;
 
Configuration - orcl1
 
  Protection Mode: MaxPerformance
  Members:
  orcl1 - Primary database
  orcl2 - Primary database in orcl2 configuration
 
  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    TraceLevel                      = 'USER'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    ObserverPingInterval            = '0'
    ObserverPingRetry               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'
    ConfigurationWideServiceName    = 'orcl1_CFG'
    ConfigurationSimpleName         = 'orcl1'
    DrainTimeout                    = '0'
 
Fast-Start Failover:  Disabled
 
Configuration Status:
SUCCESS

Remark:
This has automatically enable Data Guard Broker configuration on target orcl2 instance.

Prepare the databases for Pluggable Data Guard on source orcl1 instance:

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 11:54:40 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> edit configuration prepare dgpdb;
Enter password for DGPDB_INT account at orcl1:
Enter password for DGPDB_INT account at orcl2:
 
Prepared Data Guard for Pluggable Database at orcl2.
 
Prepared Data Guard for Pluggable Database at orcl1.

Remark:
This is creating the DBPDB_INT account at container level with PPLB_ROLE role for Oracle Data Guard protection or change roles for a PDB.

Add the pluggable database to your configuration. On target orcl2 container instance ensure no pdb1 pluggable database:

SQL> show pdbs
 
CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED                       READ ONLY  NO

Add pdb1 pluggable database to your Data Guard Broker configuration:

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 12:46:52 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> add pluggable database pdb1 at orcl2 source is pdb1 at orcl1 pdbfilenameconvert is "'ORCL1','ORCL2'";
Connected to "orcl2"
Connected to "orcl1"
Connected to "orcl2"
 
Pluggable Database "PDB1" added

Pdb1 pluggable database created in MOUNT mode in orcl2 container instance:

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED

Copy of source pluggable database datafiles

The newly created pdb1 pluggable database on orcl2 target container instance has not yet any datafiles:

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED
SQL> !ls -l /u01/app/oracle/oradata/ORCL2/pdb1
total 0

Now the goal is to copy this pluggable database datafiles to my orcl2 target container instance. The easiest way to achieve this goal is most probably RMAN but you can also use BEGIN BACKUP and the cp/scp commands.

[oracle@server01 ~]$ rman target /@orcl1 auxiliary /@orcl2
 
Recovery Manager: Release 21.0.0.0.0 - Production on Fri Aug 9 14:33:00 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
connected to target database: ORCL1 (DBID=1495184713)
connected to auxiliary database: ORCL2 (DBID=1157121117)
 
RMAN> run {
     allocate channel ch_pri type disk;
     allocate auxiliary channel ch_aux type disk;
     backup as copy pluggable database pdb1 auxiliary format '/u01/app/oracle/oradata/ORCL2/pdb1/%b';
     }
 
allocated channel: ch_pri
channel ch_pri: SID=208 device type=DISK
 
allocated channel: ch_aux
channel ch_aux: SID=600 device type=DISK
 
Starting backup at 08-AUG-24
channel ch_pri: starting datafile copy
input datafile file number=00013 name=/u01/app/oracle/oradata/ORCL1/pdb1/system01.dbf
output file name=/u01/app/oracle/oradata/ORCL2/pdb1/system01.dbf tag=TAG20240808T155620
channel ch_pri: datafile copy complete, elapsed time: 00:00:07
channel ch_pri: starting datafile copy
input datafile file number=00014 name=/u01/app/oracle/oradata/ORCL1/pdb1/sysaux01.dbf
output file name=/u01/app/oracle/oradata/ORCL2/pdb1/sysaux01.dbf tag=TAG20240808T155620
channel ch_pri: datafile copy complete, elapsed time: 00:00:07
channel ch_pri: starting datafile copy
input datafile file number=00015 name=/u01/app/oracle/oradata/ORCL1/pdb1/undotbs01.dbf
output file name=/u01/app/oracle/oradata/ORCL2/pdb1/undotbs01.dbf tag=TAG20240808T155620
channel ch_pri: datafile copy complete, elapsed time: 00:00:03
channel ch_pri: starting datafile copy
input datafile file number=00016 name=/u01/app/oracle/oradata/ORCL1/pdb1/useres01.dbf
output file name=/u01/app/oracle/oradata/ORCL2/pdb1/useres01.dbf tag=TAG20240808T155620
channel ch_pri: datafile copy complete, elapsed time: 00:00:01
Finished backup at 08-AUG-24
released channel: ch_pri
released channel: ch_aux

Change the State of Standby to APPLY-ON:

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Thu Aug 8 16:02:17 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> edit pluggable database pdb1 at orcl2 set state='apply-off';
Connected to "orcl2"
Succeeded.
DGMGRL> edit pluggable database pdb1 at orcl2 set state='apply-on';
Connected to "orcl2"
Succeeded.
DGMGRL> enable configuration all;
Succeeded.
DGMGRL> show pluggable database pdb1 at orcl2;
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              con_id 4 at orcl1
  Transport Lag:       (unknown)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      orcl2
  Average Apply Rate:  (unknown)
  Real Time Query:     OFF
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> show pluggable database pdb1 at orcl1;
 
Pluggable database 'pdb1' at database 'orcl1'
 
  Data Guard Role:     Primary
  Con_ID:              4
  Active Target:       con_id 3 at orcl2
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> sql 'alter system archive log current';
Succeeded.
DGMGRL> sql 'alter system archive log current';
Succeeded.
DGMGRL> show pluggable database pdb1 at orcl2;
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              con_id 3 at orcl1
  Transport Lag:       0 seconds (computed 10 seconds ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      orcl2
  Average Apply Rate:  3244 KByte/s
  Real Time Query:     OFF
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> show database verbose orcl1
 
Database - orcl1
 
  Role:                PRIMARY
  Intended State:      TRANSPORT-ON
  PDB Data Guard Role: SOURCE
  Data Guard Source PDB(s): 1
  Instance(s):
    orcl1
 
  Properties:
    DGConnectIdentifier             = 'orcl1'
    ObserverConnectIdentifier       = ''
    FastStartFailoverTarget         = ''
    PreferredObserverHosts          = ''
    LogShipping                     = 'ON'
    RedoRoutes                      = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ArchiveLocation                 = ''
    AlternateLocation               = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    LogXptStatus                    = '(monitor)'
    SendQEntries                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'server01'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server01.domain.com)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=orcl1_DGMGRL)(INSTANCE_NAME=orcl1)(SERVER=DEDICATED)))'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'
 
  Log file locations:
    Alert log               : /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/alert_orcl1.log
    Data Guard Broker log   : /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/drcorcl1.log
 
Database Status:
SUCCESS
 
DGMGRL> show database verbose orcl2
 
Database - orcl2
 
  Role:                PRIMARY
  Intended State:      TRANSPORT-ON
  PDB Data Guard Role: TARGET
  Data Guard Target PDB(s): 1
  Instance(s):
    orcl2
 
  Properties:
    DGConnectIdentifier             = 'orcl2'
    ObserverConnectIdentifier       = ''
    FastStartFailoverTarget         = ''
    PreferredObserverHosts          = ''
    LogShipping                     = 'ON'
    RedoRoutes                      = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ArchiveLocation                 = ''
    AlternateLocation               = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    LogXptStatus                    = '(monitor)'
    SendQEntries                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'server01'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server01.domain.com)(PORT=1541))(CONNECT_DATA=(SERVICE_NAME=orcl2_DGMGRL)(INSTANCE_NAME=orcl2)(SERVER=DEDICATED)))'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'
 
  Log file locations:
    Alert log               : /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/alert_orcl2.log
    Data Guard Broker log   : /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/drcorcl2.log
 
Database Status:
SUCCESS

Pluggable Data Guard testing

Preparation

To see if data is replicated I create a test table with one row in my account in pdb1 in current primary orcl1 container instance:

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
SQL> ALTER SESSION SET container=pdb1;
 
SESSION altered.
 
SQL> CREATE TABLE yjaquier.test01(id int, descr VARCHAR2(20)) TABLESPACE users;
 
TABLE created.
 
SQL> INSERT INTO yjaquier.test01 VALUES(1, 'One');
 
1 ROW created.
 
SQL> COMMIT;
 
COMMIT complete.

Switchover

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 15:16:18 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> switchover to pluggable database pdb1 at orcl2
Verifying conditions for Switchover...
 
Connected to "orcl2"
Connected to "orcl1"
  Source pluggable database is 'PDB1' at database 'orcl1'
 
Performing switchover NOW, please wait...
 
  Closing pluggable database 'PDB1'...
  Switching 'PDB1' to standby role...
Connected to "orcl2"
  Waiting for 'PDB1' to recover all redo data...
  Stopping recovery at 'PDB1'...
  Converting 'PDB1' to primary role...
  Opening new primary 'PDB1'...
Connected to "orcl1"
  Waiting for redo data from new primary 'PDB1'...
  Starting recovery at new standby 'PDB1'...
 
Switchover succeeded, new primary is "PDB1"

We confirm that the role has been switch. The pdb1 running on orcl2 is primary while the one running on orcl1 is new a physical standby:

DGMGRL> show pluggable database pdb1 at orcl2;
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       con_id 3 at orcl1
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> show pluggable database pdb1 at orcl1;
 
Pluggable database 'pdb1' at database 'orcl1'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              con_id 3 at orcl2
  Transport Lag:       0 seconds (computed 47 seconds ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      orcl1
  Average Apply Rate:  (unknown)
  Real Time Query:     OFF
 
Pluggable Database Status:
SUCCESS

On target orcl2 instance I can now connect to pdb1 that is open in read write and found back my test table:

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
SQL> ALTER SESSION SET container=pdb1;
 
SESSION altered.
 
SQL> SELECT * FROM yjaquier.test01;
 
        ID DESCR
---------- --------------------
         1 One

Let’s insert a new row in my test table:

SQL> INSERT INTO yjaquier.test01 VALUES(2, 'Two');
 
1 ROW created.
 
SQL> COMMIT;
 
COMMIT complete.

And if we switch back to initial state:

DGMGRL> switchover to pluggable database pdb1 at orcl1
Verifying conditions for Switchover...
 
Connected to "orcl2"
  Source pluggable database is 'PDB1' at database 'orcl2'
 
Performing switchover NOW, please wait...
 
  Closing pluggable database 'PDB1'...
  Switching 'PDB1' to standby role...
Connected to "orcl1"
  Waiting for 'PDB1' to recover all redo data...
  Stopping recovery at 'PDB1'...
  Converting 'PDB1' to primary role...
  Opening new primary 'PDB1'...
Connected to "orcl2"
  Waiting for redo data from new primary 'PDB1'...
  Starting recovery at new standby 'PDB1'...
 
Switchover succeeded, new primary is "PDB1"

We find the row recently inserted in pdb1 now running on orcl1 container instance:

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
SQL> ALTER SESSION SET container=pdb1;
 
SESSION altered.
 
SQL> SELECT * FROM yjaquier.test01;
 
        ID DESCR
---------- --------------------
         1 One
         2 Two

Failover

Let’s submit a failover:

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 16:32:55 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> failover to pluggable database pdb1 at orcl2
Verifying conditions for Failover...
 
Connected to "orcl2"
Connected to "orcl1"
  Source pluggable database is 'PDB1' at database 'orcl1'
 
Performing failover NOW, please wait...
 
  Closing pluggable database 'PDB1'...
  Converting 'PDB1' to standby role...
Connected to "orcl2"
  Waiting for 'PDB1' to recover all redo data...
  Stopping recovery at 'PDB1'...
  Converting 'PDB1' to primary role...
  Opening new primary 'PDB1'...
 
Failover succeeded, new primary is "PDB1".
DGMGRL> show pluggable database pdb1 at orcl2
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       None Available
 
Pluggable Database Status:
ORA-16912: target pluggable database may have been removed or a role change may have occurred
 
DGMGRL> show pluggable database pdb1 at orcl1
 
Pluggable database 'pdb1' at database 'orcl1'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              (unknown)
 
Pluggable Database Status:
ORA-16911: Redo data from new source database has not been registered.

At this point you would normally be obliged to completely re-instantiate the pdb1 pluggable database on former source orcl1 container instance. But with a favorable situation (no activity on pdb1 pluggable database) you can set everything back to work:

DGMGRL> edit pluggable database pdb1 at orcl1 set state='apply-off';
Succeeded.
DGMGRL> edit pluggable database pdb1 at orcl1 set state='apply-on';
Succeeded.
DGMGRL> show pluggable database pdb1 at orcl1
 
Pluggable database 'pdb1' at database 'orcl1'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              con_id 3 at orcl2
  Transport Lag:       0 seconds (computed 57 seconds ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      orcl1
  Average Apply Rate:  (unknown)
  Real Time Query:     OFF
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> show pluggable database pdb1 at orcl2
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       con_id 3 at orcl1
 
Pluggable Database Status:
SUCCESS

Pluggable Data Guard cleaning

[oracle@server01 ~]$ dgmgrl /@orcl1
DGMGRL for Linux: Release 21.0.0.0.0 - Production on Fri Aug 9 16:57:32 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
Welcome to DGMGRL, type "help" for information.
Connected to "orcl1"
Connected as SYSDBA.
DGMGRL> show pluggable database pdb1 at orcl2
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       con_id 3 at orcl1
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> remove pluggable database pdb1 at orcl1 remove datafiles;
 
Pluggable Database 'PDB1' removed.
 
No more standby pluggable databases; stopping redo transport services at source database orcl2...
Connected to "orcl2"
 
Succeeded.
DGMGRL> show pluggable database pdb1 at orcl1
 
Pluggable database 'pdb1' at database 'orcl1'
 
  No such pluggable database 'pdb1' at container database 'orcl1'
 
Failed.
DGMGRL> show configuration
 
Configuration - orcl1
 
  Protection Mode: MaxPerformance
  Members:
  orcl1 - Primary database
    orcl2 - Primary database in orcl2 configuration
 
Fast-Start Failover:  Disabled
 
Configuration Status:
SUCCESS   (status updated 50 seconds ago)
 
DGMGRL> remove configuration orcl1
ORA-16860: cannot remove a configuration that is referenced by another configuration
DGMGRL> remove configuration orcl2
Succeeded.
DGMGRL> remove configuration orcl1
Succeeded.
DGMGRL> show configuration
ORA-16532: Oracle Data Guard broker configuration does not exist
 
Configuration details cannot be determined by DGMGRL

Errors encountered

ORA-00922: missing or invalid option wth edit configuration prepare dgpdb command

When you specify a DGPDB_INT with special character like ! or * like the one I have chosen to have strong password you get:

DGMGRL> edit configuration prepare dgpdb;
Enter password for DGPDB_INT account at orcl1:
Enter password for DGPDB_INT account at orcl2:
ORA-00922: missing or invalid option

Simply re-issue the command specifying password with NO SPECIAL CHARACTERS… Secure ? Most probably a bug…

_DGMGRL legacy service name

You have most probably noticed the service name used to connect to the database i.e. orcl1_DGMGRL when issuing show database verbose command:

DGMGRL> show database verbose orcl1
 
Database - orcl1
 
  Role:                PRIMARY
  Intended State:      TRANSPORT-ON
  Redo Rate:           361 Byte/s  in 15 seconds (computed 4 seconds ago)
  PDB Data Guard Role: TARGET
  Data Guard Target PDB(s): 1
  Instance(s):
    orcl1
 
  Properties:
    DGConnectIdentifier             = 'orcl1'
    ObserverConnectIdentifier       = ''
    FastStartFailoverTarget         = ''
    PreferredObserverHosts          = ''
    LogShipping                     = 'ON'
    RedoRoutes                      = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ArchiveLocation                 = ''
    AlternateLocation               = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    LogXptStatus                    = '(monitor)'
    SendQEntries                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'server01'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server01.domain.com)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=orcl1_DGMGRL)(INSTANCE_NAME=orcl1)(SERVER=DEDICATED)))'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'
 
  Log file locations:
    Alert log               : /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/alert_orcl1.log
    Data Guard Broker log   : /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/drcorcl1.log
 
Database Status:
SUCCESS

Something from the past I would say, but in my testing I have strangely not been obliged to create but if you encounter any issue use below to add it to served services of your listener:

SQL> EXEC DBMS_SERVICE.CREATE_SERVICE('orcl1_DGMGRL','orcl1_DGMGRL');
 
PL/SQL PROCEDURE successfully completed.
 
SQL> EXEC DBMS_SERVICE.START_SERVICE('orcl1_DGMGRL','orcl1');
 
PL/SQL PROCEDURE successfully completed.

Then from listener:

[oracle@server01 ~]$ lsnrctl status listener_orcl1
 
LSNRCTL for Linux: Version 21.0.0.0.0 - Production on 08-AUG-2024 17:17:12
 
Copyright (c) 1991, 2021, Oracle.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server01.domain.com)(PORT=1531)))
STATUS of the LISTENER
------------------------
Alias                     listener_orcl1
Version                   TNSLSNR for Linux: Version 21.0.0.0.0 - Production
Start Date                06-AUG-2024 14:52:00
Uptime                    2 days 2 hr. 25 min. 13 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/homes/OraDB21Home1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/server01/listener_orcl1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server01.domain.com)(PORT=1531)))
Services Summary...
Service "1f2a48252f5d1e48e063402b4b0a02a8" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "c8209f27c6b16005e053362ee80ae60e" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "orcl1" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "orcl1_CFG" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "orcl1_DGMGRL" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "pdb1" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...
The command completed successfully

Remove it with:

SQL> EXEC dbms_service.delete_service('orcl1_DGMGRL');
 
PL/SQL PROCEDURE successfully completed.

You have also tried to change the configuration with something like, but it is not possible to change StaticConnectIdentifier:

DGMGRL> edit configuration set property StaticConnectIdentifier='orcl1';
Error: ORA-16568: cannot set property
 
Failed.

RMAN duplicate command error

I have also tried the DUPLICATE command to create this standby pluggable database like in the traditional container Data Guard but if you issue this single command:

RMAN> run {
        allocate channel ch_pri type disk;
        allocate auxiliary channel ch_aux type disk;
        duplicate pluggable database pdb1 from active database db_file_name_convert ('ORCL1', 'ORCL2');
      }
 
using target database control file instead of recovery catalog
allocated channel: ch_pri
channel ch_pri: SID=41 device type=DISK
 
allocated channel: ch_aux
channel ch_aux: SID=735 device type=DISK
 
Starting Duplicate PDB at 12-AUG-24
released channel: ch_pri
released channel: ch_aux
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate PDB command at 08/12/2024 14:58:14
RMAN-05501: aborting duplication of target database
RMAN-05630: Clone database pluggable database already exists: pdb1.

It complains that the target pdb1 pluggable database already exist and the FOR STANDY option has not yet been implemented:

RMAN> run {
        allocate channel ch_pri type disk;
        allocate auxiliary channel ch_aux type disk;
         duplicate pluggable database pdb1 for standby from active database db_file_name_convert ('ORCL1', 'ORCL2');
      }
 
allocated channel: ch_pri
channel ch_pri: SID=41 device type=DISK
 
allocated channel: ch_aux
channel ch_aux: SID=735 device type=DISK
 
Starting Duplicate PDB at 12-AUG-24
released channel: ch_pri
released channel: ch_aux
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate PDB command at 08/12/2024 15:03:38
RMAN-05501: aborting duplication of target database
RMAN-05635: duplicate pluggable database does not allow the use of FOR STANDBY clause

Maybe in 23ai the feature will be implemented to enhance Data Guard per Pluggable database…

In the case of a replication do not forget to set remote_recovery_file_dest parameter:

[oracle@server01 ~]$ rman target /@orcl1 auxiliary /@orcl2
 
Recovery Manager: Release 21.0.0.0.0 - Production on Thu Aug 8 12:32:52 2024
Version 21.15.0.0.0
 
Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.
 
connected to target database: ORCL1 (DBID=1495184713)
connected to auxiliary database: ORCL2 (DBID=1157121117)
 
RMAN> run {
     allocate channel ch_pri type disk;
     allocate auxiliary channel ch_aux type disk;
     duplicate pluggable database pdb1 from active database db_file_name_convert ('ORCL1', 'ORCL2');
   }
 
using target database control file instead of recovery catalog
allocated channel: ch_pri
channel ch1: SID=701 device type=DISK
 
Starting Duplicate PDB at 08-AUG-24
released channel: ch_pri
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate PDB command at 08/08/2024 12:32:59
RMAN-05501: aborting duplication of target database
RMAN-05655: specified REMOTE_RECOVERY_FILE_DEST parameter is NULL

Remark:
The message was really misleading for me as you have to set this parameter on orcl2 target container instance, well for reverse duplication set it on both:

SQL> ALTER SYSTEM SET remote_recovery_file_dest = '/u01/fra' scope=both;
 
SYSTEM altered.

DGM-17437

Looks like the feature is not really production ready as the number of possible errors is quite huge:

DGMGRL> switchover to pluggable database pdb1 at orcl2
Verifying conditions for Switchover...
 
Connected to "orcl2"
Connected to "orcl1"
  Source pluggable database is 'PDB1' at database 'orcl1'
 
Performing switchover NOW, please wait...
 
  Closing pluggable database 'PDB1'...
  Switching 'PDB1' to standby role...
Connected to "orcl2"
  Waiting for 'PDB1' to recover all redo data...
 
DGM-17437: Target pluggable database 'PDB1' at database 'orcl2' has not applied all redo; cannot complete role change
 
Failed.
DGMGRL> show pluggable database pdb1 at orcl2
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              con_id 3 at orcl1
  Transport Lag:       0 seconds (computed 9269 seconds ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      orcl2
  Average Apply Rate:  8 KByte/s
  Real Time Query:     OFF
 
Pluggable Database Status:
SUCCESS
 
DGMGRL> show pluggable database pdb1 at orcl1
 
Pluggable database 'pdb1' at database 'orcl1'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              (unknown)
 
Pluggable Database Status:
ORA-16911: Redo data from new source database has not been registered.

And the status of this pluggable database is clearly broken in Data Guard Broker:

DGMGRL> show pluggable database pdb1 at orcl1
 
Pluggable database 'pdb1' at database 'orcl1'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              (unknown)
 
Pluggable Database Status:
ORA-16911: Redo data from new source database has not been registered.
 
DGMGRL> show pluggable database pdb1 at orcl2
Connected to "orcl2"
 
Pluggable database 'pdb1' at database 'orcl2'
 
  Data Guard Role:     Physical Standby
  Con_ID:              3
  Source:              con_id 3 at orcl1
  Transport Lag:       0 seconds (computed 10109 seconds ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Average Apply Rate:  8 KByte/s
  Real Time Query:     OFF
 
Pluggable Database Status:
SUCCESS

The pluggable database is mounted on both instance:

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED

If you try to open it on initial primary instance (orcl1) it fails:

SQL> ALTER pluggable DATABASE pdb1 OPEN read WRITE;
ALTER pluggable DATABASE pdb1 OPEN read WRITE
*
ERROR AT line 1:
ORA-16003: standby DATABASE IS restricted TO read-only ACCESS

I have tried to cancel and relaunch the recovery process on target instance (orcl2) with, it runs forever without giving any positive feedback:

SQL> ALTER DATABASE recover managed standby DATABASE cancel;
 
DATABASE altered.
 
SQL> ALTER pluggable DATABASE pdb1 recover managed standby DATABASE;
.
.
.

The DGM-17437 error message says you have to relaunch the command but it is not possible:

DGMGRL> switchover to pluggable database pdb1 at orcl2
Verifying conditions for Switchover...
 
Connected to "orcl2"
Connected to "orcl1"
  Source pluggable database is 'PDB1' at database 'orcl1'
 
 
  Source pluggable database 'PDB1' is not open in READ WRITE mode.
 
Failed.

What I have been able to submit multiple times with no positive result is:

DGMGRL> failover to pluggable database pdb1 at orcl2
Verifying conditions for Failover...
 
Connected to "orcl2"
Connected to "orcl1"
  Source pluggable database is 'PDB1' at database 'orcl1'
 
Continuing with the switchover...
 
Connected to "orcl2"
  Waiting for 'PDB1' to recover all redo data...
 
DGM-17437: Target pluggable database 'PDB1' at database 'orcl2' has not applied all redo; cannot complete role change
 
Failed.

And I have found many ORA-00600 errors in alert.log of the database so clearly this is a bug (I recall that I have July 2024 RU and so 21.15 so latest available at the time of writing this post):

2024-09-04 15:04:20.237000 +02:00
 rfs (PID:3046142): krsr_rfs_atc: Identified database type as 'PRIMARY': Client is Foreground (PID:402238) [krsr.c:5659]
 rfs (PID:3046142): Foreign redo connection for orcl1(DBID:1495184713), Client is Foreground (PID:402238) [krsr.c:6455]
Errors in file /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_rfs_3046142.trc  (incident=36525) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [krasrfc_rx], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_36525/orcl2_rfs_3046142_i36525.trc
2024-09-04 15:04:22.433000 +02:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
 rfs (PID:3046142): Possible network disconnect with primary database [krsv.c:4860]
 rfs (PID:3046142): while processing B-1110735241.T-1.S-0 BNUM:0 BCNT:0 [krsv.c:4867]
 rfs (PID:3046142): Current process action IDLE, elapsed idle time 0 [krsv.c:4874]
 rfs (PID:3046142): RFS client GAP MANAGER (PID:402238) [krsv.c:4879]
Dumping diagnostic data in directory=[cdmp_20240904150423], requested by (instance=1, osid=3046142), summary=[incident=36525].

At this point I had to re-instantiate everything manually… There is also no references on MOS about DGM-17437 error and potential one-off patch that I could apply…

You can still *save* your pluggable database with:

DGMGRL> remove pluggable database pdb1 at orcl1 remove datafiles;
 
Pluggable Database 'PDB1' removed.

And

SQL> ALTER SESSION SET container=pdb1;
 
SESSION altered.
 
SQL> ALTER DATABASE recover managed standby DATABASE cancel;
 
DATABASE altered.
 
SQL> ALTER DATABASE activate standby DATABASE;
 
DATABASE altered.
 
SQL> ALTER DATABASE OPEN read WRITE;
 
DATABASE altered.

References

About Post Author

Yannick Jaquier

administrator

Find more about me on social media.

See author's posts

Share the knowledge!

Related Posts:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>