Skip to main content

Multitenant and standby: recover from subsetting

In the previous post we learnt how to exclude a PDB (or a datafile) from the standby database recovery. Of course, that might not be the real end goal. We may just want to skip it for now, but have the standby continue to be up-to-date for every other PDB, and eventually include the new PDB as well.

Again, standard Oracle pre-12c DBA knowledge is helpful here. These files are just missing datafiles and thus a backup can be used to restore them. The new 12c features add some quirks to this process, but the base is just sound backup and recovery.

Backup

So let's start with a proper backup: 

rman target=/

Recovery Manager: Release 12.1.0.2.0 - Production on Mon Nov 16 12:42:38 2015

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

backup database;
connected to target database: CDB2 (DBID=600824249)

Starting backup at 16-NOV-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=193 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
...
... a lot of lines omitted,
... RMAN creates one backupset for each PDB.
...
Finished backup at 16-NOV-15

Starting Control File and SPFILE Autobackup at 16-NOV-15
piece handle=/home/oracle/fra/CDB2/autobackup/2015_11_16/o1_mf_s_895927547_c4mmzw1t_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 16-NOV-15

In order to make use of the backup on the standby side, the standby must know about this backup. We could either use RMAN catalog, or simply register the backup pieces on the standby, having copied the files there first, of course: 
catalog start with '/home/oracle/fra/CDB2/';

...

searching for all files that match the pattern /home/oracle/fra/CDB2/

List of Files Unknown to the Database
=====================================
File Name: /home/oracle/fra/CDB2/205A13D4947F11DCE0530100007F5CA4/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmvjx4_.bkp
File Name: /home/oracle/fra/CDB2/244AEB6C0961185CE0530100007F6410/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmwhjd_.bkp
...
Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /home/oracle/fra/CDB2/205A13D4947F11DCE0530100007F5CA4/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmvjx4_.bkp
File Name: /home/oracle/fra/CDB2/244AEB6C0961185CE0530100007F6410/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmwhjd_.bkp
...

Offlined datafiles, standby file management auto

So let's restore one of the PDBs: 
restore pluggable database 'PDBCOPY3';

Starting restore at 16-NOV-15
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00020 to /oradata2/CDB2SBY/pdbcopy3/system01.dbf
channel ORA_DISK_1: restoring datafile 00021 to /oradata2/CDB2SBY/pdbcopy3/sysaux01.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/fra/CDB2/246FA48B945F1085E0530100007FDD58/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmyf0t_.bkp
channel ORA_DISK_1: ORA-19870: error while restoring backup piece /home/oracle/fra/CDB2/246FA48B945F1085E0530100007FDD58/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmyf0t_.bkp
ORA-19504: failed to create file "/oradata2/CDB2SBY/pdbcopy3/sysaux01.dbf"
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 1

failover to previous backup

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 11/16/2015 12:47:19
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 21 found to restore
RMAN-06023: no backup or copy of datafile 20 found to restore

But wait... no backup of datafile 20 found? How come? We just backed it up and registered here... NO! That's a trap :-) The error itself is hidden few lines above: failed to create file "/oradata2/CDB2SBY/pdbcopy3/sysaux01.dbf So we need to actually create the path manually first: 
!mkdir /oradata2/CDB2SBY/pdbcopy3

restore pluggable database pdbcopy3;

Starting restore at 16-NOV-15
using channel ORA_DISK_1

skipping datafile 20; already restored to file /oradata2/CDB2SBY/pdbcopy3/system01.dbf
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00021 to /oradata2/CDB2SBY/pdbcopy3/sysaux01.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/fra/CDB2/246FA48B945F1085E0530100007FDD58/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmyf0t_.bkp
channel ORA_DISK_1: piece handle=/home/oracle/fra/CDB2/246FA48B945F1085E0530100007FDD58/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmyf0t_.bkp tag=TAG20151116T124303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 16-NOV-15


recover pluggable database pdbcopy3;

Starting recover at 16-NOV-15
using channel ORA_DISK_1

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 11/16/2015 12:51:00
RMAN-06067: RECOVER DATABASE required with a backup or created control file

So we have restored the files ok, but RMAN cannot recover them by itself, so some additional steps are required. But the good news is that the standby recovery process will do this for us.
There is just one more step necessary before this; although the files are there on the standby server, the database marks them as offline. One can query v$datafile, or just do the usual check of the alert.log, and see the following: 
Warning: Datafile 20 (/oradata/CDB2/pdbcopy3/system01.dbf) is offline during full database recovery and will not be recovered
Warning: Datafile 21 (/oradata/CDB2/pdbcopy3/sysaux01.dbf) is offline during full database recovery and will not be recovered

So let's finish off with this PDB, by onlining the datafiles and setting to recover mode: 
alter session set container=pdbcopy3;

Session altered.

alter database datafile 20 online;

Database altered.

alter database datafile 21 online;

Database altered.

alter session set container=cdb$root;

Session altered.

alter database recover managed standby database;

And the standby database will then happily hum along.


Skipped standby

Does the picture change with the new STANDBYS clause introduced in 12.1.0.2? Indeed it does, but not dramatically.
First of all, the files don't have the proper names in v$datafile, as with standby_file_management=manual. This means we must set the new file names in the RMAN script and then switch to them: 
run {
set newname for datafile 22 to '/oradata2/CDB2SBY/pdbcopy4/system.dbf';
set newname for datafile 23 to '/oradata2/CDB2SBY/pdbcopy4/sysaux.dbf';
restore pluggable database pdbcopy4;
}

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 16-NOV-15
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00022 to /oradata2/CDB2SBY/pdbcopy4/system.dbf
channel ORA_DISK_1: restoring datafile 00023 to /oradata2/CDB2SBY/pdbcopy4/sysaux.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/fra/CDB2/246FA48B94601085E0530100007FDD58/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmywdq_.bkp
channel ORA_DISK_1: piece handle=/home/oracle/fra/CDB2/246FA48B94601085E0530100007FDD58/backupset/2015_11_16/o1_mf_nnndf_TAG20151116T124303_c4mmywdq_.bkp tag=TAG20151116T124303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 16-NOV-15

switch datafile 22 to datafilecopy;
datafile 22 switched to datafile copy "/oradata2/CDB2SBY/pdbcopy4/system.dbf"

switch datafile 23 to datafilecopy;
datafile 23 switched to datafile copy "/oradata2/CDB2SBY/pdbcopy4/sysaux.dbf"

The second difference is around how to bring the datafiles online. Instead of running alter online, we must enable recovery of the PDB itself: 
alter session set container=pdbcopy4;

Session altered.

alter database datafile 22 online
*
ERROR at line 1:
ORA-65152: cannot bring datafile online

!oerr ora 65152
65152, 00000, "cannot bring datafile online"
// *Cause:   An attempt was made to bring online one or more datafiles
//           belonging to a pluggable database that is disabled for recovery.
// *Action:  Issue the ALTER PLUGGABLE DATABASE ENABLE RECOVERY statement first
//           and retry the operation.
//

alter pluggable database enable recovery;

Pluggable database altered.
alter session set container=cdb$root;

Session altered.

alter database recover managed standby database;

And once again, following these steps, the PDB is now properly recovered by the standby, ready for a switchover or failover.

Comments

Post a Comment

Popular posts from this blog

Filter and access predicates

More than just column projections When we look around for further pointers in the tree nodes, we find more pieces resembling the column projections we have seen so far. With some experimenting, we will find out that these are access predicates and filters. First of all, the location of these pointers is not always the same, it seems that the value at 0x34 is some kind of flag, indicating whether filters and/or access predicates are present, and where. Or whether there is just one, or more of them.  It probably also indicates what other info is present, but I have no idea what info that would be or what each value means. Resembling, but different The data we see as predicates are not columns; after all, a predicate is a condition, not a single column. But the structure is similar to what we have seen with columns, and if we follow pointers further, we eventually build a tree, and some of the leaves are indeed just column projections. After some contemplation, we realize it's...
Post a Comment
Read more

dbms_alert on RAC

Not long time ago, I came across a usage of dbms_alert to manage running jobs. As the solution implemented must work also for RAC, I wanted to know whether dbms_alert works on RAC across instances. The documentation nor Metalink does not say anything (contrary to dbms_pipe, which does NOT work on RAC). So, if they don't warn, it should work... However, Julian Dyke says, that dbms_alert does not work and is the same as dbms_pipe (sources: http://juliandyke.com/Presentations/Presentations.html#ARoughGuideToRAC , page 17, or Pro Oracle Database 10g RAC on Linux, page 426). You know, never trust anybody, so I conducted a test case (10.2.0.3 on Linux x86_64, VMware ESX server, 2-node RAC): You will need two simultaneous sessions, I mark them with DWH1> and DWH2> here. DWH1> select instance_name from v$instance; INSTANCE_NAME ---------------- DWH1 DWH2> select instance_name from v$instance; INSTANCE_NAME ---------------- DWH2 DWH2> exec dbms_alert.register('TST'); ...
Post a Comment
Read more

Execution plan rows

The plan As mentioned in previous post, our example is based on the sample SH schema, with an added table FOOBAR (id number, key varchar2(30)): SELECT prod_id, key FROM products CROSS JOIN foobar WHERE prod_id in (143,144,id) and id in (1,2,3); In all reverse engineering, it's good to start with something simple and to know what we to look for. Thus we want to know what the execution plan should look like; and the more unique numbers/ids we can find, the better. It's much easier to look for a number like 0x12fa1893 than for 0x0 or 0x1. The execution plan, obtained using: SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY(null,null,'ALL'));  is I have added the CPU cost from the full detail of the execution plan in  v$sql_plan / x$kqlfxpl. Looking at the numbers, we also have rows (1 and 2 ... not very unique), bytes (34 / 30 / 8 - not bad) and what is not shown here, we also know object ids of the index and the table: 94765 and 92749 (nice). We did not use any...
Post a Comment
Read more