Standby MRP0: Background Media Recovery process shutdown

Recently I encountered ORA-00283: recovery session canceled due to errors due to ORA-00600 on standby Database. After I execute "ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;".

MRP0: Background Media Recovery process shutdown (orcl)...

Normally, ORA-00600 advised to be raised SR but i don’t have the time luxury, waiting for oracle support.

Checking the alert log, I notice that RFS process is OK. Archived Log transfer is OK on the standby database.

But managed Standby Recovery not using Real Time Apply after ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

Below are the revenant lines from alert log file

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSIONAttempt to start background Managed Standby Recovery process (orcl)Sun Sep 09 14:11:02 2012MRP0 started with pid=27, OS id=12230
MRP0: Background Managed Standby Recovery process started (orcl) started logmerger processSun Sep 09 14:11:07 2012Managed Standby Recovery starting Real Time ApplyParallel Media Recovery started with 2 slavesCompleted: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSIONWaiting for all non-current ORLs to be archived...All non-current ORLs have been archived.Media Recovery Log /u01/app/dg/1_148_766776473.dbfErrors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc  (incident=27842):ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Incident details in: /u01/app/oracle/diag/rdbms/orcldg/orcl/incident/incdir_27842/orcl_pr00_12232_i27842.trcErrors with log /u01/app/dg/1_148_766776473.dbfMRP0: Background Media Recovery terminated with error 600Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Managed Standby Recovery not using Real Time ApplyRecovery interrupted!Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Slave exiting with ORA-600 exceptionErrors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_mrp0_12230.trc  (incident=27834):ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
Incident details in: /u01/app/oracle/diag/rdbms/orcldg/orcl/incident/incdir_27834/orcl_mrp0_12230_i27834.trcSun Sep 09 14:11:09 2012Trace dumping is performing id=[cdmp_20120909141109]Sun Sep 09 14:11:09 2012Sweep [inc][27842]: completedSweep [inc][27834]: completedSweep [inc2][27842]: completedMRP0: Background Media Recovery process shutdown (orcl)

Next was to check the archived redo that are applied: [see doc: http://docs.oracle.com/cd/B19306_01/server.102/b14239/create_ps.htm#i77231] 

SELECT SEQUENCE#, APPLIED, REGISTRAR FROM V$ARCHIVED_LOG where APPLIED='NO' ORDER BY SEQUENCE#;

148         NO         RFS
149         NO         RFS
150         NO         RFS
...

Starting from log sequence 148, log are not applied on standby DB.

The trace files in the alert log above,  did not give any meaningful help. So I considered using RMAN to apply the archived redo log.

But received an error in RMAN pointing towards log 148.(unknown/missing/corrupt).

Next, to confirm that archived log 148 is the culprit. I execute the command in trace files location:

[oracle@red1 trace]$ grep 148 *

Lone and behold found:

orcl_rfs_23788.trc:ORA-19502: write error on file "/u01/app/dg/1_148_766776473.dbf", block number 82560 (block size=512)orcl_rfs_23788.trc:ORA-19502: write error on file "/u01/app/dg/1_148_766776473.dbf", block number 82560 (block size=512)

So what do?

Remove the u01/app/dg/1_148_766776473.dbf from the standby database.

Copy the u01/app/dg/1_148_766776473.dbf from the primary database to the standby database.

$scp -rC primary:/u01/app/dg/1_148_766776473.dbf  standby:/u01/app/dg/1_148_766776473.dbf

On the standby database:

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

Checked the alert log:

Media Recovery Log and real-time apply successful

The End….

HTH

Tobi