Recently I encountered ORA-00283: recovery session canceled
due to errors due to ORA-00600 on standby Database. After I execute "ALTER
DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM
SESSION;".
MRP0: Background Media Recovery process shutdown (orcl)...
Normally, ORA-00600 advised to be raised SR but i don’t have
the time luxury, waiting for oracle support.
Checking the alert log, I notice that RFS process is OK.
Archived Log transfer is OK on the standby database.
But managed Standby Recovery not using Real Time Apply after
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE
DISCONNECT FROM SESSION;
Below are the revenant lines from alert log file
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSIONAttempt to start background Managed Standby Recovery process (orcl)Sun Sep 09 14:11:02 2012MRP0 started with pid=27, OS id=12230
MRP0: Background Managed Standby Recovery process started (orcl) started logmerger processSun Sep 09 14:11:07 2012Managed Standby Recovery starting Real Time ApplyParallel Media Recovery started with 2 slavesCompleted: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSIONWaiting for all non-current ORLs to be archived...All non-current ORLs have been archived.Media Recovery Log /u01/app/dg/1_148_766776473.dbfErrors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc (incident=27842):ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Incident details in: /u01/app/oracle/diag/rdbms/orcldg/orcl/incident/incdir_27842/orcl_pr00_12232_i27842.trcErrors with log /u01/app/dg/1_148_766776473.dbfMRP0: Background Media Recovery terminated with error 600Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Managed Standby Recovery not using Real Time ApplyRecovery interrupted!Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Slave exiting with ORA-600 exceptionErrors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_pr00_12232.trc:ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl/trace/orcl_mrp0_12230.trc (incident=27834):ORA-00600: internal error code, arguments: [2619], [148], [], [], [], [], [], [], [], [], [], []ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
Incident details in: /u01/app/oracle/diag/rdbms/orcldg/orcl/incident/incdir_27834/orcl_mrp0_12230_i27834.trcSun Sep 09 14:11:09 2012Trace dumping is performing id=[cdmp_20120909141109]Sun Sep 09 14:11:09 2012Sweep [inc][27842]: completedSweep [inc][27834]: completedSweep [inc2][27842]: completedMRP0: Background Media Recovery process shutdown (orcl)
Next was to check the archived redo that are applied: [see doc: http://docs.oracle.com/cd/B19306_01/server.102/b14239/create_ps.htm#i77231]
SELECT SEQUENCE#, APPLIED, REGISTRAR FROM V$ARCHIVED_LOG
where APPLIED='NO' ORDER BY SEQUENCE#;
148 NO RFS
149 NO RFS
150 NO RFS
...
Starting from log sequence 148, log are not applied on
standby DB.
The trace files in the alert log above, did not give any meaningful help. So I
considered using RMAN to apply the archived redo log.
But received an error in RMAN pointing towards log
148.(unknown/missing/corrupt).
Next, to confirm that archived log 148 is the culprit. I
execute the command in trace files location:
[oracle@red1 trace]$ grep 148 *
Lone and behold found:
orcl_rfs_23788.trc:ORA-19502: write error on file "/u01/app/dg/1_148_766776473.dbf", block number 82560 (block size=512)orcl_rfs_23788.trc:ORA-19502: write error on file "/u01/app/dg/1_148_766776473.dbf", block number 82560 (block size=512)
So what do?
Remove the u01/app/dg/1_148_766776473.dbf from the standby
database.
Copy the u01/app/dg/1_148_766776473.dbf from the primary
database to the standby database.
$scp -rC primary:/u01/app/dg/1_148_766776473.dbf standby:/u01/app/dg/1_148_766776473.dbf
On the standby database:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING
CURRENT LOGFILE DISCONNECT FROM SESSION;
Checked the alert log:
Media Recovery Log and real-time apply successful
The End….
HTH
Tobi
