{"id":193,"date":"2016-10-21T21:41:58","date_gmt":"2016-10-22T03:41:58","guid":{"rendered":"http:\/\/www.aamirharoon.com\/?p=193"},"modified":"2020-10-29T23:53:55","modified_gmt":"2020-10-30T04:53:55","slug":"oracle-11gr2-dataguard-resolving-ora-19909","status":"publish","type":"post","link":"http:\/\/www.aamirharoon.com\/?p=193","title":{"rendered":"Oracle 11gr2 Dataguard &#8211; Resolving ORA 19909"},"content":{"rendered":"<p>Just wanted to write quick blog about an error we got on Standby Database and steps I performed to resolved it. &nbsp;Saved me from re-building the dataguard database.&nbsp;<img loading=\"lazy\" decoding=\"async\" style=\"margin-left: 3px; margin-right: 3px; vertical-align: middle; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none;\" src=\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png\" width=\"16\" height=\"16\"><\/p>\n<p>Configuration: &nbsp;All databases were on 11gR2 (11.2.0.4) single node using Oracle Restart.  We had one Primary database (prmy) and two Standby Database (stdby1 and stdby2). &nbsp;Primary was shipping logs to two standby database. &nbsp;In other words it wasn&#8217;t cascade configuration. &nbsp;For testing purposes we changed stdby1 to primary. &nbsp;The&nbsp;log_archive_dest_2 and&nbsp;log_archive_dest_3 were set to ship logs to other two databases. &nbsp;So as soon as stdby1 switched to Primary role, it start shipping logs to stdby2.<\/p>\n<p>Now stdby2 was getting redo logs from two different primary databases (prmy and stdby1) and getting very confused <img loading=\"lazy\" decoding=\"async\" style=\"margin-left: 3px; margin-right: 3px; vertical-align: middle; -webkit-box-shadow: none; -moz-box-shadow: none; box-shadow: none;\" src=\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F61F.png\" width=\"16\" height=\"16\">. &nbsp;It started reporting following errors in alert logs:<\/p>\n<pre class=\"c\" data-blogger-escaped-name=\"code\">ORA-19909: datafile 1 belongs to an orphan incarnation\nORA-01110: data file 1: '+DG_DATA_01\/stdby2\/datafile\/system.278.901783129'\nManaged Standby Recovery not using Real Time Apply\nRecovery Slave PR00 previously exited with exception 19909\nWed Apr 13 12:02:53 2016\nMRP0: Background Media Recovery process shutdown (stdby2)\nWed Apr 13 12:02:53 2016\nCompleted: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT\n<\/pre>\n<p>First thing I did is set log_archive_dest_state_2 and log_archive_dest_state_3 parameters to defer on stdby1.<\/p>\n<p>On stdby2, I removed all the archive logs since stdby1 became primary.<\/p>\n<pre class=\"c\" data-blogger-escaped-name=\"code\">ASMCMD&gt; rm -r 2016_04_13 2016_04_12\n\nYou may delete multiple files and\/or directories.\n\nAre you sure? (y\/n) y\n\n<\/pre>\n<p>Performed crosscheck to update RMAN catalog and controlfile.  But got ORA 1580 error.  To resolve this error you have to set the snapshot controlfile to temp file and set it back to original location.<\/p>\n<pre class=\"brush: bash; gutter: false; title: stdby2; notranslate\" title=\"stdby2\">\nRMAN&amp;gt; crosscheck archivelog all;\nORA-01580: error creating control backup file \/oracle\/product\/11.2.0\/dbhome_2\/dbs\/snapcf_prmy.f\nORA-27040: file create error, unable to create file\nLinux-x86_64 Error: 2: No such file or directory\nAdditional information: 1\nRMAN-08132: WARNING: cannot update recovery area reclaimable file list\n\nRMAN&amp;gt; CONFIGURE SNAPSHOT CONTROLFILE NAME TO '\/tmp\/snapcf_prmy.f';\n\nold RMAN configuration parameters:\nCONFIGURE SNAPSHOT CONTROLFILE NAME TO '\/oracle\/product\/11.2.0\/dbhome_2\/dbs\/snapcf_prmy.f';\nnew RMAN configuration parameters:\nCONFIGURE SNAPSHOT CONTROLFILE NAME TO '\/tmp\/snapcf_prmy.f';\nnew RMAN configuration parameters are successfully stored\n\nRMAN&amp;gt; crosscheck archivelog all;\n\nRMAN&amp;gt; CONFIGURE SNAPSHOT CONTROLFILE NAME TO '\/oracle\/product\/11.2.0\/dbhome_2\/dbs\/snapcf_prmy.f';\n\nold RMAN configuration parameters:\nCONFIGURE SNAPSHOT CONTROLFILE NAME TO '\/tmp\/snapcf_prmy.f';\nnew RMAN configuration parameters:\nCONFIGURE SNAPSHOT CONTROLFILE NAME TO '\/oracle\/product\/11.2.0\/dbhome_2\/dbs\/snapcf_prmy.f';\nnew RMAN configuration parameters are successfully stored\n<\/pre>\n<p>Now stdby2 doesn&#8217;t have archive logs from stdby1.  So, if I start Managed Recovery Process (MRP) on stdby2, it should start requesting logs from prmy database.<\/p>\n<pre class=\"brush: sql; gutter: false; title: stdby2; notranslate\" title=\"stdby2\">\nSQL&amp;gt; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;\n\nORA-19909: datafile 1 belongs to an orphan incarnation\nORA-01110: data file 1: '+DG_DATA_01\/stdby2\/datafile\/system.278.901783129'\n<\/pre>\n<p>hmmm&#8230;.got the same error as before.  stdby2 database always been in mounted state, so what is the deal with incarnation.  So I took a look at the incarnation.<\/p>\n<pre class=\"brush: bash; gutter: false; title: stdby2; notranslate\" title=\"stdby2\">\nRMAN&amp;gt; list incarnation of database;\n\nList of Database Incarnations\n\nDB Key&amp;nbsp; Inc Key  DB Name&amp;nbsp; DB ID&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;STATUS&amp;nbsp;  Reset SCN&amp;nbsp;   Reset Time\n------- -------- -------- -----------  -------- ----------   ----------\n476830&amp;nbsp; 476831&amp;nbsp;  PRMY&amp;nbsp;&amp;nbsp;   934143187&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;PARENT&amp;nbsp;  1&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;   28-january&amp;nbsp; -2014 12:14:11 pm\n476830&amp;nbsp; 28649947 PRMY&amp;nbsp;&amp;nbsp;   934143187&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;CURRENT  186709296840 11-april&amp;nbsp;&amp;nbsp;&amp;nbsp; -2016 11:23:26 am\n<\/pre>\n<p>A new incarnation was created around the same time stdby1 role was change to Primary.  Oracle implicitly created a new incarnation after receiving redo log from stdby1.  This is easy to fix:<\/p>\n<pre class=\"c\" data-blogger-escaped-name=\"code\">RMAN&gt; reset database to incarnation 476831;\n\ndatabase reset to incarnation 476831\n\nRMAN&gt; exit\n<\/pre>\n<p>After reseting the database to correct incarnation, MRP started successfully and stdby2 started to synchronized with prmy database.<\/p>\n<p>A simple understanding of system and dataguard saved me from rebuilding dataguard instance.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Just wanted to write quick blog about an error we got on Standby Database and steps I performed to resolved it. &nbsp;Saved me from re-building<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-193","post","type-post","status-publish","format-standard","hentry","category-11gr2"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Oracle 11gr2 Dataguard - Resolving ORA 19909 - The Database Handyman<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.aamirharoon.com\/?p=193\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Oracle 11gr2 Dataguard - Resolving ORA 19909 - The Database Handyman\" \/>\n<meta property=\"og:description\" content=\"Just wanted to write quick blog about an error we got on Standby Database and steps I performed to resolved it. &nbsp;Saved me from re-building\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.aamirharoon.com\/?p=193\" \/>\n<meta property=\"og:site_name\" content=\"The Database Handyman\" \/>\n<meta property=\"article:published_time\" content=\"2016-10-22T03:41:58+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-10-30T04:53:55+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png\" \/>\n<meta name=\"author\" content=\"Aamir\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@aamir814\" \/>\n<meta name=\"twitter:site\" content=\"@aamir814\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Aamir\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193\"},\"author\":{\"name\":\"Aamir\",\"@id\":\"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63\"},\"headline\":\"Oracle 11gr2 Dataguard &#8211; Resolving ORA 19909\",\"datePublished\":\"2016-10-22T03:41:58+00:00\",\"dateModified\":\"2020-10-30T04:53:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193\"},\"wordCount\":660,\"publisher\":{\"@id\":\"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63\"},\"image\":{\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png\",\"articleSection\":[\"Oracle Database 11gR2\"],\"inLanguage\":\"en\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193\",\"url\":\"https:\/\/www.aamirharoon.com\/?p=193\",\"name\":\"Oracle 11gr2 Dataguard - Resolving ORA 19909 - The Database Handyman\",\"isPartOf\":{\"@id\":\"http:\/\/www.aamirharoon.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png\",\"datePublished\":\"2016-10-22T03:41:58+00:00\",\"dateModified\":\"2020-10-30T04:53:55+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#breadcrumb\"},\"inLanguage\":\"en\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.aamirharoon.com\/?p=193\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#primaryimage\",\"url\":\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png\",\"contentUrl\":\"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.aamirharoon.com\/?p=193#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/www.aamirharoon.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Oracle 11gr2 Dataguard &#8211; Resolving ORA 19909\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/www.aamirharoon.com\/#website\",\"url\":\"http:\/\/www.aamirharoon.com\/\",\"name\":\"The Database Handyman\",\"description\":\"My notes about Oracle Database, APEX, OEM Cloud Control and whatever I learn\",\"publisher\":{\"@id\":\"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/www.aamirharoon.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63\",\"name\":\"Aamir\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"http:\/\/www.aamirharoon.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/63cc5bb07711a1b9719cc47e13a8205072859c1aef30fac28f412baa84b9cf9b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/63cc5bb07711a1b9719cc47e13a8205072859c1aef30fac28f412baa84b9cf9b?s=96&d=mm&r=g\",\"caption\":\"Aamir\"},\"logo\":{\"@id\":\"http:\/\/www.aamirharoon.com\/#\/schema\/person\/image\/\"},\"sameAs\":[\"http:\/\/www.aamirharoon.com\/about\/\",\"https:\/\/x.com\/aamir814\"],\"url\":\"http:\/\/www.aamirharoon.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Oracle 11gr2 Dataguard - Resolving ORA 19909 - The Database Handyman","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.aamirharoon.com\/?p=193","og_locale":"en_US","og_type":"article","og_title":"Oracle 11gr2 Dataguard - Resolving ORA 19909 - The Database Handyman","og_description":"Just wanted to write quick blog about an error we got on Standby Database and steps I performed to resolved it. &nbsp;Saved me from re-building","og_url":"https:\/\/www.aamirharoon.com\/?p=193","og_site_name":"The Database Handyman","article_published_time":"2016-10-22T03:41:58+00:00","article_modified_time":"2020-10-30T04:53:55+00:00","og_image":[{"url":"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png","type":"","width":"","height":""}],"author":"Aamir","twitter_card":"summary_large_image","twitter_creator":"@aamir814","twitter_site":"@aamir814","twitter_misc":{"Written by":"Aamir","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.aamirharoon.com\/?p=193#article","isPartOf":{"@id":"https:\/\/www.aamirharoon.com\/?p=193"},"author":{"name":"Aamir","@id":"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63"},"headline":"Oracle 11gr2 Dataguard &#8211; Resolving ORA 19909","datePublished":"2016-10-22T03:41:58+00:00","dateModified":"2020-10-30T04:53:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.aamirharoon.com\/?p=193"},"wordCount":660,"publisher":{"@id":"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63"},"image":{"@id":"https:\/\/www.aamirharoon.com\/?p=193#primaryimage"},"thumbnailUrl":"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png","articleSection":["Oracle Database 11gR2"],"inLanguage":"en"},{"@type":"WebPage","@id":"https:\/\/www.aamirharoon.com\/?p=193","url":"https:\/\/www.aamirharoon.com\/?p=193","name":"Oracle 11gr2 Dataguard - Resolving ORA 19909 - The Database Handyman","isPartOf":{"@id":"http:\/\/www.aamirharoon.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.aamirharoon.com\/?p=193#primaryimage"},"image":{"@id":"https:\/\/www.aamirharoon.com\/?p=193#primaryimage"},"thumbnailUrl":"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png","datePublished":"2016-10-22T03:41:58+00:00","dateModified":"2020-10-30T04:53:55+00:00","breadcrumb":{"@id":"https:\/\/www.aamirharoon.com\/?p=193#breadcrumb"},"inLanguage":"en","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.aamirharoon.com\/?p=193"]}]},{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/www.aamirharoon.com\/?p=193#primaryimage","url":"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png","contentUrl":"http:\/\/www.aamirharoon.com\/wp-content\/plugins\/wp-emoji-one\/icons\/1F600.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.aamirharoon.com\/?p=193#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/www.aamirharoon.com\/"},{"@type":"ListItem","position":2,"name":"Oracle 11gr2 Dataguard &#8211; Resolving ORA 19909"}]},{"@type":"WebSite","@id":"http:\/\/www.aamirharoon.com\/#website","url":"http:\/\/www.aamirharoon.com\/","name":"The Database Handyman","description":"My notes about Oracle Database, APEX, OEM Cloud Control and whatever I learn","publisher":{"@id":"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/www.aamirharoon.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en"},{"@type":["Person","Organization"],"@id":"http:\/\/www.aamirharoon.com\/#\/schema\/person\/dbcd256c4a97c6e27b7cc408200a9b63","name":"Aamir","image":{"@type":"ImageObject","inLanguage":"en","@id":"http:\/\/www.aamirharoon.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/63cc5bb07711a1b9719cc47e13a8205072859c1aef30fac28f412baa84b9cf9b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/63cc5bb07711a1b9719cc47e13a8205072859c1aef30fac28f412baa84b9cf9b?s=96&d=mm&r=g","caption":"Aamir"},"logo":{"@id":"http:\/\/www.aamirharoon.com\/#\/schema\/person\/image\/"},"sameAs":["http:\/\/www.aamirharoon.com\/about\/","https:\/\/x.com\/aamir814"],"url":"http:\/\/www.aamirharoon.com\/?author=1"}]}},"_links":{"self":[{"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=\/wp\/v2\/posts\/193","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=193"}],"version-history":[{"count":32,"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=\/wp\/v2\/posts\/193\/revisions"}],"predecessor-version":[{"id":331,"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=\/wp\/v2\/posts\/193\/revisions\/331"}],"wp:attachment":[{"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=193"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=193"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.aamirharoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=193"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}