Oracle Database Appliance – Installing Bundle Patch 2.3.0.0.0

On 23 july 2012 ODA bundle patch 2.3.0.0.0 was released, a long awaited version which should (and indeed does) include multi-home support. As with each new bundle patch, in this version the patch process is getting more robust and the long awaited support for rolling upgrade of the Grid Infrastructure and RAC databases is implemented. Unfortunately the installation of the infrastructure part still requires downtime of the complete cluster stack and rebooting of both nodes (although you can skip the reboot part of the infra patch and reboot the nodes at a more convenient time).

At the beginning I was very surprised how well this new bundle patch installed on one of our test ODA’s. It installed without any problems and in a couple of hours I brought this ODA from 2.1.0.3.1 to 2.3.0.0.0 (2.1.0.3.1 -> 2.2.0.0.0 – only infra + Grid and 2.2.0.0.0 -> 2.3.0.0.0) including database upgrade. Unfortunately after this successfull installation and upgrade of 7 test databases, on the second ODA I patched after that, the first node crashed during patching of the 2.3 infra component. After cleanup and restarting I was able to fix this ODA, but while patching the third ODA, the second node crashed during the installation of the 2.3 infra patch component. Unfortunately cleanup and restarting the patch didn’t help.

On this ODA I now have 1 node with an old BIOS version and even worse, the first node having an applied core configuration key, but on the second node OAK doesn’t know anymore about this applied core configuration key on the first node.

Solution for the core_config_key problem: run the oakcli apply core_config_key <key file> again on the second node. After the reboot (it will only reboot the second node) OAK knows about the key again.

My guess is that the problem lies in the combination of the usage of core configuration keys and the BIOS upgrade that is done during the installation of the infra component/phase of the 2.3 bundle patch. At the moment I have an SR opened with Oracle Support and I can’t patch my other ODA’s until the problem is solved.

Although I have problems with the installation of bundle patch 2.3.0.0.0 I will describe the patching process here, to get some idea of how long things take and what questions you can expect.

Patch installation – component infra

The first component that needs to be patched is the infra component that for bundle patch 2.3.0.0.0 consists of firmware patches (BIOS, ILOM, SAS disks) and of course a new version of OAK. As mentioned, the installation of the infra component requires downtime of the complete cluster (the complete Grid Infrastructure will be shutdown).

Mandatory requirements before installing the 2.3 infra component:

  • Copy the zipfile containing the 2.3 bundle patch (p13982331_23000_Linux-x86-64.zip) to BOTH nodes of the ODA
  • Unpack this patch using the oakcli unpack -package command on BOTH nodes of the ODA
  • You need the passwords (just sudo isn’t enough) for OS user root

After that start the installation of the infra component using the command (run as user root):

 oakcli update -patch 2.3.0.0.0 --infra

It will take around 45 minutes from starting the patch process until both ODA nodes are rebooted and the complete CRS stack and databases are up and running again (time may differ a little depending on the number of databases).

Also make sure to do the checks mentioned in the 2.3 bundle patch release notes on BOTH NODES (run as user root):

dmidecode -t 1 | grep "Serial Number"
fwupdate list disk | grep -A5 CONTROLLER

For the dmidecode command, the serial number of your ODA should be returned on both nodes. For the fwupdate command the output should contain a value in the “BIOS version” column for controllers c1 and c2 on both nodes. If any of these checks fail, reboot the node and check again.

Patch installation – component GI

As of ODA bundle patch 2.3.0.0.0, patching (installation of an PSU) the Grid Infrastructure requires is done in a rolling fashion. So unlike the infra component all RAC databases will be available during the patch process. The process will first patch the Grid Infra environment on ODA node 1 (SC0) and when it is finished and all instances are started again, it will start patching the Grid Infrastructure on ODA node 2 (SC1).

Requirements before installing the 2.3 GI component:

  • You need the passwords (just sudo isn’t enough) for OS users root and grid

After that start the installation of the gi component using the command (run as user root):

 oakcli update -patch 2.3.0.0.0 --gi

It will take around 30 minutes for the Grid Infrastructure patch to complete and as mentioned doesn’t require downtime for RAC databases.

Patch installation – creating 11.2.0.3.3 dbhome

Because I didn’t have my ODA’s running on ODA version 2.2.0.0.0 yet, mainly because this bundle patch introduced a very nasty bug with crashing 11.2.0.2.x (RAC) databases, I had to install mandatory parts of this bundle patch first to be able to install the 2.3 bundle patch. I decided to only install the infra and GI component of the 2.2 bundle patch and use the new 2.3 command oakcli create dbhome command to create a new 11.2.0.3 oracle home and use the oakcli upgrade database command to upgrade my databases from 11.2.0.2.5 (ODA 2.1.0.3.1) to 11.2.0.3.3 (ODA 2.3.0.0.0). Using the database upgrade functionality of ODA 2.3 also doesn’t have the known issue of ODA 2.2 that databases with a databases name containing capitals couldn’t get upgraded.

Requirements for creating an Oracle home (11.2.0.3.3):

  • You need the passwords (just sudo isn’t enough) for OS users root and oracle
  • You need the password for user SYS on the ASM instances

To create a new 11.2.0.3.3 Oracle database home use the following command (run as user root):

create dbhome -version 11.2.0.3.3

It takes around 6 minutes to create this new Oracle home.

Patch installation – upgrading databases

After creating the new 11.2.0.3.3 Oracle home for databases I upgraded all databases running Oracle 11.2.0.2.5 to Oracle 11.2.0.3.3 using the oakcli upgrade database command.

Requirements for upgrading databases using the oakcli upgrade database command:

  • You need the passwords (just sudo isn’t enough) for OS users root and oracle
  • A list of all available dbhomes
    Use the command oakcli show dbhomes
  • A list of all available databases
    Use the command oakcli show databases

To upgrade 1 database use the following command (run as user root):

oakcli upgrade database -db <dbname> -to <destination home name>

Example:
oakcli upgrade database -db mcldbts1 -to OraDb11203_home1

To upgrade all databases running from one Oracle home (serial process – run as user root):

oakcli upgrade database -from <source home name> -to <destination home name>

Example:
oakcli upgrade database -from testdbb1 -to OraDb11203_home1

The database(s) gets upgraded by oakcli using the DBUA in the background. The logfiles of the upgrade process can be found under the directory structure:

/u01/app/oracle/cfgtoollogs/dbua

This entry was posted in Database Appliance and tagged , , , , , , , . Bookmark the permalink.

2 Responses to Oracle Database Appliance – Installing Bundle Patch 2.3.0.0.0

  1. Sanjay says:

    Marcel,
    Thanks for the article.
    Does the command “create dbhome -version 11.2.0.3.3” create an oracle home with the string 11.2.0.3.3 in the path or is it restricted to 11.2.0.3?

    Thanks!

    • marcel says:

      Hi Sanjay,

      The command created a new dbhome with name OraDb11203_home1 in the following path /u01/app/oracle/product/11.2.0.3/dbhome_1
      so it doesn’t include the PSU number in the home name. As I understood from the ODA MOS articles PSU’s are applied inplace (in the selected Oracle homes) so
      putting the PSU number in the path wouldn’t be a very good choice.

      Regards,

      Marcel

Leave a Reply

Your email address will not be published. Required fields are marked *

Blue Captcha Image
Refresh

*