Oracle Database Appliance – Installing Patch Bundle 2.2

On 17 April 2012 ODA patch bundle 2.2.0.0.0 was released, which includes an Linux kernel upgrade (2.6.18-194.32.1.0.1.e15 to 2.6.32-300.11.1.el5uek – Unbreakable Enterprise Kernel), Oracle Grid Infrastructure patchset 11.2.0.3.2 and Oracle RDBMS patchset 11.2.0.3.2.
Oracle made the installation of various parts of the patch bundle a lot more flexible. Now you can install the infrastructure (OS, firmware, OAK, etc.), GI and RDBMS parts independently from each other. You can even choose to just install the Oracle RDBMS software without upgrading the existing databases and keep them running on the current 11.2.0.2.x patchset. This way you can just install the “infra” part without upgrading the rest. Unfortunately you have to upgrade to GI patchset 11.2.0.3.2 directly after upgrading the “infra” part if you are using ACFS, because ACFS in GI 11.2.0.2.x doesn’t work with the new Linux kernel.

Patch installation – part 1 (infra)

As with the previous ODA patch bundles, the installation of patch bundle 2.2 isn’t rolling and you will have downtime. The first part of the installation (infra) does a shutdown of the complete CRS stack (cluster) and will reboot both nodes at the end (at the same time). Depending on how many databases are running on your ODA, it will take 45 – 60 minutes before everything is running again.

Patch installation – part 2 (GI & RDBMS)

Especially if you are using ACFS (/cloudfs filesystem) you will have to install the Grid Infrastructure (GI) patchset 11.2.0.3.2 as soon as possible. Installation of the GI patchset isn’t rolling, so again the complete CRS stack (cluster) will be shutdown. I have chosen to install the GI patchset and RDBMS patchset (software only) in one “oakcli update -patch 2.2.0.0.0 –gi –database” run and it took around 45 minutes to complete (the cluster resources like databases aren’t available during this period).

Known issues

There are some “know issues” that come with this patch bundle. The most annoying ones I will sum down here:

• The privileges on the “oracle” executable in the old (11.2.0.2) home become incorrect after installing this patch bundle. So you have to fix them as noted in the patch readme.
• With the installation of the 2.2.0.0.0 patch bundle you can choose to let all or a selection of databases be upgraded to 11.2.0.3. However there is a bug which prevents databases that have capitalized database names to be upgraded automatically.

Problems I ran into

Although the installation of ODA patch bundle 2.2.0.0.0 went without visible problems, and it fixed a problem with a shared disk (state details for a disk: PredictiveFail), it introduced a new and I think a much bigger problem with the availability of our RAC databases. Whenever I reboot one of the ODA nodes, the RAC instances on the remaining node crash with an ORA-07445 (generated by the LMD0 daemon – Global Enqueue Service Daemon), see the error below. I have an Service Request opened for this and I will update this post whenever the problem is solved.

SKGXP: ospid 16245: network interface with IP address 169.254.117.116 no longer running (check cable)
Exception [type: SIGSEGV, Invalid permissions for mapped object] [ADDR:0x7FEA51FBB592] [PC:0x7FEA548AA5E7, skgxp_local_status_change()+191] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/tstdb1/tstdb11/trace/tstdb11_lmd0_16245.trc  (incident=121697):
ORA-07445: exception encountered: core dump [skgxp_local_status_change()+191] [SIGSEGV] [ADDR:0x7FEA51FBB592] [PC:0x7FEA548AA5E7] [Invalid permissions for mapped object] []

Problem solved!

Finally the problem with the crashing RAC instance (the surviving one) has been solved. You have to upgrade your databases (RDBMS) to 11.2.0.3 or if you need to be running on 11.2.0.2 you will have to apply patch 12628521 –SKGXP V3.4 – CUMULATIVE FIXES PATCH 6.1 (For description of the bug fixed see MOS note 11711682.8).

The problem is being described as an 11.2.0.2 generic problem, but the problem doesn’t occur on ODA 2.1.0.3.1 so it has be some combination of either the new GI (11.2.0.3) that comes with ODA 2.2.0.0.0 or the kernel/OS upgrade. I did ask Oracle about this, but they didn’t know (wanted to look further into this).