Surviving the Grid Infrastructure Jan 2018 PSU Patch

Lessons from the Trenches: Surviving the Grid Infrastructure Jan 2018 PSU Patch

Recently, I set out on a routine maintenance journey: applying the latest Patch Release Update (GI RU 12.2.0.1.180116) across a fresh Grid Infrastructure 12.2 and RDBMS setup with several RAC databases. What seemed like a straightforward task turned into a great learning experience full of unexpected hurdles, workarounds, and practical troubleshooting insights. Here is the story of how it went and what to look out for on your next patching adventure!

1. Mind the OPatch Version: Avoid the Wallet Trap

Rule number one: always grab the absolute latest version of OPatch before starting. While the patch readme mentioned that OPatch version 12.2.0.1.6 or later was sufficient, using 12.2.0.1.6 threw a surprising error:

OPATCHAUTO-68021: The following argument(s) are required: [-wallet]

A quick check on My Oracle Support (Doc ID 2270185.1) revealed that OPatchauto 12.2.0.1.6 required creating a wallet file with passwords on every single node. Fortunately, upgrading to OPatch version 12.2.0.10 completely removed this mandatory wallet requirement, saving a ton of unnecessary setup time.

2. Clean Up Your Central Inventory

Because opatchauto relies heavily on the inventory.xml file (found in <inventory_loc>/ContentsXML), it is vital to ensure that your inventory only contains active, intended homes—in my case, just the GI 12.2 and RDBMS 12.2 homes.

If you have stale or unneeded homes registered, clean them up beforehand using commands like:

/grid/app/12.2.0/grid/oui/bin/runInstaller -silent -detachHome ORACLE_HOME="/opt/oracle/oraclex/product/12.2.0/db_1"

/grid/app/12.2.0/grid/oui/bin/runInstaller -silent -detachHome ORACLE_HOME="/opt/oracle/agent/agent_13.2.0.0.0"

Double-check that your cluster node lists inside inventory.xml aren't empty. If node entries are missing, update them with:

/grid/app/12.2.0/grid/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/grid/app/12.2.0/grid "CLUSTER_NODES=<node1,node2,node3,...>"

3. Handle ACFS Filesystems with Care

Another critical step involves ACFS (ASM Cluster File System). Per Oracle Support Doc ID 1591616.1, active ACFS filesystems must be manually unmounted before starting opatchauto. Here is the recommended sequence:

1. Locate the ACFS resources:

# crsctl stat res -w "TYPE = ora.acfs.type" -p | grep VOLUME

2. Stop the filesystem resource as root:

# srvctl stop filesystem -d <volume device path> -n <node>

4. What Happens When ACFS Isn't Stopped? (And How to Recover)

In one instance, an ACFS filesystem was left running by mistake. Sure enough, the patch run failed towards the end with ADVM/ACFS uninstallation errors and an exit code 42:

2018/01/31 22:40:44 CLSRSC-205: Failed to uninstall ADVM/ACFS

OPATCHAUTO-68061: The orchestration engine failed with return code 1

opatchauto failed with error code 42

If you encounter this lockup, don't panic! Here is the recovery roadmap that got everything back on track:

1. Disable CRS as root: crsctl disable crs

2. Reboot the node to clear any busy device states.

3. Re-enable CRS once back online: crsctl enable crs

4. Resume patching: opatchauto resume

5. Check cluster resources. If ora.mgmtdb is offline, bring it up using: srvctl start mgmtdb

Summary: Key Takeaways

Patching complex RAC environments can be unpredictable, but taking a proactive approach makes all the difference. Remember these core guidelines for your next GI RU patching cycle:

• Always update OPatch first: Using the latest build (12.2.0.10+) bypasses unnecessary requirements like wallet creation.
• Keep your inventory clean: Detach obsolete ORACLE_HOMEs and verify node lists before invoking opatchauto.
• Unmount ACFS before patching: Safely stop ACFS filesystems to prevent driver uninstallation lockups mid-patch.

Know your recovery procedure: If opatchauto fails due to busy devices, a quick CRS disable, node reboot, and opatchauto resume will save the day.