Patching Oracle Database Appliance (ODA)

From dbawiki
Revision as of 14:18, 27 January 2017 by Stuart (talk | contribs) (→‎Check NFS mounts are working)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Useful ODA documents:


Bundle patches for ODA are installed in 3 stages.
Upto and including 12.1.2.6, this is performed as so:

oakcli update -patch 12.1.2.6.0 --infra     # patches the server firm/software
oakcli update -patch 12.1.2.6.0 --gi        # patches the Grid (ASM/ACFS) firm/software
oakcli update -patch 12.1.2.6.0 --database  # patches the ORACLE_HOME(s) software

From 12.1.2.7 onwards this has changed to:

oakcli update -patch 12.1.2.9.0 --server    # patches the server and Grid firm/software
oakcli update -patch 12.1.2.9.0 --storage   # patches the Shared storage including HDD, SSD, Controller, and Expander
oakcli update -patch 12.1.2.9.0 --database  # patches the ORACLE_HOME(s) software

Contents

Patch preparation[edit]

Show version details of all ODA components
Note: all oakcli commands must be executed as root 

/opt/oracle/oak/bin/oakcli show version -detail

Patching ODA means patching the O/S, the Grid Infrastructure (GI) and the RDBMS (or any combination thereof).

Download latest ODA patch bundle[edit]

Latest ODA patches are found in Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)

Download and copy zip files[edit]

Download the files on the relevant patch page for your hardware, copy them to wherever you store patches on the server and Read the README.

Copy zip files to /tmp[edit]

Copy the zip files from your patch storage area to /tmp on both ODA_BASE nodes.

Check space[edit]

Make sure there is enough space in /root, /tmp, $ORACLE_HOME, and /opt for the unzippped / installed files.

Verify ODA version[edit]

/opt/oracle/oak/bin/oakcli show version

Unpack zip files[edit]

Do not use unzip! 

cd /tmp
oakcli unpack –package p25064836_121290_Linux-x86-64_1of2.zip 
oakcli unpack –package p25064836_121290_Linux-x86-64_2of2.zip

Delete zips from /tmp[edit]

If unpack -package was successful, files can be removed from /tmp. 

rm -rf /tmp/p25064836_121290_Linux-x86-64*

Check if the patch is likely to work![edit]

The following command will inform you whether the patch will succeed. If the output indicates a possible problem, for example, an unpacked version of the patch is not available on the system, you may wish to defer applying the patch until you have downtime to fix the expected problem.
From Node 0: 

oakcli validate -c ospatch -ver <version to be installed>
e.g.
oakcli validate -c ospatch -ver 12.1.2.9.0

Remove any non-standard RPMs[edit]

If the validation check reveals any RPMs, they must be removed prior to patching.
Note: Perform check on both nodes, there may be differences! 

rpm -e --nodeps libtermcap-devel.x86_64
rpm -e --nodeps yum-downloadonly.noarch
rpm -e --nodeps gpg-pubkey
rpm -e --nodeps check_mk-agent.noarch
rpm -e --nodeps readline-devel.x86_64

Backup sysctl.conf[edit]

This file will probably be modified by any server patches 

export DATESTAMP=`date +%Y%m%d`
cp /etc/sysctl.conf /etc/sysctl.conf_${DATESTAMP}

Patch the ODA infrastructure[edit]

The oakcli patching process has component specific logging enabled for each component being patched.
These log files can be found under the directory /opt/oracle/oak/log/$(hostname)/patch/12.1.2.9.0

Review size of shared repository[edit]

oakcli show repo

and add space if necessary 

oakcli configure repo srepo -incsize 100G

Stop any running jobs[edit]

Before updating the repository, agent, server, or database, ensure that there will be no jobs running or pending during the update window.

Ready for patching...[edit]

Perform any specifics such as

  • Set database in blackout in OEM
  • Any other monitoring software?
  • Ensure archivelogs are not deleted. They will be needed for application on physical standby later.
  • anything else that could be a problem.

For example, here we use rlwrap for command line editing so these aliases will no longer work as we just uninstalled the readline package so comment them for now. 

cd
vi .bash_profile
#alias sqlplus='rlwrap -D2 -ic sqlplus'
#alias dgmgrl='rlwrap -D2 -ic dgmgrl'
#alias rman='rlwrap -D2 -ic rman'
#alias asmcmd='rlwrap -D2 -ic asmcmd'
#alias lsnrctl='rlwrap -D2 -ic lsnrctl'
#alias adrci='rlwrap -D2 -ic adrci'
#alias impdp='rlwrap -D2 -ic impdp'
#alias expdp='rlwrap -D2 -ic expdp'

If these commands are going to be used, they will need unaliasing before use! e.g. 

unalias lsnrctl

Stop the Snap Management Utility[edit]

/opt/oracle/smu/bin/smu stop

Shutdown any virtual machines[edit]

oakcli stop vm odavmdp1 -force
oakcli stop vm odavmdp2 -force
oakcli stop vm odavmdb3 -force
oakcli stop vm odavmoms1 -force

Shutdown Grid agent / Listeners[edit]

export ORACLE_SID=AGENT
ORAENV_ASK=NO
. oraenv
emctl stop agent
export TNS_ADMIN=$ORACLE_HOME/network/admin/
lsnrctl stop LISTENER_DATAGUARD

Shutdown and backup ODA_BASE[edit]

Note: Do this on both nodes! 

oakcli stop oda_base

cd /OVS
tar -cvzf  oakDom1.odavm01-rac.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1

Optonally reconfigure ODA_BASE[edit]

oakcli configure oda_base

Restart ODA_BASE[edit]

After tarball has been successfully created, ODA_BASE can be restarted. 

oakcli start oda_base

Stop the shared repositories[edit]

This was done before applying GI patch but as server patch now includes GI, stop repositories before applying server patch. 

oakcli stop repo srepo
oakcli show repo

List the components that require patching[edit]

If any component shows as “Up-to-date” for the supported version, then it indicates that the existing version on the system is current and that component will not be patched. 

From node 0:
/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --verify

Apply the ODA infrastructure (server) patch[edit]

Nodes can be patched one at a time by using the -local switch
If both nodes are to be patched automatically (rolling patch), do not specify -local switch but command must then be executed from node 0. 

From node 0:
/opt/oracle/oak/bin/oakcli update -patch <patch bundle version> --server -local
e.g.
/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --server -local

Note: If using '-local' switch, do not run this command simultaneously on both nodes!

Run post server patch patch checks[edit]

Check cluster is working ok. 

su - grid
export ORACLE_SID=+ASM1
ORAENV_ASK=NO
. oraenv
${ORACLE_HOME}/bin/crsctl check crs
/opt/oracle/oak/bin/oakcli show version -detail

Check NFS mounts are working[edit]

List mount points that should be mounted 

grep nfs /etc/fstab | egrep -e "^#" -v | wc -l

List mount points that are mounted 

df -hPt nfs | tail -n +2 | wc -l

Patch the Grid infrastructure[edit]

Note: As of 12.1.2.7.0, this step is included in the '--server' patch

Stop the shared repositories[edit]

oakcli stop repo srepo
oakcli show repo

Apply the GI patch[edit]

Note: As of 12.1.2.7.0, this step is included in the '--server' patch

Carefully read the known issues before applying the GI Patch. 

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.6.0 --gi

Check cluster health[edit]

${ORACLE_HOME}/grid/bin/crsctl check crs

Patch the Shared Storage[edit]

Note: This section is new as of 12.1.2.7.0

Stop the shared repositories[edit]

oakcli stop repo srepo
oakcli show repo

Apply the Storage patch[edit]

Note: The storage patching has no '-local' option. It must start from node 0, node 0 is patched first followed by the node 1. Both nodes will be rebooted.
From node 0: 

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --storage

Check cluster health[edit]

${ORACLE_HOME}/grid/bin/crsctl check crs

Check/change some file permissions[edit]

su - grid
export ORACLE_SID=+ASM1
ORAENV_ASK=NO
. oraenv
ls -al ${ORACLE_HOME}/bin/osdbagrp ${ORACLE_HOME}/lib/acfsreplcrs.pl
chmod 755 ${ORACLE_HOME}/bin/osdbagrp ${ORACLE_HOME}/lib/acfsreplcrs.pl

Restore sysctl.conf[edit]

Check for higher values of variables.

Patch the RDBMS[edit]

Apply the database patch[edit]

Once the infrastructure and GI patches (or server and storage) have been completed successfully, the RDBMS patches can be applied.
It can be applied locally on each node or as a rolling upgrade and both nodes will be patched automatically.
If '-local' is not specified, the command must be run from node 0. 

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0  --database <-local>

Verify database versions[edit]

oakcli show dbhomes -detail
oakcli show databases -detail

Post ODA patch activities[edit]

Restart any physical standby databases[edit]

Restart RAC cluster[edit]

svrctl start database

Verify all ODA components are now up-to-date[edit]

/opt/oracle/oak/bin/oakcli show version -detail

Restart any virtual machines[edit]

oakcli start vm odavmdp1
oakcli start vm odavmdp2
oakcli start vm odavmdb3
oakcli start vm odavmoms1

Remount ZFS volumes on virtual machines[edit]

Log in to each of the VMs in turn 

sudo mount /orasoft
sudo mount /orabackups/RMAN
sudo mount /orabackups/datapump_hist

Restart OMS[edit]

login to OMS server 

cd /u01/OracleHomes/Middleware/oms/bin
./emctl status oms
./emctl start oms

Check ODA Grid agent[edit]

Kill any old processes and restart agent 

export ORACLE_SID=agent
ORAENV_ASK=NO
. oraenv
ps -ef | grep emagent
kill -9 <pid where agent is 12.1.0.6>
emctl start agent

Reinstall any previously uninstalled RPMs[edit]

Check for later versions?

  • Nagios plugin
cd /orasoft/rpm
yum install check-mk-agent-1.2.6p14-e23d6a3d4540a28a.noarch.rpm
  • Oracle Snap Management Utility
cd /orasoft/oracle_snapmanager/oracle-smu-1.2.0-ga/Linux/noarch/
rpm -i oracle-smu-1.2.0-27.noarch.rpm
/opt/oracle/smu/bin/smu start
  • other
yum install libtermcap-devel.x86_64
yum install yum-downloadonly.noarch
yum install gpg-pubkey
yum install readline-devel.x86_64

Uncomment any specific Oracle commands[edit]

Like these aliases setup for readline wrapping 

alias sqlplus='rlwrap -D2 -ic sqlplus'
alias dgmgrl='rlwrap -D2 -ic dgmgrl'
alias rman='rlwrap -D2 -ic rman'
alias asmcmd='rlwrap -D2 -ic asmcmd'
alias lsnrctl='rlwrap -D2 -ic lsnrctl'
alias adrci='rlwrap -D2 -ic adrci'
alias impdp='rlwrap -D2 -ic impdp'
alias expdp='rlwrap -D2 -ic expdp'

Uncomment any jobs that clean up archivelogs[edit]

Take databases out of blackout on OEM[edit]

...and any other monitoring software

Check public keys[edit]

Patching may destroy public keys 

ssh admin@oda02-rac -p 8002

If password is requested, keys are corrupt.

When it all goes tits up[edit]

infra patch fails with[edit]

[root@odavm02-rac bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'odavm02-rac'
CRS-2673: Attempting to stop 'ora.crsd' on 'odavm02-rac'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'odavm02-rac'
CRS-2679: Attempting to clean 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac'
CRS-2680: Clean of 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac' failed
CRS-2799: Failed to shut down resource 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'odavm02-rac' has failed
CRS-2675: Stop of 'ora.crsd' on 'odavm02-rac' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'odavm02-rac'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'odavm02-rac' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
  • Solution

Recreate STAGING volume on ODA_BASE. 

[grid@odavm01-rac ~]$ asmcmd
ASMCMD> volcreate -G DATA -s 1024G DBSTAGING
ASMCMD> volinfo -G DATA  DBSTAGING
Diskgroup Name: DATA

         Volume Name: DBSTAGING
         Volume Device: /dev/asm/dbstaging-358
         State: ENABLED
         Size (MB): 1048576
         Resize Unit (MB): 32
         Redundancy: MIRROR
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage:
         Mountpath:


[grid@odavm01-rac ~]$ /sbin/mkfs -t acfs /dev/asm/dbstaging-358
mkfs.acfs: version                   = 11.2.0.4.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/dbstaging-358
mkfs.acfs: volume size               = 1099511627776
mkfs.acfs: Format complete.


[grid@odavm01-rac ~]$ /sbin/acfsutil registry -a -f /dev/asm/dbstaging-358 /staging
acfsutil registry: mount point /staging successfully added to Oracle Registry


[grid@odavm01-rac ~]$ /sbin/acfsutil registry -l /staging
Device : /dev/asm/dbstaging-358 : Mount Point : /staging : Options : none : Nodes : all : Disk Group : DATA : Volume : DBSTAGING


[root@odavm01-rac ~]# /bin/mount -t acfs /dev/asm/dbstaging-358 /staging

[root@odavm01-rac /]# chown oracle:oinstall /staging

[root@odavm01-rac ~]# df -h /staging
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/dbstaging-358
                      1.0T  2.2G 1022G   1% /staging

infra patch fails with[edit]

INFO: Running  prepatching on node 1
ERROR  : Ran '/usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra' and it returned code(1) and output is:
 INFO: 2015-10-30 21:57:52: checking avaiable free space on /tmp, /u01 and /opt
 WARNING: 2015-10-30 21:57:52: Free space on / is less than 3G.
 INFO: 2015-10-30 21:57:52: Please free up space on /.
 WARNING: 2015-10-30 21:57:52: Free space on / is less than 3G.
 INFO: 2015-10-30 21:57:52: Please free up space on /.
 ERROR: 2015-10-30 21:57:52: One or more volumes are running low on free space
 INFO: 2015-10-30 21:57:52: Please look at the log /opt/oracle/oak/log/odavm02-rac/patch/12.1.2.5.0/prepatch_5974.log for more details

error at<Command = /usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra> and errnum=<1>
ERROR  : Command = /usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra did not complete successfully. Exit code 1 #Step -1#
Exiting...
ERROR: Failed to apply the patch
  • Solution

Clean the / directory and ensure at least 3G available!

gi patch fails with[edit]

[root@odavm01-rac ~]# /opt/oracle/oak/bin/oakcli update -patch  12.1.2.5.0 --gi
WARNING: One or more shared repos are running. They need to be stopped before patching the GI
You have new mail in /var/spool/mail/root
[root@sdtcsynoda01-rac ~]# oakcli show repo

          NAME                          TYPE            NODENUM  FREE SPACE     STATE           SIZE
          odarepo1                      local           0               N/A     N/A             N/A
          odarepo2                      local           1               N/A     N/A             N/A
          srepo                         shared          0            14.88%     ONLINE          307200.0M
          srepo                         shared          1            14.88%     ONLINE          307200.0M
  • Solution

Stop the shared repositories 

[root@odavm01-rac ~]# oakcli stop repo srepo

Successfully stopped SharedRepo srepo on node 1
Successfully stopped SharedRepo srepo on node 0

[root@odavm01-rac ~]# oakcli show repo

          NAME                          TYPE            NODENUM  FREE SPACE     STATE           SIZE
          odarepo1                      local           0               N/A     N/A             N/A
          odarepo2                      local           1               N/A     N/A             N/A
          srepo                         shared          0               N/A     OFFLINE         N/A
          srepo                         shared          1               N/A     OFFLINE         N/A

If Grid agent does not restart[edit]

[oracle@odavm02-rac SH]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Status agent Failure:unable to connect to http server at https://odavm02-rac.localwan.net:3872/emd/lifecycle/main/. [peer not authenticated]


[oracle@odavm02-rac SH]$ emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent status could not be determined.  Check the agent process
Consult emctl.log and emagent.nohup in: /u01/app/oracle/agent/12.1.0.4/agent_inst/sysman/log
  • Problem

Grid agent was probably started from wrong ORACLE_HOME

  • Solution

Kill any old agent processes and restart 

ps -ef | grep emagent
kill -9 <emagent processes>
emctl start agent
Retrieved from "https://mailnest.com/dbawiki/index.php?title=Patching_Oracle_Database_Appliance_(ODA)&oldid=2583"