TSM

• Some tips from Oxford University
• Some useful TSM server queries
• TSM - How to determine management class and retention using dsmc - nice syntax highlighting!

Tivoli Storage Management

Default location for files

Linux, AIX
dsm.sys,dsm.opt               /opt/tivoli/tsm/client/ba/bin
dsmerror.log, dsmsched.log    /opt/tivoli/tsm/client/ba/bin

Solaris
dsm.sys,dsm.opt               /usr/tivoli/tsm/client/ba/bin
dsmerror.log, dsmsched.log    /var/log

Restart ITM Monitoring agent

To see if the Oracle module is installed

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/bin/cinfo -i

and see which agents are running

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/bin/cinfo -r

and see which agents are configured (or not)

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/bin/cinfo -o

See which instances have been setup to be monitored

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/smitools/scripts/kdyedit -h /opt/IBM/ITM list

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/smitools/scripts/kciedit -h /opt/IBM/ITM list

Add an instance to be monitored

$ITM_HOME/smitools/scripts/kdyedit -h /opt/IBM/ITM -r itmora -t rz add -i $ORACLE_SID

This one should also be done in order that rc.d scripts know to start the agent as itmora instead of root

$ITM_HOME/smitools/scripts/kciedit -h /opt/IBM/ITM -r itmora -t rz add -i $ORACLE_SID

Restart the Oracle agents

To avoid issues with orphaned processes, restart is best done with a combination of stop, cleanup, start.
Stop all Oracle agents

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/smitools/scripts/StartAgent.sh stop rz

or, the standard method...

su - itmora -c "/bin/ksh -c '/opt/IBM/ITM/bin/itmcmd agent -o TIBPFW start rz >/tmp/itmcmd_TIBPFW.log 2>&1'"

Check agent processes have stopped

ps -ef | grep [k]rz

Any remaining processes (containing a SID) should be killed.
It is normal to have agents for 61, ul and ux still running.

Check agent RunInfo file is correct
If this file is corrupted, the StartAgent.sh script does not function correctly

vi $ITM_HOME/config/.ConfigData/RunInfo

...and remove any lines containing database SIDs in the 4th column

Start all Oracle agents

$ITM_HOME/smitools/scripts/StartAgent.sh start rz

or
...using ITM standard command to restart one agent

/usr/bin/su - itmora -c "/opt/IBM/ITM/bin/itmcmd agent  -o EMREP  stop or"

Perform cleanup as above

/usr/bin/su - itmora -c "/opt/IBM/ITM/bin/itmcmd agent  -o EMREP  start or"

Check the agent is running

ps -ef | grep [k]rz

Reconfigure agent

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/bin/itmcmd config -A rz

Troubleshooting

cinfo -r reports processes not running (as well as running)

eg:

coupprod  or    25690298                    DMMPRD  ...process not running
coupprod  or    5832894                     DMMPRD  ...process not running
coupprod  ux    39452858  root    Apr23     None    ...running            
coupprod  or    43319374  itmora  Apr24     DMMPRD  ...running            

need to remove the first 2 lines...

cd /opt/IBM/ITM/config/.ConfigData
vi RunInfo

and remove the unnecessary lines

Remove / uninstall / decommission an ITM agent

Check agent is configured

export ITM_HOME=/opt/IBM/ITM
$ITM_HOME/bin/cinfo -o rz

Stop the running agent

SID=WM9TNSBX
$ITM_HOME/smitools/scripts/StartAgent.sh stop rz -o ${SID}

Delete config files for the instance

cd $ITM_HOME/config
ls -al *rz_${SID}*
rm -f *rz_${SID}*
ls -al *rz_${SID}*
ls: 0653-341 The file *rz_WM9TNSBX* does not exist.

Modify the rz environment file

vi $ITM_HOME/config/.ConfigData/krzenv

Modify the rz agent startup file

• Remove line from "start_all" section
• Remove line from "stop_all" section

vi /etc/rc.itm1

Recheck if the agent is configured (SID should have gone from list)

$ITM_HOME/bin/cinfo -o rz

Automate modification of ORACLE_HOME in config files and restart ITM agent for each database

After an upgrade, the hard-coded(!!!) path to ORACLE_HOME needs modifying to use the new location

export ITM_HOME=/opt/IBM/ITM
cd ${ITM_HOME}/config
for i in $(find . -type f -name "*_rz_*cfg"); do
    perl -p -i.bck -e 's/11.2.0.3/11.2.0.4/g' $i
    db=$(echo $i | awk -F_ '{print $NF}' | awk -F\. '{print $1}')
    /usr/bin/su - itmora -c "/bin/ksh -c '${ITM_HOME}/bin/itmcmd agent -o $db stop rz'"
    /usr/bin/su - itmora -c "/bin/ksh -c '${ITM_HOME}/bin/itmcmd agent -o $db start rz'"
done
${ITM_HOME}/bin/cinfo -r

Starting the TSM client

On Unix

dsmc -se=so_u_`hostname`_orx_d_bax

On Windows

dsmc -optfile=c:\oracle\SID\admin\dsm.opt

Send a file to TSM

dsmc archive -server=SO_U_ubenoa31_ORX_D_BAX "/oracle/export/NSWORKP1/export_NSWORKP1_D_FULL_201301030007_01.dmp"

Query the archive to see if a file is available in TSM

dsmc query archive -server=SO_U_ubenoa31_ORX_D_BAX "/oracle/export/NSWORKP1/export_NSWORKP1_D_FULL_20130103*"

Query all the files sent to the monthly node since start of FEB 2013 (using dateformat YYYY-MM-DD)

dsmc q ar -date=3 -fromdate=2013-02-01 -subdir=yes -filesonly -se=SO_U_`uname -n`_ORX_M_BAX "/oracle/export/*"

and on Windows...

dsmc query archive -dateformat=3 -fromdate=2013-02-01 -subdir=yes -filesonly -optfile='C:\home\ibmtools\etc\oracle\SO_I_WDCRHBP05_ORX_D_SOL.opt'

Note: You will get an error with the above command as Windows ignores the dateformat on the command line!
To fix it, put dateformat 3 in the opt file.

List all export jobs on this TSM node

cd /oracle/export
dsmc q ar -subdir=yes -se=SO_U_UBENOA26_ORX_D_BAX "expdp*"

Retrieve file from TSM

dsmc retrieve -pick -server=SO_U_ubenoa31_ORX_D_BAX "/oracle/export/NSWORKP1/export_NSWORKP1_D_FULL_20130103*"

This opens an interactive window listing all files that match the file query (note the wildcard)
Files can be toggled on and off for recovery by pressing the number associated with the file then recover with the letter o (for ok!).

If several versions of the same file exist, use the description field to make the distinction

dsmc retr -server=SO_U_`hostname`_ORX_D_SOL -descr="*30-01-2014*" "/oracle/export/AGP/export_AGP_D_FULL_20130103*"

Retrieve a file from TSM and save it to a different name

dsmc retr -server=SO_U_`hostname`_ORX_D_SOL /oracle/export/AGP/export_AGP_D_FULL_201301031200.dmp renamedfile.dmp

Restore a file from TSM (from backup not archive)

dsmc rest /home/bey9at77/Documents/*  /tmp/restore/ -su=yes  -inactive -pick

Check backup is performed across DWDM (diagonal) to TSM server in remote location

Some customers want the backups of the databases running in one data centre to be sent to the other data centre providing "data centre down" recovery.
Put this into a temporary file on the management server...

cat <<'EOCAT' >/tmp/tsmchk.ksh
export HN=$(hostname); for i in $(ps -ef | grep [p]mon | awk -F_ '{print $NF}'); do export SID=${i}; perl -00 -n -e 'print "$ENV{'HN'}|$ENV{'SID'}|$1\n" if m/TSM_$ENV{'SID'}_TDPO.*?erveraddress\s+([\w-]+)?/s' /etc/tsm/dsm.sys; done
EOCAT
chmod +x /tmp/tsmchk.ksh

then send it to, and run it on, all the satellite servers...

/home/ibmtools/scripts/oracle/doscp -f /tmp/tsmchk.ksh
/home/ibmtools/scripts/oracle/dosh2 -c "ksh /tmp/tsmchk.ksh"

This spits out a list of host | Instance | TSM server
If this list is pasted into a spreadsheet, a formula in Column 4 will show any bad ones (still backed up to the local data centre).

=IF( MID(A1,3,2) = (MID(C1,4,2) ) ,"<===","")

Obviously this is customer specific! Character positions 3-4 of the hostname show where the server is hosted, positions 4-5 of the TSM server name show its location.

Show details about all management classes on the TSM server

dsmc query mgmtclass -detail

Query running sessions on a TSM node

dsmc q sess -se=SO_U_SOLAX005_ORX_D_SOL

and on Windows...

dsmc q sess -optfile='C:\home\ibmtools\etc\oracle\SO_I_WDCRHBP05_ORX_D_SOL.opt'

Find the default management class on the domain

dsmc q mgmt | grep 'Default Mgmt Class Name' | awk -F' ' '{print $NF}'

Force a management class

The TSM domain has a default management class. This can be overridden in both the dsm.opt file (include options) and on the command line.
Archiving files:

dsmc archive -archmc=M14 -se=SO_U_`hostname`_ORX_D_BAX "/oracle/export/NSWORKP1/export_NSWORKP1_D_FULL_201301030007_01.dmp"

Restart LANFree Storage agent

udevdba31:root[/home/root]# ps -ef|grep dsmsta
    root 13828246        1   0   Mar 26      - 512:39 /usr/tivoli/tsm/StorageAgent/bin/dsmsta
    root 29491300 62259284   0 11:58:29  pts/1  0:00 grep dsmsta
udevdba31:root[/home/root]# kill 13828246
udevdba31:root[/home/root]# ps -ef|grep dsmsta
    root 41418862 62259284   0 11:58:48  pts/1  0:00 grep dsmsta
udevdba31:root[/home/root]# cat /etc/inittab | grep stg
autostgagnt:2:off:/usr/tivoli/tsm/StorageAgent/bin/rc.tsmstgagnt
udevdba31:root[/home/root]# at now
/usr/tivoli/tsm/StorageAgent/bin/rc.tsmstgagnt<CTRL-D>
Job root.1366365724.a will be run at Fri Apr 19 12:02:04 DFT 2013.
udevdba31:root[/home/root]# ps -ef|grep dsmsta
    root 20906202 62259284   0 12:02:28  pts/1  0:00 grep dsmsta
    root 64225302  3735904   0 12:02:16      -  0:01 dsmsta quiet
udevdba31:root[/home/root]#

Update number of mount points for a node

tsm: TSMSRV1>upd node CLUBDEV maxnummp=4
ANR2063I Node CLUBDEV updated.

Start dsmc in administrator mode

dsmadmc -id=ibmdba -pa=$DSMADMPWD -se=baxmac

Add -comma or -tab to command line in order to get the query results in a list format

dsmadmc -id=ibmdba -pa=$DSMADMPWD -se=baxmac -comma

Check TSM sessions are running Lanfree

Message number 415 shows the relevant sessions

dsmadmc -id=ibmdba -pa=$DSMADMPWD -se=baxnos -comma "select * from actlog where msgno=415 and message like '%SO_U_JDBEOP1_ORA_X_BAX%' order by DATE_TIME desc fetch first 1 rows only"

DATE_TIME: 2015-12-30 00:49:42.000000
MSGNO: 415
SEVERITY: I
MESSAGE: ANR0415I Session 1011613 proxied by STABENOUERP07MAC started for node SO_U_JDBEOP1_ORA_X_BAX. (SESSION: 1011613)

Parsing the session id out of the message, we can query the summary to find more info.

dsmadmc -id=ibmdba -pa=$DSMADMPWD -se=baxnos "select * from summary where number=1011613"

      START_TIME: 2015-12-30 00:49:42.000000
        END_TIME: 2015-12-30 00:49:44.000000
        ACTIVITY: BACKUP
          NUMBER: 1011613
          ENTITY: SO_U_JDBEOP1_ORA_X_BAX
        COMMMETH: Tcp/Ip
         ADDRESS: 10.100.119.145:61762
   SCHEDULE_NAME: 
        EXAMINED: 0
        AFFECTED: 1
          FAILED: 0
           BYTES: 262155
            IDLE: 0
          MEDIAW: 0
       PROCESSES: 1
      SUCCESSFUL: YES
     VOLUME_NAME: 
      DRIVE_NAME: 
    LIBRARY_NAME: 
        LAST_USE: 
       COMM_WAIT: 1
NUM_OFFSITE_VOLS:

Clean up export dump and log files if they are on tape

for i in `find /oracle/export -name "*dmp" -type f`; do
    sid=`echo $i | awk -F_ '{print $2}'`
    echo dsmc q ar -se=so_u_`hostname`_orx_d_bax $i 
    dsmc q ar -se=so_u_`hostname`_orx_d_bax $i >/tmp/results.$$
    if grep $i /tmp/results.$$; then
            echo rm -f $i
                 rm -f $i
    fi
done
rm /tmp/results.$$