PRAQNIQUES
=================================================================================================================
Index for this document.... Search on the sequence number in case there is more full documentation (eg. find 18.)
8E08B46DCB99DA4278C860F37A
Daily use (20120517-1012)
=================================================================================================================
01. Convert dos2unix in place - perl -i -pe 's/\r//g' <fully qualified path/name>
Convert with nawk 'sub("$", "\r")' LOGS/20111107_073010_stageing_content.txt > LOGS/20111107_073010_stageing_content.txt3
Convert with sed in perl - system(" sed -e 's/\$/\r/' ${xcontfile} > ${xattachfile} ");
01a. Convert unix2dos - perl -pi -e 's/\n/\r\n/'
Convert unix2dos - sed s/$/$'\r'/ < input.txt > output.txt
02. List dirs indented - ls -R | grep ':$' | sed -e 's/:$//' -e 's/[^-][^\/]*\//|--/g' -e 's/^/ /'
03. List the netgroups on a server - cat /etc/passwd |grep +@ |awk -F: '{print $1}' |awk -F\@ '{print $2}'
Must be on a Solaris box for this command. eg. pcredit01
EXAMPLE:
login as: as EE pwh-sas17lp2
g8udoc@pwh-sas17lp2's password:
% sudo -H -u badftpp sudo-sh; export TMOUT=0; cd /export/appl/badops/tools/dse
g8udoc's Password:
% export TMOUT=0; cd /export/appl/badops/tools/dse
% /export/appl/badops/tools/dse/WRK/getnetg.ksh dcredit-sas03lp2
04. nmon -h (help) -t for total CPU - wait for the screen to refresh as it is many CPUs
05. Display or list only files Type the following command: ls -l | grep -v '^d'
06. EMM reports: http://alsysadm-epmw05b/ssmview/filesys/index.html
07. Turn off sudo timeout once you are in sudo: export TMOUT=0
08. MUST BE ON SOLARIS psysadm50% ldaplist -l passwd m5prod |grep userPassword userPassword: {crypt}*NOLOGIN*
09. Reboot schedules found at:
http://techops/cgi-bin/autosys/reboot_jobs.pl?format=1 and
10. IP address - /usr/sbin/ifconfig -a
11. direct login - m5prod - (must be solaris) dcredit01% ldaplist -l passwd m5prod |grep userPassword (KEY IS ctl-f/ldapdcc)
userPassword: {crypt}*NOLOGIN*
12. LUN/SAN - > lsvg (result rootvg oscopyvg1 DG01_T1 DG01_T0 DG01_T3 DG01_T2 oscopyvg2
, > lsvg DG01_T2 | grep FREE
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 1
PRAQNIQUES
13. Usage investigation - du -khsa /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/*
14. Usage investigation (AIX) - du -sg /export/appl/lw_data_d/prod/lwprod/lfm/*
15. Numeric userid (solaris only - dcredit01) ldaplist -l passwd m5prod |grep idNumber
gidNumber: 8209
uidNumber: 33639
16. perl create a dir if it does not exist - unless(-d $dir){mkdir $dir or die;}
17. perl code for MD5 perl for files hash
18. Here are the command lines to test SAS - Nishit
19. Evaluate FS sizes with df - Eval this from John on 8/26
20. Creation of a lockdown doc and ticket xxx
21. ASD install issues 4/22/11
22. ps -ef |grep rsync |grep badops - check for hung async (John Simon)
23. date; pwd; ls -laR|grep -v "\-\-\- "|grep -v "total "|grep -v "vbadf" <--- get universal read/execute
24. date; uname -a ; df -gs /export/appl/lw_data_p3 <---AIX df
25. Outlook logon correction faq
26. Link to good advanced vi (Visual editer cmds) http://staff.washington.edu/rells/R110/
27. Perl output to screen AND logfile - LoadAll.sh 2>&1 | tee -a Load.log
28. Delete a file with no name or special char in the name.
First find out file inode number with any one of the following command:
stat file-name OR ls -il file-name If you cannot type the file name, just use * instead of file-name.
Use find command as follows to find and remove the specified file:
find . -inum [inode-number] -exec rm -i {} \;
29. find and list all directorys in the current directory
find . | xargs perl -e '@ARGV = grep( -d $_ , @ARGV); print "@ARGV \n"'
30. Link to Cygwin - \\fanniemae.com\CORP\DC\SHARED\DAC\CYGWIN-15X
31. Change the -mdate (last modified date) - touch -t 200012311800 employees.txt
32. rsync --dry-run -razvv /export/appl/lw_archive_p/* /export/appl/bad_archive1_p (From John Simon)
then if all looks good
nohup rsync -razvv /export/appl/lw_archive_p/* /export/appl/bad_archive1_p &
Note: remove the -z turn of the real-time verify to speed a factor of 2x
Runs ~11GB/hr on same server. Remove z to transfer raw to speed up and use cksum against directories to verify.
==================================================
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 2
PRAQNIQUES
EXAMPLE WITH OUTOUT NAMED bad_data1_w [and nohup]
rsync --dry-run -ravv /export/appl/lw_data_p2/* /export/appl/bad_data1_w
then if all looks good
nohup rsync -ravv /export/appl/lw_data_p2/* /export/appl/bad_data1_w &
To check the structure of the output:
ls -laR /export/appl/bad_data1_w/* | grep ":" | grep -v "drw" | tail -5
To watch it grow:
cd /export/appl/badops/tools/arc/
WRK/archive/watcher51.bsh bad_data1_w
33. To play with - alias jobs=` ps -e -a -o user,pid,etime,pcpu,args | grep /export/apps/sas | grep -v /utilities/bin | grep -v grep | sort`
From 17lp2 lwprod sudo alias
34. grep examples (LOTS)
35. How to get a server or dns name for and IP address - by Kate Dougherty
% nslookup <IP Address>
example:
35a. $ cp filename{,.bak} --> In this one-liner filename{,.bak} gets copied filename.bak
36. Using cut to Print out unique lines in a file
- Print out unique lines in a file
cat LOGS/CMDSRUN.txt| sort -n | uniq |head
- Print out unique lines in a file for csv selecting 2nd field
cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -nr | uniq |head
cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename
37, Get "ALL" the ldap information for userid g8udoc - by Kate Dougherty
- % ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep gid
38. =================================================================================================================
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 3
PRAQNIQUES
Global find and/or replace
find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print <--- find in the directory files that contain paul_obrecht and print them
find ./VAL/RTI.d20120913DCCbackup/ | xargs perl -pi -e 's/paul_obrecht/ /g' <--- find in the directory files that contain paul_obrecht and replace them with a blank
39. =================================================================================================================
GET A NAME FROM A USERID ON UNIX
=====================================
Check OS: uname -a
Solaris: ldaplist -l passwd g8udoc|grep cn:
AIX: getable -a passwd g8udoc
Linux: ldapsearch???
Linux:
40. #############################################################################################################
Get a nice clean recursive chmod without going outside the directory (like following links) (John Simon)
=====================================
chmod -R -h 755 /export/appl/ain_archive2_p/LPpop
41. . =============================================================================================================
Quickies:
df -g |grep add ß Check the EPF env is on the server (addprod= prod, addacpt=acpt)
EPF Schuff:
Qtr: (extention [zip or tar] MUST be the same for bert as used in fred&ernie); -t is for trace only for checking – remove for live execution
ssh -o "StrictHostKeyChecking no" g8udoc@astate-db03 ls -ltr /export/appl/addacpt/controlfile|grep G7190 |grep 20122 ß ACPT
Fred: % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -f -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t
Ernie % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -e -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t
Bert: % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -b -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t
Mon: (MON has 6 digit date, below is month 12 || QTR is 5 digits [q-1-4], above is 2nd)
ssh -o "StrictHostKeyChecking no" g8udoc@pstate-db03 ls -ltr /export/appl/addprod/controlfile|grep G7190 |grep 201212 ß PROD
Fred: % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -f -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t
Ernie % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -e -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 4
PRAQNIQUES
Bert: % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -b -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t
42. =================================================================================================================
TRANSFER A FILE WITH SCP THAT HAS A SAS GDG (INVALID # CHARACTER IN THE FILE NAME)
On the sourceserver:
% ls -il lasdbkp*
% find . -inum 13
% find . -inum 13 -exec echo {} \;
% find . -inum 13 -exec chmod 777 {} \;
% find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \;
On the target server:
% cd /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/
% find . -inum 77921 -exec mv {} dstrs_clm_draft.sas7bdat
% ls -ltr
% uname -a; pwd;cksum dstrs_clm_draft.sas7bdat
43. How 2 add an acl on a linux box========================================================================================
1) Sudo to t2runp on plgrid-ugc04 -dcc
2) cd /appl/t2prod -dcc
3) mkdir -p /appl/t2prod/mfcw/stage -dcc
4) setfacl -m u:a0prod:rwx /appl/t2prod/mfcw/stage –dcc
44. =================================================================================================================
Find if a server name is in the name server
sudo-sh[badftpp@pwh-sas17lp2]$ nslookup pwh-sas17lp2
Server:
158.137.218.75
Address:
158.137.218.75#53
Name: pwh-sas17lp2.fanniemae.com
Address: 10.204.64.27
sudo-sh[badftpp@pwh-sas17lp2]$
45. Add one of yours! =================================================================================================================
=================================================================================================================
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 5
PRAQNIQUES
=================================================================================================================
=================================================================================================================
=================================================================================================================
=================================================================================================================
=================================================================================================================
=================================================================================================================
=================================================================================================================
BELOW ARE CHRONOLOGICAL REFERENCES - (search in line number (LIKE: 38.) to see if there is more explanation)
=================================================================================================================
//////////////////////////////////////////
sudo-sh[lwftpp@pwarehouse-rsas17lp1]$ date; pwd; ls -R | grep ':$' | sed -e 's/:$//' -e 's/[^-][^\/]*\//|--/g' -e 's/^/ /'
Tue Jun 7 13:09:17 EDT 2011
/export/appl/lw_data_d/prod
|--lwprod
|--|--lfm
|--|--|--lfm_201103
|--|--|--|--basedata
|--|--|--|--inputs
|--|--|--|--outdata
|--|--|--lfm_201104
|--|--|--|--basedata
|--|--|--|--inputs
|--|--|--|--logs
|--|--|--|--|--saslog
=================================================================================================================
6/24/11 From Karuna Nidhi Chelmella(show dir sizes/usages)
sudo-sh[badftpp@tcredit01]$ pwd
/appl/fa_data_t/cw_0/input/interestRate
sudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/*
12G 2.2.CW/grp1
14G 2.2.CW/grp2
9.8G 2.2.CW/grp3
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 6
PRAQNIQUES
48G 2.2.CW/grp4
76G 2.2.CW/nat
sudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/nat/*|grep 201106
3.1G 2.2.CW/nat/201106
0K 2.2.CW/nat/201106IS
0K 2.2.CW/nat/201106IS_binary_suse
0K 2.2.CW/nat/201106IS_binary_unix
sudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/nat/*|grep 201105
5.7G 2.2.CW/nat/201105
746M 2.2.CW/nat/201105IS
670M 2.2.CW/nat/201105IS_binary_suse
670M 2.2.CW/nat/201105IS_binary_unix
sudo-sh[badftpp@tcredit01]$ du -khsa /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/*
12G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp1
14G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp2
9.8G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp3
48G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp4
76G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/nat
sudo-sh[badftpp@tcredit01]$
=================================================================================================================
09. 6/10/11 From John Simon...
Clarke, Dale C [5:12 PM]:
John is there a central log for weekend reboots that are public?
Simon, John V [5:13 PM]:
Autosys: http://techops/cgi-bin/autosys/reboot_jobs.pl?format=1
Clarke, Dale C [5:13 PM]:
thx
Simon, John V [5:13 PM]:
VCS: http://uxweb/Policies/FailoverSchedules/VCS-Failover-Schedule.htm
Clarke, Dale C [5:15 PM]:
=================================================================================================================
6/5/11 From John Simon...
How to see if it has direct login - m5prod
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 7
PRAQNIQUES
psysadm50% ldaplist -l passwd m5prod |grep userPassword
userPassword: {crypt}*NOLOGIN*
//////////
See if a user id has dierect native login.
ldaplist -l passwd <userid> |grep userPassword
For example, if direct login is disabled it will look like this:
psysadm04% ldaplist -l passwd vnothp |grep userPassword
userPassword: {crypt}*NOLOGIN*
If direct login is NOT disabled there will be an encrypted string:
psysadm04% ldaplist -l passwd g2ujas |grep userPassword
userPassword: {crypt}CMvrgwerav49sj
//////////
=================================================================================================================
6/5/11 From John Simon...
Quick question... Do u use putty? If so is there a NO TMOUT type command like the sudo
Answer:
sudo -H -u m5prod sudo-sh
export TMOUT=0
command
=================================================================================================================
6/3/11 From John Simon...
See if a user id has dierect native login.
ldaplist -l passwd <userid> |grep userPassword
For example, if direct login is disabled it will look like this:
psysadm04% ldaplist -l passwd vnothp |grep userPassword
userPassword: {crypt}*NOLOGIN*
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 8
PRAQNIQUES
If direct login is NOT disabled there will be an encrypted string:
psysadm04% ldaplist -l passwd g2ujas |grep userPassword
userPassword: {crypt}CMvrgwerav49sj
5/30/11
=================================================================================================================
Dale,
These answers are pretty straight forward.
1. Q: What is the AD access that should requested here?
ANSWER: since you have to write to the folder, you will need, Read/Write (-RW, .FC, etc)
2.1. Q: From the listing Karuna sent how would I know the access needed;
ANSWER: Again pretty easy to figure out: The security group aligns EXACTLY up with the AD PATH
- Since the AD PAth is: .... fanniemae.com\corp\DC\Shared\Finance\SFFinanceBAD\HPF
This means the associated security group would be: SG-DC-Shared-Finance-SFFinanceBAD*
NOTE: the SG in front means "Security Group"
While in my prifile I requested to add access to this path and pulled up the following:
- SG-DC-SHARED-FINANCE-SFFINANCEBAD-FC
2.2 Q: i.e. what alias?
ANSWER: The alias that is required is already in the HPFA Phase 2 runbook: > cd SFFINANCEBAD/HPF
This means that you can infer the alias is: SFFINANCEBAD and translates to: \\fanniemae.com\corp\DC\Shared\Finance\SFFinanceBAD
3. Q: Should we all request it?
ANSWER: Yes, at least ALL Engineers
5/24/11
=================================================================================================================
From: Simon, John V
Sent: Wednesday, May 18, 2011 11:04 AM
To: Chen, Jennifer; Hadavi, Behrooz; Elsinger, James A; Milner, Christopher (Contractor)
Cc: Mahensaria, Udit (Contractor); Karayi, Sanjith K
Subject: RE: HPFA Production Security with SFTP
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 9
PRAQNIQUES
Here is an example of how to generate, convert and run this. In this case it was UNIX to UNIX. The only difference for AD would be to take the converted public key and have it
added to the user you are logging into on the AD side.:
$ pwd
/tmp/g2ujas
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/g2ujas/.ssh/id_rsa): /tmp/g2ujas/id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /tmp/g2ujas/id_rsa.
Your public key has been saved in /tmp/g2ujas/id_rsa.pub.
The key fingerprint is:
85:ef:52:b1:5e:72:bc:bd:01:93:7f:65:8b:e1:c6:ed g2ujas@pstate-etl04
$ ls -la
total 64
drwxr-xr-x 2 g2ujas admstaff 247 May 18 10:52 .
drwxrwxrwt 7 root sys
1912 May 18 10:51 ..
-rw------- 1 g2ujas admstaff 1675 May 18 10:52 id_rsa
-rw-r--r-- 1 g2ujas admstaff 401 May 18 10:52 id_rsa.pub
$ ssh-keygen -e -f id_rsa.pub > fsecure.pub
$ cat fsecure.pub
---- BEGIN SSH2 PUBLIC KEY ---Comment: "2048-bit RSA, converted from OpenSSH by g2ujas@pstate-etl04"
AAAAB3NzaC1yc2EAAAABIwAAAQEA0JntIWU2eC4WhoP6ES/hzpe/Js5lHpilh0JH3EDvt9
YKxP5c3oXalzGZyxgG1o7cQlWfPtKB4hcNOEnwCvrkm0ZuceLbDpvDO2MakCvtVcmIeLa9
xFhZ0A/apTc0wSPiJ5fJUdZj+c6lmI9OQWxcpOUYFDdjtPRXlNnPjt3xVlDyxm0RWz0PQH
bx3ff1hhHXrKMhH+YE2O0OdvTzsMZ0FKpExGubtK0Wx3wAkB/S0Ns4KpVxq3wtUvqpg7U0
1OYuyVN25bzkEgpAyRhFNO1woaLymD7HKfVrhuVx/cwt3PbOsnJsbV8yZSlu2ZiEiDztcX
UoqoWZow1mfmfVNYOZmw==
---- END SSH2 PUBLIC KEY ---$ cat id_rsa.pub
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA0JntIWU2eC4WhoP6ES/hzpe/Js5lHpilh0JH3EDvt9YKxP5c3oXalzGZyxgG1o7cQlWfPtKB4hcNOEnwCvrkm0ZuceLbDpvDO2MakCvtVcmIeLa9xF
hZ0A/apTc0wSPiJ5fJUdZj+c6lmI9OQWxcpOUYFDdjtPRXlNnPjt3xVlDyxm0RWz0PQHbx3ff1hhHXrKMhH+YE2O0OdvTzsMZ0FKpExGubtK0Wx3wAkB/S0Ns4KpVxq3wtUvqpg7U01OY
uyVN25bzkEgpAyRhFNO1woaLymD7HKfVrhuVx/cwt3PbOsnJsbV8yZSlu2ZiEiDztcXUoqoWZow1mfmfVNYOZmw== g2ujas@pstate-etl04
$ ls -la
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 10
PRAQNIQUES
total 96
drwxr-xr-x 2 g2ujas admstaff 449 May 18 10:53 .
drwxrwxrwt 7 root sys
1912 May 18 10:53 ..
-rw-r--r-- 1 g2ujas admstaff
0 May 18 10:52 commercial.pub
-rw-r--r-- 1 g2ujas admstaff 75 May 18 10:56 config
-rw-r--r-- 1 g2ujas admstaff 511 May 18 10:52 fsecure.pub
-rw------- 1 g2ujas admstaff 1675 May 18 10:52 id_rsa
-rw-r--r-- 1 g2ujas admstaff 401 May 18 10:52 id_rsa.pub
$ cat config
IdentityFile /tmp/g2ujas/id_rsa
UserKnownHostsFile /tmp/g2ujas/known_hosts
$ sftp -F /tmp/g2ujas/config g2ujas@dsysadm15
Connecting to dsysadm15...
Could not create directory '/home/g2ujas/.ssh'.
The authenticity of host 'dsysadm15 (10.210.68.15)' can't be established.
RSA key fingerprint is f2:ec:6d:6b:13:2a:3f:ec:9a:85:54:34:2a:d2:cb:79.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dsysadm15,10.210.68.15' (RSA) to the list of known hosts.
sftp> ls -la
drwxrwxr-x 23 g2ujas admstaff 2048 Apr 18 16:54 .
dr-xr-xr-x 31 root root
31 May 18 10:55 ..
drwxr-xr-x 3 g2ujas admstaff
96 Jan 14 2008 .NetscapeCalendar
-rw------- 1 g2ujas admstaff 594 Jun 21 2008 .TTauthority
-rw------- 1 g2ujas admstaff
55 Aug 4 2010 .Xauthority
-rw-r--r-- 1 g2ujas admstaff 451 Sep 2 2004 .Xdefaults
drwx------ 2 g2ujas admstaff
96 Oct 25 2004 .acrobat
-rw------- 1 g2ujas admstaff 237 Oct 25 2004 .acrosrch
drwx------ 2 g2ujas admstaff
96 Oct 25 2004 .adobe
-rw------- 1 root other
10479 Apr 7 17:03 .bash_history
-rw-r--r-- 1 g2ujas admstaff 859 Mar 29 2010 .cshrc
-rw-r--r-- 1 g2ujas admstaff 1731 Nov 24 2004 .desksetdefaults
drwxr-xr-x 14 g2ujas admstaff 1024 Jun 25 2008 .dt
5/12/11
=================================================================================================================
_____________________________________________
From: Perera, Anthony H
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 11
PRAQNIQUES
Sent: Thursday, May 12, 2011 4:38 PM
To: Chelmella, Karuna Nidhi; Clarke, Dale C
Cc: Elsinger, James A; BA-D Operations
Subject: How to check diskspace on a server
<k2uahp@pcredit-sas01lp2> 21: lsvg
rootvg
DG01_T1
DG01_T2
DG01_T3
oscopyvg1
oscopyvg2
<k2uahp@pcredit-sas01lp2> 22:
<k2uahp@ccredit-sas01lp2> 25: lsvg DG01_T2| grep FREE
MAX LVs:
4096
FREE PPs:
780 (49920 megabytes)
<k2uahp@ccredit-sas01lp2> 26:
21. ASD install issues 4/22/11
=================================================================================================================
From: Beutler, Joshua (Contractor)
Sent: Friday, April 22, 2011 11:20 AM
To: Clarke, Dale C
Subject: ASD install issues
Dale,
If you have any other install issues relating to drive space, try this fix and see if it helps.
Go to Start > Settings > Control Panel > System Management > Advanced Tab.
Increase the size of the temporary program download folder.
If you have any other questions please feel free to contact the 7s.
Thanks!
Josh
CSC
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 12
PRAQNIQUES
4/21/11 ROM Flash for 8440p?
=================================================================================================================
HP Notebook System BIOS Update (ROM Family 68CCU)
From: Teates, Mike B (Contractor)
Sent: Thursday, April 21, 2011 2:16 PM
To: Teates, Mike B (Contractor); Clarke, Dale C
Subject: Conversation with Teates, Mike B (Contractor), Clarke, Dale C
Teates, Mike B (Contractor) [2:09 PM]:
http://bizsupport1.austin.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=321957&prodSeriesId=4095872&swItem=ob-854861&prodNameId=4095873&swEnvOID=4060&swLang=13&taskId=135&mode=4&idx=3
20. 3/16/11 Creation of a lockdown doc and ticketxxx
=================================================================================================================
From: Elsinger, James A
Sent: Wednesday, March 16, 2011 2:02 AM
To: Madduri, Anuradha X (Contractor); Jones, Annette X
Cc: Rogensues, Gary (Contractor); Perera, Anthony H; Clarke, Dale C
Subject: Plan for RPS Production environment setup and LOCKDOWN
Importance: High
Anu/Annette,
Can one of you please submit an OPERATIONAL, LOW Risk Change ticket to support a Production Lockdown of the RPS production environment. Please plan the lockdown for
3/17 0800 -7/231700. Attached is the DRAFT version of the full lockdown instructions
The full path of the RPS buildout in is the Draft Runbook. However, below is the full list:
Server: pcredit-sas01lp2
/sastemp_rps1_p SAS Temp*
1
/export/appl/rpsprod
Code/Scripts
/export/appl/rpsprod/bin
Code/Scripts
/export/appl/rps_data1_p Data Space
/export/appl/rps_data2_p Data Space
/export/appl/rps_archive1_p Archive Space
250
2
2
2
2
3
rpsprod
10
10
500
250
500
rpsprod
rpsprod
vbadfp
rpsprod
rpsprod
rpsprod
Unix Praqniques (Practical Techniques)by Dale C Clarke
1750
rpsprod
rpsprod
rpsprod
rpsprod
rpsprod
755
750
750
750
750
Page 13
PRAQNIQUES
Long window Justification: Additional time is needed to support a pre-production shakeout of this new EUC.
Low Risk Justification: There is currently no executions occurring in this new RPS production area.
The details of the ticket should be the information noted in the DRAFT Production Runbook that related to the EUC setup; mkdirs, transferring ANY staged data, the setting up of
Oracle interface as well as review of FS and Directory Ownership and Permissions to align with BADOps standard practices.
Gary,
Please verify the rpsprod SUDO is working properly on the production server.
Elsinger, James A
BA&D Operations Manager
Analytics, Finance & Controls
(202) 520-1889
vvvvvvvv=========================================================================================================
Operational Remedy Change Ticket - Detailed Description:
===========================================
Purpose: This Operational Change ticket is to prepare the <EUC> environment for controlled process by managing ownership and permissions. This ticket is also to verify and
ensure the input data for <EUC> can be successfully read or transferred from interfaces servers in associated SLA’s and/or the server list as noted below for current month data
or setup of historical data.
LONG WINDOW JUSTIFICATION: This lockdown process is scheduled to run into a weekend and could therefore take 5-6 days.
LOW RISK JUSTIFICATION: This change does not require downtime or reboot. No user sessions or applications are impacted.
Hosts affected:
================
pcredit-sas01lp2,
<Additional Servers see Final Draft Runbook>
Please verify you can sudo to both the OPS Transfer ID as well as the EUC Prod Execution ID <
SETUP INSTRUCTIONS
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 14
PRAQNIQUES
===============================================
- As necessary, make sure the Execution Engineer (EE) is logged into pcredit-sas01lp2 and sudo-ed as necessary to RPS Prod Execution ID: rpsprod
Alternate: sudo-ed to OPS ID <badftpp> to support file transfers…
Verification:
NON PROD: > sftp <EE ID>@dcredit-sas01lp1
PROD: > sftp <EE ID>@ pcredit-sas01lp2
1.) Build structure
Create the following sub-folder in <EUC> AD structure:
Example: \\fanniemae.com\corp\dc\shared\RE07\PROD\GAGOValSingleFam\HPFA\2009Q3_200912\Data
Create the FS and Directory Structure (SEE Runbook line ??
Example:
As necessary, make sure that the Execution Engineer is logged into pwh-sas17lp2 and sudo-ed to
> login /usr/SYSADM/bin/ssh to pwh-sas17lp2<using Execution Engineers own ID>
> sudo -H -u badftpp sudo-sh
> mkdir /export/appl/<APPLCODE>_data_p/input
> mkdir /export/appl/<APPLCODE>_data_p/output
> mkdir /export/appl/<APPLCODE>_data_p/input/
> mkdir /export/appl/<APPLCODE>_data_p/output/200912
> quit
2.) Transfer all “One Time” Staged data (SEE Runbook line ?? <If Applicable>
3.) Adjust / Configure permissions and ownership for both Data and Code areas to ensure structure and data files are in lockdown.
- Ensure the permissions and ownership are correct as they are noted in the Infrastructure REQUIREMENTS document
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 15
PRAQNIQUES
LOCKDOWN INSTRUCTIONS
===================================
Parameter: <APPLCODE> = “rps”
***3a.) CODE
- In General: Code area Settings: ID = vbadfp, Group - <APPLCODE>prod
Instructions
====================
Log in to pwh-sas17lp2 using CM ID (vbadfp) and cd /export/appl/<APPLCODE>prod
Ensure CM ID owns all code (vbadfp)
- Code transfers should be completed with CM ID
NOTE: Please change the group ownership of code from faptcm (IF present) to EUC UNIX group to "<APPLCODE>prod" on Production and <APPLCODE>acpt on Acceptance of
the code areas to be in line with standard practices. <--CONFIRMED (listing in the Activity tab)
- chgroup as necessary to <APPLCODE>prod permissions - 750 (ID = RW, Group should be Read Only, No universal
- NOTE: If there was any permission or ownership issues, pass to SCM Team to review the deployment script and make these adjustments as well.
Code Area Verification:
- Once complete with vbadfp owner ls -laR /export/appl/<APPLCODE>prod/<code area>
o
Example: /export/appl/rpsprod/bin
* The code directory should only be 750 Recursive and NOT contain Universal Read/Write or Execute.
- cd to path /export/appl/<APPLCODE>prod
- all sub directory permissions should be 750 unless otherwise noted.
***3b.) DATA
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 16
PRAQNIQUES
o
o
In General:
Input data file Settings: ID = OPS Transfer ID (badftpp), chgroup should be <APPLCODE>prod or <APPLCODE>acpt in acceptance permissions – 640
Output and result data file Settings: ID = OPS EUC Execution ID (rpsprod), chgroup should be <APPLCODE>prod or <APPLCODE>acpt in acceptance permissions – 640
Instructions
====================
Log in to pwh-sas17lp2 using rpsprod and cd /export/appl/<APPLCODE>_data_p or <APPLCODE>_data_a in acceptance
Data Area Verification:
- Ownership of the data input directory should be the OPS ID (badftpp).
If OPS ID is not the owner of the input directory, sudo to the ID that owns the directory and make a ".d" copy of the current one (ex. d20090928)
then exit out of sudo and sudo into the OPS ID. Once sudoed with badftpp, mkdir a new inputs directory and chgrp to <APPLCODE>prod
NOTE Ensure you copy all previous files in the backups directory back to the new inputs directory.
- Once complete ls -ltr (ls -laR?) while in path /export/appl/<APPLCODE>_data_p/input/200909
- Permissions should be 640 for all files, same should be for output as well
- cd to path /export/appl/<APPLCODE>_data_p
- FS permissions on ALL data directories should be 750
- Ownership of the data input directory should be the OPS ID (badftpp) with group-<APPLCODE>prod.
- Owner of output directory should be the Appl ID (<APPLCODE>prod) with group = <APPLCODE>prod
4.) Confirm accessibility of the stages input files in both rps_curr and prev_mth
BADOps will confirm that all of the required input files can be transferred:
*** Ensure that the files can be read Permissions 644 or group readable and available for transfer during transfers.
4a.) As necessary confirm data can be written to the locked down input directory by verifying the availability and successful transfer of the Input Data. Please consult BADOps
Manager for step 3a and 3b
Unix to Unix <See pre-execution Runbook steps>
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 17
PRAQNIQUES
NOTE EE will drop a foo test file as necessary on UNIX file destination path
Unix to AD < If applicable> <See pre-execution Runbook steps>
NOTE Alias for the AD path will be required and needs to be verified. EE will drop a foo test file as necessary
4b). if necessary: Save a copy of the current input file
3c) Assure access and that the Unix and AD accesses are correct through all lines of the Runbook. Due to the aggressive schedule this may not be possible until the execution.
Mitigation should then be recorded in the Errata column of the Runbook.
The validating of the transfer lines and will take 4-6 hours or more
4) Create/verify the execution code for all the transfers to be done and add them to the final draft runbook. This is VERY resource intensive. Due to the aggressive schedule this
may not be possible until the execution. Code development should then be recorded in the Errata column of the runbook
*** Ensure that the files can be read (Permissions 640 or group readable) and available for transfer during executions.
5.) Notification:
Once all validation steps are complete, BADOps Rep will notify CreditSupp Manager (Jim Elsinger) and BADOps Mailbox.
- OPS to coordinate with RPS as well as input source systems and Downstream systems like FAVM Business
NOTE: Keep this change ticket open until consumer has verified the data and structure is correct.
Shakeout:
========
ls -la <filename>
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 18
PRAQNIQUES
Backout:
=======
rm <filename>
Test / Verification Result:
============
All verification results and adjustments will be noted in the worklog. Also all adjust to Runbook steps will be noted in the Runbook Errata Column.
^^^^^^=================================================================================================================
=================================================================================================================
Please note, CORELOGIC
CLIENT
CONTACT
LOGIN NAME:
P455W0RD
Fannie Mae
James Elsinger
jelsinger
u85EGJ9a
Fannie Mae
Dale C Clarke
dclarke
HJKcXH96
Fannie Mae
Anthony H Perera
aperera
85qq9WAe
Fannie Mae
Gary Rogensues
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 19
PRAQNIQUES
grogensues
bL5SqtSk
From Anthony 6/17/11
=================================================================================================================
Get a LUN or SAN free space
<s8ujsm@pcredit-sas01lp3> 1: lsvg
rootvg
oscopyvg1
DG01_T1
DG01_T0
DG01_T3
DG01_T2
oscopyvg2
<s8ujsm@pcredit-sas01lp3> 2: lsvg DG01_T2 | grep FREE
MAX LVs: 4096 FREE PPs: 1010 (64640 megabytes)
^^^^^^=================================================================================================================
From Dale Password ldap check password must be on solaris xcredit01
Key is ldapdcc
login as: g8udoc
g8udoc@dcredit01's password:
Last login: Fri Jul 8 12:33:21 2011 from 10.136.197.253
FANNIE MAE MANAGEMENT APPROVED PURPOSES ONLY
**** SYSTEM ACTIVITIES MAY BE MONITORED ****
<g8udoc@dcredit01> 2: uname -a
SunOS dcredit01 5.10 Generic_142900-03 sun4u sparc SUNW,Netra-T12
<g8udoc@dcredit01> 3: bash
bash-3.00$ ldaplist -l passwd auqprod |grep userPassword
userPassword: {crypt}*NOLOGININITIAL*
bash-3.00$ ldaplist -l passwd badftpp |grep userPassword
userPassword: {crypt}YM2U7.ddJ8R5g
bash-3.00$
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 20
PRAQNIQUES
=================================================================================================================
17. MD5 perl for files hash
sudo-sh[badftpp@pwarehouse-rsas17lp1]$ csum -h MD5 ./md5sum.tst
00ba45f9cbd9dba978eea6ce97dfd82a ./md5sum.tst
sudo-sh[badftpp@pwarehouse-rsas17lp1]$ ./md5sum.pl ./md5sum.tst
00ba45f9cbd9dba978eea6ce97dfd82a ./md5sum.tst
sudo-sh[badftpp@pwarehouse-rsas17lp1]$ cat ./md5sum.tst
somethingsomething
sudo-sh[badftpp@pwarehouse-rsas17lp1]$ cat ./md5sum.pl
#! /usr/bin/perl
use Digest::MD5;
use IO::File;
my $chk = Digest::MD5->new();
foreach my $file (@ARGV)
{
$chk->addfile(IO::File->new($file));
print "",$chk->hexdigest," $file \n";
}
sudo-sh[badftpp@pwarehouse-rsas17lp1]$
=================================================================================================================
18. Here are the command lines to test SAS 1. Running with auqprod UID
> login /usr/bin/ssh to cwh-sas17lp2
> sudo -H -u auqprod sudo-sh
> cd /export/appl/auqprod
> /export/apps/sas_9.1/sas -noterminal -is 'proc setinit; run ;' /dev/null -log /tmp/test.log
> cat /tmp/test.log
> rm -f /tmp/test.log
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 21
PRAQNIQUES
2. Running with any other UID
> login /usr/bin/ssh to cwh-sas17lp2
> /export/apps/sas_9.1/sas -noterminal -work /tmp -sasuser /tmp -is 'proc setinit; run ;' /dev/null -log /tmp/test.log
> cat /tmp/test.log
> rm -f /tmp/test.log
a) If you see following error, SAS does not have execute permission
The file access permissions do not allow the specified action..
b) Change SAS command as below if we agree to use contingency SAS installation as it is (not changing to match production) /export/apps/sas_9.1.3_sp4/sas
=================================================================================================================
19. Eval this from John on 8/26
=================================================================================================================
sudo-sh[root@pcredit-sas01lp2]# ps -ef |egrep '450720|598016|610452|725028|761996|958636|1126516|1159178|1581270'
root 450720 508016 0 Aug 14 - 8:56 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_481272/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 483346 610452 0 Aug 19 - 1:05 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
root 528608 1581270 0 Aug 21 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
root 573674 450720 0 Aug 14 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
root 598016 1163346 0 16:46:46 pts/4 0:00 sudo-sh
root 610452 1941684 0 Aug 19 - 8:50 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_622246/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 655576 761996 0 Aug 15 - 1:05 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
root 680144 958636 0 06:00:28 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 22
PRAQNIQUES
root 725028 442586 0 Aug 23 - 9:05 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1150717/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 761996 393404 0 Aug 15 - 8:39 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_769173/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 958636 1691750 0 06:00:28 - 8:47 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1966601/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 971004 1126516 0 Aug 17 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
root 1126516 892972 0 Aug 17 - 8:55 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1128165/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 1143014 725028 0 Aug 23 - 1:09 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete .
"/export/appl/badops"
badftpp 1159178 1872024 0 15:04:10 pts/3 0:00 sudo-sh
root 1581270 995518 0 Aug 21 - 8:46 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1596486/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops"
root 1736720 598016 0 16:54:14 pts/4 0:00 egrep 450720|598016|610452|725028|761996|958636|1126516|1159178|1581270
root 1822966 598016 7 16:54:14 pts/4 0:00 ps -ef
sudo-sh[root@pcredit-sas01lp2]# kill 450720 483346 528608 573674 610452 655576 680144 725028 761996 958636 971004 1126516 1143014 1581270
sudo-sh[root@pcredit-sas01lp2]# ps -ef |egrep '450720|598016|610452|725028|761996|958636|1126516|1159178|1581270'
root 442590 598016 1 16:55:20 pts/4 0:00 egrep 450720|598016|610452|725028|761996|958636|1126516|1159178|1581270
root 598016 1163346 1 16:46:46 pts/4 0:00 sudo-sh
root 725046 598016 7 16:55:20 pts/4 0:00 ps -ef
badftpp 1159178 1872024 0 15:04:10 pts/3 0:00 sudo-sh
sudo-sh[root@pcredit-sas01lp2]# df -k |grep badop
/dev/badops 31457280 31450428 1% 100 1% /export/appl/badops
sudo-sh[root@pcredit-sas01lp2]#
================================================================================================================================================
==================================================================================
24. sudo-sh[lwftpp@pwh-sas17lp2]$
sudo-sh[lwftpp@pwh-sas17lp2]$ date; uname -a ; df -gs /export/appl/lw_data_p3
Mon Oct 3 16:33:09 EDT 2011
AIX pwh-sas17lp2 3 5 00C371A04C00
Filesystem GB blocks Free* %Used Iused %Iused Mounted on
/dev/lw_data_p3 500.00 148.06 71% 3641 1% /export/appl/lw_data_p3
sudo-sh[lwftpp@pwh-sas17lp2]$
=================================================================================================================
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 23
PRAQNIQUES
25. When you get a dialog box when connecting to Outlook the has a User prompt looking like EXSW01NS/<your userID> requesting a password (which it then rejects) try the
fillowing.
Change the EXSW01NS/<your userID> to fanniemae/<your userID>, enter your correct password. It should then connect. This works when opening Communicator as well.
This happens most often when your connection has been lost or when changing from wired to wireless but may have other causes.
=================================================================================================================
TOOLS:
sudo-sh[badftpp@pwarehouse-rsas17lp1]$ pwd; ls -ltr *.ksh
/export/appl/badops/tools/devl
-rwx------ 1 badftpp badftpp
2706 Oct 12 18:17 ckperlver.ksh
-rwx------ 1 badftpp badftpp
2195 Oct 12 18:20 cktoolsize.ksh
sudo-sh[badftpp@pwarehouse-rsas17lp1]$
=================================================================================================================
34. grep examples:
SPECIFIC ONE LINERS:
######## Find if there are ANY files in a directory, recursively that contain data:
Where file/dir,
- not owned or grouped by lwftpp or lwprod
- not an laR total line, colon title, or blank
ls -laR /export/appl/badops/export|grep -v '[lwftpp|lwprod]'|grep -v total|grep -v ":"|grep -v '^$'|wc
% Tells if there are ANY non-zero size directories: du /export/appl/badops/export|grep -v '^0'|wc
######## An easy, one liner command to replace a test string in multiple files in a sub-directory using recursion.
% grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<chgword>/g'
***CAUTION BACK UP YOUR DIRECTORY BEFORE MAKING THE ERROR YOU CERTAINLY WILL ((-8
OR Use the perl oneliner that provides the option of creating a backup of each file changed replacing the sed:
% perl -p -i.bak -e ’s/<oldword>/<chgword>/g’ *
Note: In both cases some characters require negation with the backslash (experiment)
GENERAL:
% grep -v '^$' - grep OUT blank lines
% Search vivek in any case (i.e. case insensitive search): grep -i -w vivek /etc/passwd
% Search vivek or raj in any case: grep -E -i -w 'vivek|raj' /etc/passwd
% Match Vivek or vivek:grep '[vV]ivek' filename
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 24
PRAQNIQUES
OR grep '[vV][iI][Vv][Ee][kK]' filename for any upper/lower in all positions
% You can also match digits (i.e match vivek1 or Vivek2 etc): grep -w '[vV]ivek[0-9]' filename
OR You can match two numeric digits (i.e. match foo11, foo12 etc): grep 'foo[0-9][0-9]' filename
% You are not limited to digits, you can match at least one letter: grep '[A-Za-z]' filename
% Display all the lines containing either a "w" or "n" character: grep [wn] filename
% Display any lines starting with a dot and digit: grep '^\.[0-9]' filename
% Match a character "v" two times in a line: egrep "v{2}" filename
% Match both "col" and "cool": egrep 'co{1,2}l' filename
% Match any row of at least three letters 'c': egrep 'c{3,}' filename
% Search directories recursively: grep -r 'hello' /home/gigi
% The -l option print will only the filename (e.g., /etc/crontab, /etc/group) if the file contains a hit: grep -l root /etc/*
Can be used to create an array for a command to process.
CLASS brackets:
Within a bracket expression, the name of a character class enclosed in "[:" and ":]" stands for the list of all characters belonging to that class. Standard character class names
are:
[:alnum:] - Alphanumeric characters.
[:alpha:] - Alphabetic characters
[:blank:] - Blank characters: space and tab.
[:digit:] - Digits: '0 1 2 3 4 5 6 7 8 9'.
[:lower:] - Lower-case letters: 'a b c d e f g h i j k l m n o p q r s t u v w x y z'.
[:space:] - Space characters: tab, newline, vertical tab, form feed, carriage return, and space.
[:upper:] - Upper-case letters: 'A B C D E F G H I J K L M N O P Q R S T U V W X Y Z'.
In this example match all upper case letters: grep '[:upper:]' filename
WILDCARDS (Dot):
You can use the "." for a single character match. In this example match all 3 character word starting with "b" and ending in "t":
% grep '\<b.t\>' filename
Where,
The Dot matches ANY character
\< Match the empty string at the beginning of word
\> Match the empty string at the end of word.
% Print all lines with exactly two characters: grep '^..$' filename
ESCAPING (backslash before a character):
% In an IP search all three dots need to be escaped: grep '192\.168\.1\.254' /etc/hosts
% The following example will only match an IP address: egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' filename
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 25
PRAQNIQUES
% The following will match word Linux or UNIX in any case: egrep -i '^(linux|unix)' filename
% Search for all lines matching '--test--' using -e option.
Without -e, grep would attempt to parse '--test--' as a list of options: grep -e '--test--' filename
Anchors:
% grep -v '^$' - grep OUT blank lines
% Match line only containing foo: grep '^foo$' filename
% Find lines ending with word foo: grep 'foo$' filename
% Display only lines starting with the word vivek only i.e. do not display vivekgite, vivekg etc: grep -w ^vivek /etc/passwd
%
%
ls -laR /export/appl/lw_archive_p2|grep -v lwftpp|grep -v lwprod|head grep -v total|grep -v ":"|wc
=================================================================================================================
35. How to get a server or dns name for and IP address - Dougherty, Kate [11:46 PM]:
just run nslookup <IP Address>
Example:
bash-2.03$ nslookup 10.204.64.21
Server: psysadm-dn07.fanniemae.com
Address: 158.137.218.75
Name: pwarehouse-rsas17lp1.fanniemae.com
Address: 10.204.64.21
bash-2.03$
=================================================================================================================
35a. $ cp filename{,.bak} --> In this one-liner filename{,.bak} gets copied filename.bak
This one-liner copies the file named filename to a file named filename.bak. Here is how it works.
It uses brace expansion to construct a list of arguments for the cp command. Brace expansion is a mechanism by which arbitrary strings may be generated.
In this one-liner filename{,.bak} gets brace expanded to filename filename.bak and puts in place of the brace expression. The command becomes cp filename filename.bak and
file gets copied.
=================================================================================================================
36. Print out unique lines in a file
- Print out unique lines in a file
cat LOGS/CMDSRUN.txt| sort -n | uniq |head
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 26
PRAQNIQUES
- Print out unique lines in a file for csv selecting 2nd field
cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -nr | uniq |head
cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -n | uniq |head
Simplified: cut -d',' -f 2 <file> | sort -nu
MORE:
bash$ cat list-1 list-2 list-3 | sort | uniq > final.list
# Concatenates the list files,
# sorts them,
# removes duplicate lines,
# and finally writes the result to an output file.
bash$ cat testfile
This line occurs only once.
This line occurs twice.
This line occurs twice.
This line occurs three times.
This line occurs three times.
This line occurs three times.
bash$ uniq -c testfile
1 This line occurs only once.
2 This line occurs twice.
3 This line occurs three times.
bash$ sort testfile | uniq -c | sort -nr
3 This line occurs three times.
2 This line occurs twice.
1 This line occurs only once.
SCRIPT:
#!/bin/bash
# wf.sh: Crude word frequency analysis on a text file.
# This is a more efficient version of the "wf2.sh" script.
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 27
PRAQNIQUES
# Check for input file on command-line.
ARGS=1
E_BADARGS=85
E_NOFILE=86
if [ $# -ne "$ARGS" ] # Correct number of arguments passed to script?
then
echo "Usage: `basename $0` filename"
exit $E_BADARGS
fi
if [ ! -f "$1" ]
# Check if file exists.
then
echo "File \"$1\" does not exist."
exit $E_NOFILE
fi
########################################################
# main ()
sed -e 's/\.//g' -e 's/\,//g' -e 's/ /\
/g' "$1" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr
#
=========================
#
Frequency of occurrence
# Filter out periods and commas, and
#+ change space between words to linefeed,
#+ then shift characters to lowercase, and
#+ finally prefix occurrence count and sort numerically.
# Arun Giridhar suggests modifying the above to:
# . . . | sort | uniq -c | sort +1 [-f] | sort +0 -nr
# This adds a secondary sort key, so instances of
#+ equal occurrence are sorted alphabetically.
# As he explains it:
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 28
PRAQNIQUES
# "This is effectively a radix sort, first on the
#+ least significant column
#+ (word or string, optionally case-insensitive)
#+ and last on the most significant column (frequency)."
#
# As Frank Wang explains, the above is equivalent to
#+
. . . | sort | uniq -c | sort +0 -nr
#+ and the following also works:
#+
. . . | sort | uniq -c | sort -k1nr -k
########################################################
exit 0
bash$ cat testfile
This line occurs only once.
This line occurs twice.
This line occurs twice.
This line occurs three times.
This line occurs three times.
This line occurs three times.
bash$ ./wf.sh testfile
6 this
6 occurs
6 line
3 times
3 three
2 twice
1 only
1 once
1. cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename
2. It is even possible to specify a linefeed as a delimiter if apropos.
The trick is to actually embed a linefeed (RETURN) in the command sequence.
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 29
PRAQNIQUES
bash$ cut -d'
' -f3,7,19 testfile
This is line 3 of testfile.
This is line 7 of testfile.
This is line 19 of testfile.
37. =================================================================================================================
bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc
Could not create directory '/home/g8udoc/.ssh'.
dn: uid=g8udoc,ou=people,ou=corporate,dc=fanniemae,dc=com
shadowLastChange: 15555
fmpasswordresetdate: 20120803005933Z
userPassword: {crypt}d5Q9tVme3qYb2
fmDepartment: Enterprise Technology Svcs
fmdivision: Technology
fmbusinessunit: Ent Data Technology
fmcostcenter: 485
fmProfileOwner: 1
roomNumber: 2450B
physicalDeliveryOfficeName: 2h-2n/14
fmManagerLevel: Associates
fmservicedate: 02/28/2011 12:00:00 AM EST
fmjoblevel: 5
title: Applctn Support Analyst III
fmlegalname: Dale C Clarke
fmemployeestatus: A
fmPrivilegedID: No
fmSharedID: No
employeeNumber: 07887
employeeType: Employee
fmofficelocation: 4000 Wisconsin Avenue
fmfloor: Second
telephoneNumber: 202-752-3465
shadowMin: -1
shadowFlag: 0
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 30
PRAQNIQUES
shadowExpire: -1
shadowInactive: -1
shadowMax: -1
shadowWarning: 7
homeDirectory: /home/g8udoc
uidNumber: 46496
gidNumber: 5069
gecos: Dale C Clarke
loginShell: /bin/csh
mailAlternateAddress: g8udoc@fanniemae.com
mailRoutingAddress: dale_c_clarke@exchange.fanniemae.com
mail: dale_c_clarke@fanniemae.com
mailHost: exchange.fanniemae.com
cn: Dale C Clarke
fmmarketroom: No
fmpeoplemanager: 0
fmContractorType: Regular
fmbusinesscritical: No
fmkiosk: No
fmmanageruid: g8ujxe
fmstatus: Active
manager: James Elsinger
initials: C
fmisterminated: No
fmregion: dc
uid: g8udoc
givenName: Dale
objectClass: inetorgperson
objectClass: organizationalPerson
objectClass: person
objectClass: top
objectClass: fannieMaePerson
objectClass: mailRecipient
objectClass: posixAccount
objectClass: shadowAccount
objectClass: account
sn: Clarke
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 31
PRAQNIQUES
bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep UID
Could not create directory '/home/g8udoc/.ssh'.
bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep uid
Could not create directory '/home/g8udoc/.ssh'.
dn: uid=g8udoc,ou=people,ou=corporate,dc=fanniemae,dc=com
uidNumber: 46496
fmmanageruid: g8ujxe
uid: g8udoc
bash-2.05b$
38. =================================================================================================================
Find in a directory all occurrences of "paul_obrecht“ and print; then replace all occurrences, then print again to show there are none found.
//////////////////////////////// example of the first file in the print below before it is edited
sudo-sh[badftpp@pwh-sas17lp2]$ cat ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES
#|gnum|G9999|
#|dsesrc|RTI|
#|dsesrcsrvr|pwh-sas17lp2|
#|dsedest|"M&A RESEARCH"|
#|dsedestsrvr|pcredit-sas01lp3|
#|dsedistro|"paul_obrecht daniel_dausman dean_m_henderson af-c_operations ba-d_operations"|
#|dsesubject|"TRANSFERS COMPLETED SUCCESSFULLY"|
#|dseerrorsubject|"TRANSFERS FAILED"|
#|dsebody|"PLEASE SEE THE ATTACHED FILE FOR COMPLETE TRANSFER DETAILS."|
#|dseendnote|"Please contact af-c_operations@fanniemae.com, if you have any questions or concerns regarding this transfer."|
#|dseattachmentT|TRANSFER.log|
#|dseattachmentE|ERROR.log|
sudo-sh[badftpp@pwh-sas17lp2]$
//////////////////////////////////////////
sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print ß find - found 5 occurrences
#|dsedistro|"paul_obrecht daniel_dausman dean_m_henderson af-c_operations ba-d_operations"|
./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES
#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|
./VAL/RTI.d20120913DCCbackup/rtit2300.1_g6941_rti-ldm.listN
#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 32
PRAQNIQUES
./VAL/RTI.d20120913DCCbackup/rtit2300.2_g6941_rti-ldm.listN
#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|
./VAL/RTI.d20120913DCCbackup/rtit2300.3_g6941_rti-ldm.listN
#|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"|
./VAL/RTI.d20120913DCCbackup/rtit2300.4_g6941_rti-ldm.listN
sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ | xargs perl -pi -e 's/paul_obrecht/ /g' ßfind and change in place perl command
Can't do inplace edit: ./VAL/RTI.d20120913DCCbackup/ is not a regular file.
sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print ß find – found none
sudo-sh[badftpp@pwh-sas17lp2]$ ß None found
//////////////////////////////// example of the first file after edited
sudo-sh[badftpp@pwh-sas17lp2]$ cat ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES
#|gnum|G9999|
#|dsesrc|RTI|
#|dsesrcsrvr|pwh-sas17lp2|
#|dsedest|"M&A RESEARCH"|
#|dsedestsrvr|pcredit-sas01lp3|
#|dsedistro|" daniel_dausman dean_m_henderson af-c_operations ba-d_operations"| ß Notice the extra space – could be a replacement name
#|dsesubject|"TRANSFERS COMPLETED SUCCESSFULLY"|
#|dseerrorsubject|"TRANSFERS FAILED"|
#|dsebody|"PLEASE SEE THE ATTACHED FILE FOR COMPLETE TRANSFER DETAILS."|
#|dseendnote|"Please contact af-c_operations@fanniemae.com, if you have any questions or concerns regarding this transfer."|
#|dseattachmentT|TRANSFER.log|
#|dseattachmentE|ERROR.log|
sudo-sh[badftpp@pwh-sas17lp2]$
39. =================================================================================================================
GET A NAME FROM A USERID ON UNIX
=====================================
Solaris:
bash-3.00$ ldaplist -l passwd <userID> |grep cn:
cn: Dale C Clarke
bash-3.00$
AIX:
bash-2.05b$ getable -a passwd <userID>
g8udoc:*NP*:46496:5069:Dale C Clarke:/home/g8udoc:/bin/csh
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 33
PRAQNIQUES
Name...........: Dale C Clarke
Title..........:
Employee Type..: Employee
Telephone No...: 202-752-3465
Manager........: James Elsinger
bash-2.05b$
Linux: Not yet found
42. =================================================================================================================
50. TRANSFER A FILE WITH SCP THAT HAS A SAS GDG
////////////////////////
sudo-sh[ainprod@pwh-sas17lp2]$ cd /export/appl/ain_archive_p/LASD/
sudo-sh[ainprod@pwh-sas17lp2]$ ls -il
total 34331200
18 -rwxr-xr-x 1 ainprod ainprod 1539776512 Jul 02 2010 dstrs_clm_draft.sas7bdat_bad100702
10 -rw-r--r-- 1 ainprod ainprod 2587697152 Aug 03 11:05 lasdbkp#048.sas7bdat
11 -rw-r--r-- 1 ainprod ainprod 2622562304 Sep 03 09:46 lasdbkp#049.sas7bdat
12 -rw-r--r-- 1 ainprod ainprod 2658934784 Oct 03 09:47 lasdbkp#050.sas7bdat
13 -rw-r--r-- 1 ainprod ainprod 2690850816 Nov 03 09:58 lasdbkp#051.sas7bdat
15 -rw-r--r-- 1 ainprod ainprod 2723553280 Dec 03 09:56 lasdbkp#052.sas7bdat
17 -rw-r--r-- 1 ainprod ainprod 2754224128 Jan 03 08:35 lasdbkp.sas7bdat
sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec cksum {} \;
1009243769 2690850816./lasdbkp#051.sas7bdat
sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \;
badftpp@dcredit-sas02lp1's password:
sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \;
badftpp@dcredit-sas02lp1's password:
Could not chdir to home directory /appl/badftpp: A file or directory in the path name does not exist.
lasdbkp#051.sas7bdat
100% 2597MB 40.6MB/s 01:04
sudo-sh[ainprod@pwh-sas17lp2]$
RENAME ON THE TARGET SERVER:
bash-3.2$ cd /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 34
PRAQNIQUES
bash-3.2$ ls -il
total 5255568
77921 -rw-r--r-- 1 badftpp badftpp 2690850816 Jan 11 13:24 lasdbkp#051.sas7bdat
bash-3.2$ uname -a; pwd;find . -inum 77921 -exec cksum {} \;
AIX dcredit-sas02lp1 1 6 00F62A394C00
/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD
1009243769 2690850816 ./lasdbkp#051.sas7bdat
bash-3.2$ find . -inum 77921 -exec mv {} dstrs_clm_draft.sas7bdat
bash-3.2$ ls -ltr
total 5255568
-rw-r--r-- 1 badftpp badftpp 2690850816 Jan 11 13:24 dstrs_clm_draft.sas7bdat
bash-3.2$ uname -a; pwd;cksum dstrs_clm_draft.sas7bdat \;
AIX dcredit-sas02lp1 1 6 00F62A394C00
/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD
1009243769 2690850816 dstrs_clm_draft.sas7bdat
bash-3.2$
43. =================================================================================================================
HOW-2 find man pages
sudo-sh[ainprod@pwh-sas17lp2]$ apropos acl
aclconvert(1) - Converts the access control information of a file system object from onetype to another.
acledit(1) - Edits the access control information of a file.
aclget(1)
- Displays the access control information of a file.
aclgettypes(1) - Gets ACL types supported by a file system path.
aclput(1)
- Sets the access control information of a file.
ctaclfck(1) - Verifies the contents of a clustersecurity services ACL file.
sudo-sh[ainprod@pwh-sas17lp2]$ apropos aclput
aclput(1)
- Sets the access control information of a file.
sudo-sh[ainprod@pwh-sas17lp2]$ man aclput
=================================================================================================================
=================================================================================================================
Unix Praqniques (Practical Techniques)by Dale C Clarke
Page 35