PRAQNIQUES ================================================================================================================= Index for this document.... Search on the sequence number in case there is more full documentation (eg. find 18.) 8E08B46DCB99DA4278C860F37A Daily use (20120517-1012) ================================================================================================================= 01. Convert dos2unix in place - perl -i -pe 's/\r//g' <fully qualified path/name> Convert with nawk 'sub("$", "\r")' LOGS/20111107_073010_stageing_content.txt > LOGS/20111107_073010_stageing_content.txt3 Convert with sed in perl - system(" sed -e 's/\$/\r/' ${xcontfile} > ${xattachfile} "); 01a. Convert unix2dos - perl -pi -e 's/\n/\r\n/' Convert unix2dos - sed s/$/$'\r'/ < input.txt > output.txt 02. List dirs indented - ls -R | grep ':$' | sed -e 's/:$//' -e 's/[^-][^\/]*\//|--/g' -e 's/^/ /' 03. List the netgroups on a server - cat /etc/passwd |grep +@ |awk -F: '{print $1}' |awk -F\@ '{print $2}' Must be on a Solaris box for this command. eg. pcredit01 EXAMPLE: login as: as EE pwh-sas17lp2 g8udoc@pwh-sas17lp2's password: % sudo -H -u badftpp sudo-sh; export TMOUT=0; cd /export/appl/badops/tools/dse g8udoc's Password: % export TMOUT=0; cd /export/appl/badops/tools/dse % /export/appl/badops/tools/dse/WRK/getnetg.ksh dcredit-sas03lp2 04. nmon -h (help) -t for total CPU - wait for the screen to refresh as it is many CPUs 05. Display or list only files Type the following command: ls -l | grep -v '^d' 06. EMM reports: http://alsysadm-epmw05b/ssmview/filesys/index.html 07. Turn off sudo timeout once you are in sudo: export TMOUT=0 08. MUST BE ON SOLARIS psysadm50% ldaplist -l passwd m5prod |grep userPassword userPassword: {crypt}*NOLOGIN* 09. Reboot schedules found at: http://techops/cgi-bin/autosys/reboot_jobs.pl?format=1 and 10. IP address - /usr/sbin/ifconfig -a 11. direct login - m5prod - (must be solaris) dcredit01% ldaplist -l passwd m5prod |grep userPassword (KEY IS ctl-f/ldapdcc) userPassword: {crypt}*NOLOGIN* 12. LUN/SAN - > lsvg (result rootvg oscopyvg1 DG01_T1 DG01_T0 DG01_T3 DG01_T2 oscopyvg2 , > lsvg DG01_T2 | grep FREE Unix Praqniques (Practical Techniques)by Dale C Clarke Page 1 PRAQNIQUES 13. Usage investigation - du -khsa /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/* 14. Usage investigation (AIX) - du -sg /export/appl/lw_data_d/prod/lwprod/lfm/* 15. Numeric userid (solaris only - dcredit01) ldaplist -l passwd m5prod |grep idNumber gidNumber: 8209 uidNumber: 33639 16. perl create a dir if it does not exist - unless(-d $dir){mkdir $dir or die;} 17. perl code for MD5 perl for files hash 18. Here are the command lines to test SAS - Nishit 19. Evaluate FS sizes with df - Eval this from John on 8/26 20. Creation of a lockdown doc and ticket xxx 21. ASD install issues 4/22/11 22. ps -ef |grep rsync |grep badops - check for hung async (John Simon) 23. date; pwd; ls -laR|grep -v "\-\-\- "|grep -v "total "|grep -v "vbadf" <--- get universal read/execute 24. date; uname -a ; df -gs /export/appl/lw_data_p3 <---AIX df 25. Outlook logon correction faq 26. Link to good advanced vi (Visual editer cmds) http://staff.washington.edu/rells/R110/ 27. Perl output to screen AND logfile - LoadAll.sh 2>&1 | tee -a Load.log 28. Delete a file with no name or special char in the name. First find out file inode number with any one of the following command: stat file-name OR ls -il file-name If you cannot type the file name, just use * instead of file-name. Use find command as follows to find and remove the specified file: find . -inum [inode-number] -exec rm -i {} \; 29. find and list all directorys in the current directory find . | xargs perl -e '@ARGV = grep( -d $_ , @ARGV); print "@ARGV \n"' 30. Link to Cygwin - \\fanniemae.com\CORP\DC\SHARED\DAC\CYGWIN-15X 31. Change the -mdate (last modified date) - touch -t 200012311800 employees.txt 32. rsync --dry-run -razvv /export/appl/lw_archive_p/* /export/appl/bad_archive1_p (From John Simon) then if all looks good nohup rsync -razvv /export/appl/lw_archive_p/* /export/appl/bad_archive1_p & Note: remove the -z turn of the real-time verify to speed a factor of 2x Runs ~11GB/hr on same server. Remove z to transfer raw to speed up and use cksum against directories to verify. ================================================== Unix Praqniques (Practical Techniques)by Dale C Clarke Page 2 PRAQNIQUES EXAMPLE WITH OUTOUT NAMED bad_data1_w [and nohup] rsync --dry-run -ravv /export/appl/lw_data_p2/* /export/appl/bad_data1_w then if all looks good nohup rsync -ravv /export/appl/lw_data_p2/* /export/appl/bad_data1_w & To check the structure of the output: ls -laR /export/appl/bad_data1_w/* | grep ":" | grep -v "drw" | tail -5 To watch it grow: cd /export/appl/badops/tools/arc/ WRK/archive/watcher51.bsh bad_data1_w 33. To play with - alias jobs=` ps -e -a -o user,pid,etime,pcpu,args | grep /export/apps/sas | grep -v /utilities/bin | grep -v grep | sort` From 17lp2 lwprod sudo alias 34. grep examples (LOTS) 35. How to get a server or dns name for and IP address - by Kate Dougherty % nslookup <IP Address> example: 35a. $ cp filename{,.bak} --> In this one-liner filename{,.bak} gets copied filename.bak 36. Using cut to Print out unique lines in a file - Print out unique lines in a file cat LOGS/CMDSRUN.txt| sort -n | uniq |head - Print out unique lines in a file for csv selecting 2nd field cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -nr | uniq |head cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename 37, Get "ALL" the ldap information for userid g8udoc - by Kate Dougherty - % ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep gid 38. ================================================================================================================= Unix Praqniques (Practical Techniques)by Dale C Clarke Page 3 PRAQNIQUES Global find and/or replace find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print <--- find in the directory files that contain paul_obrecht and print them find ./VAL/RTI.d20120913DCCbackup/ | xargs perl -pi -e 's/paul_obrecht/ /g' <--- find in the directory files that contain paul_obrecht and replace them with a blank 39. ================================================================================================================= GET A NAME FROM A USERID ON UNIX ===================================== Check OS: uname -a Solaris: ldaplist -l passwd g8udoc|grep cn: AIX: getable -a passwd g8udoc Linux: ldapsearch??? Linux: 40. ############################################################################################################# Get a nice clean recursive chmod without going outside the directory (like following links) (John Simon) ===================================== chmod -R -h 755 /export/appl/ain_archive2_p/LPpop 41. . ============================================================================================================= Quickies: df -g |grep add ß Check the EPF env is on the server (addprod= prod, addacpt=acpt) EPF Schuff: Qtr: (extention [zip or tar] MUST be the same for bert as used in fred&ernie); -t is for trace only for checking – remove for live execution ssh -o "StrictHostKeyChecking no" g8udoc@astate-db03 ls -ltr /export/appl/addacpt/controlfile|grep G7190 |grep 20122 ß ACPT Fred: % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -f -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Ernie % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -e -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Bert: % ./run_epf.pl -a=20122 -g=G7190 -runenv=acpt -runper=QTR -b -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Mon: (MON has 6 digit date, below is month 12 || QTR is 5 digits [q-1-4], above is 2nd) ssh -o "StrictHostKeyChecking no" g8udoc@pstate-db03 ls -ltr /export/appl/addprod/controlfile|grep G7190 |grep 201212 ß PROD Fred: % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -f -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Ernie % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -e -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t Unix Praqniques (Practical Techniques)by Dale C Clarke Page 4 PRAQNIQUES Bert: % ./run_epf.pl -a=201212 -g=G7190 -runenv=prod -runper=MON -b -ovrfile=/export/appl/rps_data1_p/input/BVDB/AVM-RPS_20122.zip -t 42. ================================================================================================================= TRANSFER A FILE WITH SCP THAT HAS A SAS GDG (INVALID # CHARACTER IN THE FILE NAME) On the sourceserver: % ls -il lasdbkp* % find . -inum 13 % find . -inum 13 -exec echo {} \; % find . -inum 13 -exec chmod 777 {} \; % find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \; On the target server: % cd /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ % find . -inum 77921 -exec mv {} dstrs_clm_draft.sas7bdat % ls -ltr % uname -a; pwd;cksum dstrs_clm_draft.sas7bdat 43. How 2 add an acl on a linux box======================================================================================== 1) Sudo to t2runp on plgrid-ugc04 -dcc 2) cd /appl/t2prod -dcc 3) mkdir -p /appl/t2prod/mfcw/stage -dcc 4) setfacl -m u:a0prod:rwx /appl/t2prod/mfcw/stage –dcc 44. ================================================================================================================= Find if a server name is in the name server sudo-sh[badftpp@pwh-sas17lp2]$ nslookup pwh-sas17lp2 Server: 158.137.218.75 Address: 158.137.218.75#53 Name: pwh-sas17lp2.fanniemae.com Address: 10.204.64.27 sudo-sh[badftpp@pwh-sas17lp2]$ 45. Add one of yours! ================================================================================================================= ================================================================================================================= Unix Praqniques (Practical Techniques)by Dale C Clarke Page 5 PRAQNIQUES ================================================================================================================= ================================================================================================================= ================================================================================================================= ================================================================================================================= ================================================================================================================= ================================================================================================================= ================================================================================================================= BELOW ARE CHRONOLOGICAL REFERENCES - (search in line number (LIKE: 38.) to see if there is more explanation) ================================================================================================================= ////////////////////////////////////////// sudo-sh[lwftpp@pwarehouse-rsas17lp1]$ date; pwd; ls -R | grep ':$' | sed -e 's/:$//' -e 's/[^-][^\/]*\//|--/g' -e 's/^/ /' Tue Jun 7 13:09:17 EDT 2011 /export/appl/lw_data_d/prod |--lwprod |--|--lfm |--|--|--lfm_201103 |--|--|--|--basedata |--|--|--|--inputs |--|--|--|--outdata |--|--|--lfm_201104 |--|--|--|--basedata |--|--|--|--inputs |--|--|--|--logs |--|--|--|--|--saslog ================================================================================================================= 6/24/11 From Karuna Nidhi Chelmella(show dir sizes/usages) sudo-sh[badftpp@tcredit01]$ pwd /appl/fa_data_t/cw_0/input/interestRate sudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/* 12G 2.2.CW/grp1 14G 2.2.CW/grp2 9.8G 2.2.CW/grp3 Unix Praqniques (Practical Techniques)by Dale C Clarke Page 6 PRAQNIQUES 48G 2.2.CW/grp4 76G 2.2.CW/nat sudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/nat/*|grep 201106 3.1G 2.2.CW/nat/201106 0K 2.2.CW/nat/201106IS 0K 2.2.CW/nat/201106IS_binary_suse 0K 2.2.CW/nat/201106IS_binary_unix sudo-sh[badftpp@tcredit01]$ du -khsa 2.2.CW/nat/*|grep 201105 5.7G 2.2.CW/nat/201105 746M 2.2.CW/nat/201105IS 670M 2.2.CW/nat/201105IS_binary_suse 670M 2.2.CW/nat/201105IS_binary_unix sudo-sh[badftpp@tcredit01]$ du -khsa /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/* 12G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp1 14G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp2 9.8G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp3 48G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/grp4 76G /appl/fa_data_t/cw_0/input/interestRate/2.2.CW/nat sudo-sh[badftpp@tcredit01]$ ================================================================================================================= 09. 6/10/11 From John Simon... Clarke, Dale C [5:12 PM]: John is there a central log for weekend reboots that are public? Simon, John V [5:13 PM]: Autosys: http://techops/cgi-bin/autosys/reboot_jobs.pl?format=1 Clarke, Dale C [5:13 PM]: thx Simon, John V [5:13 PM]: VCS: http://uxweb/Policies/FailoverSchedules/VCS-Failover-Schedule.htm Clarke, Dale C [5:15 PM]: ================================================================================================================= 6/5/11 From John Simon... How to see if it has direct login - m5prod Unix Praqniques (Practical Techniques)by Dale C Clarke Page 7 PRAQNIQUES psysadm50% ldaplist -l passwd m5prod |grep userPassword userPassword: {crypt}*NOLOGIN* ////////// See if a user id has dierect native login. ldaplist -l passwd <userid> |grep userPassword For example, if direct login is disabled it will look like this: psysadm04% ldaplist -l passwd vnothp |grep userPassword userPassword: {crypt}*NOLOGIN* If direct login is NOT disabled there will be an encrypted string: psysadm04% ldaplist -l passwd g2ujas |grep userPassword userPassword: {crypt}CMvrgwerav49sj ////////// ================================================================================================================= 6/5/11 From John Simon... Quick question... Do u use putty? If so is there a NO TMOUT type command like the sudo Answer: sudo -H -u m5prod sudo-sh export TMOUT=0 command ================================================================================================================= 6/3/11 From John Simon... See if a user id has dierect native login. ldaplist -l passwd <userid> |grep userPassword For example, if direct login is disabled it will look like this: psysadm04% ldaplist -l passwd vnothp |grep userPassword userPassword: {crypt}*NOLOGIN* Unix Praqniques (Practical Techniques)by Dale C Clarke Page 8 PRAQNIQUES If direct login is NOT disabled there will be an encrypted string: psysadm04% ldaplist -l passwd g2ujas |grep userPassword userPassword: {crypt}CMvrgwerav49sj 5/30/11 ================================================================================================================= Dale, These answers are pretty straight forward. 1. Q: What is the AD access that should requested here? ANSWER: since you have to write to the folder, you will need, Read/Write (-RW, .FC, etc) 2.1. Q: From the listing Karuna sent how would I know the access needed; ANSWER: Again pretty easy to figure out: The security group aligns EXACTLY up with the AD PATH - Since the AD PAth is: .... fanniemae.com\corp\DC\Shared\Finance\SFFinanceBAD\HPF This means the associated security group would be: SG-DC-Shared-Finance-SFFinanceBAD* NOTE: the SG in front means "Security Group" While in my prifile I requested to add access to this path and pulled up the following: - SG-DC-SHARED-FINANCE-SFFINANCEBAD-FC 2.2 Q: i.e. what alias? ANSWER: The alias that is required is already in the HPFA Phase 2 runbook: > cd SFFINANCEBAD/HPF This means that you can infer the alias is: SFFINANCEBAD and translates to: \\fanniemae.com\corp\DC\Shared\Finance\SFFinanceBAD 3. Q: Should we all request it? ANSWER: Yes, at least ALL Engineers 5/24/11 ================================================================================================================= From: Simon, John V Sent: Wednesday, May 18, 2011 11:04 AM To: Chen, Jennifer; Hadavi, Behrooz; Elsinger, James A; Milner, Christopher (Contractor) Cc: Mahensaria, Udit (Contractor); Karayi, Sanjith K Subject: RE: HPFA Production Security with SFTP Unix Praqniques (Practical Techniques)by Dale C Clarke Page 9 PRAQNIQUES Here is an example of how to generate, convert and run this. In this case it was UNIX to UNIX. The only difference for AD would be to take the converted public key and have it added to the user you are logging into on the AD side.: $ pwd /tmp/g2ujas $ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/g2ujas/.ssh/id_rsa): /tmp/g2ujas/id_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /tmp/g2ujas/id_rsa. Your public key has been saved in /tmp/g2ujas/id_rsa.pub. The key fingerprint is: 85:ef:52:b1:5e:72:bc:bd:01:93:7f:65:8b:e1:c6:ed g2ujas@pstate-etl04 $ ls -la total 64 drwxr-xr-x 2 g2ujas admstaff 247 May 18 10:52 . drwxrwxrwt 7 root sys 1912 May 18 10:51 .. -rw------- 1 g2ujas admstaff 1675 May 18 10:52 id_rsa -rw-r--r-- 1 g2ujas admstaff 401 May 18 10:52 id_rsa.pub $ ssh-keygen -e -f id_rsa.pub > fsecure.pub $ cat fsecure.pub ---- BEGIN SSH2 PUBLIC KEY ---Comment: "2048-bit RSA, converted from OpenSSH by g2ujas@pstate-etl04" AAAAB3NzaC1yc2EAAAABIwAAAQEA0JntIWU2eC4WhoP6ES/hzpe/Js5lHpilh0JH3EDvt9 YKxP5c3oXalzGZyxgG1o7cQlWfPtKB4hcNOEnwCvrkm0ZuceLbDpvDO2MakCvtVcmIeLa9 xFhZ0A/apTc0wSPiJ5fJUdZj+c6lmI9OQWxcpOUYFDdjtPRXlNnPjt3xVlDyxm0RWz0PQH bx3ff1hhHXrKMhH+YE2O0OdvTzsMZ0FKpExGubtK0Wx3wAkB/S0Ns4KpVxq3wtUvqpg7U0 1OYuyVN25bzkEgpAyRhFNO1woaLymD7HKfVrhuVx/cwt3PbOsnJsbV8yZSlu2ZiEiDztcX UoqoWZow1mfmfVNYOZmw== ---- END SSH2 PUBLIC KEY ---$ cat id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0JntIWU2eC4WhoP6ES/hzpe/Js5lHpilh0JH3EDvt9YKxP5c3oXalzGZyxgG1o7cQlWfPtKB4hcNOEnwCvrkm0ZuceLbDpvDO2MakCvtVcmIeLa9xF hZ0A/apTc0wSPiJ5fJUdZj+c6lmI9OQWxcpOUYFDdjtPRXlNnPjt3xVlDyxm0RWz0PQHbx3ff1hhHXrKMhH+YE2O0OdvTzsMZ0FKpExGubtK0Wx3wAkB/S0Ns4KpVxq3wtUvqpg7U01OY uyVN25bzkEgpAyRhFNO1woaLymD7HKfVrhuVx/cwt3PbOsnJsbV8yZSlu2ZiEiDztcXUoqoWZow1mfmfVNYOZmw== g2ujas@pstate-etl04 $ ls -la Unix Praqniques (Practical Techniques)by Dale C Clarke Page 10 PRAQNIQUES total 96 drwxr-xr-x 2 g2ujas admstaff 449 May 18 10:53 . drwxrwxrwt 7 root sys 1912 May 18 10:53 .. -rw-r--r-- 1 g2ujas admstaff 0 May 18 10:52 commercial.pub -rw-r--r-- 1 g2ujas admstaff 75 May 18 10:56 config -rw-r--r-- 1 g2ujas admstaff 511 May 18 10:52 fsecure.pub -rw------- 1 g2ujas admstaff 1675 May 18 10:52 id_rsa -rw-r--r-- 1 g2ujas admstaff 401 May 18 10:52 id_rsa.pub $ cat config IdentityFile /tmp/g2ujas/id_rsa UserKnownHostsFile /tmp/g2ujas/known_hosts $ sftp -F /tmp/g2ujas/config g2ujas@dsysadm15 Connecting to dsysadm15... Could not create directory '/home/g2ujas/.ssh'. The authenticity of host 'dsysadm15 (10.210.68.15)' can't be established. RSA key fingerprint is f2:ec:6d:6b:13:2a:3f:ec:9a:85:54:34:2a:d2:cb:79. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dsysadm15,10.210.68.15' (RSA) to the list of known hosts. sftp> ls -la drwxrwxr-x 23 g2ujas admstaff 2048 Apr 18 16:54 . dr-xr-xr-x 31 root root 31 May 18 10:55 .. drwxr-xr-x 3 g2ujas admstaff 96 Jan 14 2008 .NetscapeCalendar -rw------- 1 g2ujas admstaff 594 Jun 21 2008 .TTauthority -rw------- 1 g2ujas admstaff 55 Aug 4 2010 .Xauthority -rw-r--r-- 1 g2ujas admstaff 451 Sep 2 2004 .Xdefaults drwx------ 2 g2ujas admstaff 96 Oct 25 2004 .acrobat -rw------- 1 g2ujas admstaff 237 Oct 25 2004 .acrosrch drwx------ 2 g2ujas admstaff 96 Oct 25 2004 .adobe -rw------- 1 root other 10479 Apr 7 17:03 .bash_history -rw-r--r-- 1 g2ujas admstaff 859 Mar 29 2010 .cshrc -rw-r--r-- 1 g2ujas admstaff 1731 Nov 24 2004 .desksetdefaults drwxr-xr-x 14 g2ujas admstaff 1024 Jun 25 2008 .dt 5/12/11 ================================================================================================================= _____________________________________________ From: Perera, Anthony H Unix Praqniques (Practical Techniques)by Dale C Clarke Page 11 PRAQNIQUES Sent: Thursday, May 12, 2011 4:38 PM To: Chelmella, Karuna Nidhi; Clarke, Dale C Cc: Elsinger, James A; BA-D Operations Subject: How to check diskspace on a server <k2uahp@pcredit-sas01lp2> 21: lsvg rootvg DG01_T1 DG01_T2 DG01_T3 oscopyvg1 oscopyvg2 <k2uahp@pcredit-sas01lp2> 22: <k2uahp@ccredit-sas01lp2> 25: lsvg DG01_T2| grep FREE MAX LVs: 4096 FREE PPs: 780 (49920 megabytes) <k2uahp@ccredit-sas01lp2> 26: 21. ASD install issues 4/22/11 ================================================================================================================= From: Beutler, Joshua (Contractor) Sent: Friday, April 22, 2011 11:20 AM To: Clarke, Dale C Subject: ASD install issues Dale, If you have any other install issues relating to drive space, try this fix and see if it helps. Go to Start > Settings > Control Panel > System Management > Advanced Tab. Increase the size of the temporary program download folder. If you have any other questions please feel free to contact the 7s. Thanks! Josh CSC Unix Praqniques (Practical Techniques)by Dale C Clarke Page 12 PRAQNIQUES 4/21/11 ROM Flash for 8440p? ================================================================================================================= HP Notebook System BIOS Update (ROM Family 68CCU) From: Teates, Mike B (Contractor) Sent: Thursday, April 21, 2011 2:16 PM To: Teates, Mike B (Contractor); Clarke, Dale C Subject: Conversation with Teates, Mike B (Contractor), Clarke, Dale C Teates, Mike B (Contractor) [2:09 PM]: http://bizsupport1.austin.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=321957&prodSeriesId=4095872&swItem=ob-854861&prodNameId=4095873&swEnvOID=4060&swLang=13&taskId=135&mode=4&idx=3 20. 3/16/11 Creation of a lockdown doc and ticketxxx ================================================================================================================= From: Elsinger, James A Sent: Wednesday, March 16, 2011 2:02 AM To: Madduri, Anuradha X (Contractor); Jones, Annette X Cc: Rogensues, Gary (Contractor); Perera, Anthony H; Clarke, Dale C Subject: Plan for RPS Production environment setup and LOCKDOWN Importance: High Anu/Annette, Can one of you please submit an OPERATIONAL, LOW Risk Change ticket to support a Production Lockdown of the RPS production environment. Please plan the lockdown for 3/17 0800 -7/231700. Attached is the DRAFT version of the full lockdown instructions The full path of the RPS buildout in is the Draft Runbook. However, below is the full list: Server: pcredit-sas01lp2 /sastemp_rps1_p SAS Temp* 1 /export/appl/rpsprod Code/Scripts /export/appl/rpsprod/bin Code/Scripts /export/appl/rps_data1_p Data Space /export/appl/rps_data2_p Data Space /export/appl/rps_archive1_p Archive Space 250 2 2 2 2 3 rpsprod 10 10 500 250 500 rpsprod rpsprod vbadfp rpsprod rpsprod rpsprod Unix Praqniques (Practical Techniques)by Dale C Clarke 1750 rpsprod rpsprod rpsprod rpsprod rpsprod 755 750 750 750 750 Page 13 PRAQNIQUES Long window Justification: Additional time is needed to support a pre-production shakeout of this new EUC. Low Risk Justification: There is currently no executions occurring in this new RPS production area. The details of the ticket should be the information noted in the DRAFT Production Runbook that related to the EUC setup; mkdirs, transferring ANY staged data, the setting up of Oracle interface as well as review of FS and Directory Ownership and Permissions to align with BADOps standard practices. Gary, Please verify the rpsprod SUDO is working properly on the production server. Elsinger, James A BA&D Operations Manager Analytics, Finance & Controls (202) 520-1889 vvvvvvvv========================================================================================================= Operational Remedy Change Ticket - Detailed Description: =========================================== Purpose: This Operational Change ticket is to prepare the <EUC> environment for controlled process by managing ownership and permissions. This ticket is also to verify and ensure the input data for <EUC> can be successfully read or transferred from interfaces servers in associated SLA’s and/or the server list as noted below for current month data or setup of historical data. LONG WINDOW JUSTIFICATION: This lockdown process is scheduled to run into a weekend and could therefore take 5-6 days. LOW RISK JUSTIFICATION: This change does not require downtime or reboot. No user sessions or applications are impacted. Hosts affected: ================ pcredit-sas01lp2, <Additional Servers see Final Draft Runbook> Please verify you can sudo to both the OPS Transfer ID as well as the EUC Prod Execution ID < SETUP INSTRUCTIONS Unix Praqniques (Practical Techniques)by Dale C Clarke Page 14 PRAQNIQUES =============================================== - As necessary, make sure the Execution Engineer (EE) is logged into pcredit-sas01lp2 and sudo-ed as necessary to RPS Prod Execution ID: rpsprod Alternate: sudo-ed to OPS ID <badftpp> to support file transfers… Verification: NON PROD: > sftp <EE ID>@dcredit-sas01lp1 PROD: > sftp <EE ID>@ pcredit-sas01lp2 1.) Build structure Create the following sub-folder in <EUC> AD structure: Example: \\fanniemae.com\corp\dc\shared\RE07\PROD\GAGOValSingleFam\HPFA\2009Q3_200912\Data Create the FS and Directory Structure (SEE Runbook line ?? Example: As necessary, make sure that the Execution Engineer is logged into pwh-sas17lp2 and sudo-ed to > login /usr/SYSADM/bin/ssh to pwh-sas17lp2<using Execution Engineers own ID> > sudo -H -u badftpp sudo-sh > mkdir /export/appl/<APPLCODE>_data_p/input > mkdir /export/appl/<APPLCODE>_data_p/output > mkdir /export/appl/<APPLCODE>_data_p/input/ > mkdir /export/appl/<APPLCODE>_data_p/output/200912 > quit 2.) Transfer all “One Time” Staged data (SEE Runbook line ?? <If Applicable> 3.) Adjust / Configure permissions and ownership for both Data and Code areas to ensure structure and data files are in lockdown. - Ensure the permissions and ownership are correct as they are noted in the Infrastructure REQUIREMENTS document Unix Praqniques (Practical Techniques)by Dale C Clarke Page 15 PRAQNIQUES LOCKDOWN INSTRUCTIONS =================================== Parameter: <APPLCODE> = “rps” ***3a.) CODE - In General: Code area Settings: ID = vbadfp, Group - <APPLCODE>prod Instructions ==================== Log in to pwh-sas17lp2 using CM ID (vbadfp) and cd /export/appl/<APPLCODE>prod Ensure CM ID owns all code (vbadfp) - Code transfers should be completed with CM ID NOTE: Please change the group ownership of code from faptcm (IF present) to EUC UNIX group to "<APPLCODE>prod" on Production and <APPLCODE>acpt on Acceptance of the code areas to be in line with standard practices. <--CONFIRMED (listing in the Activity tab) - chgroup as necessary to <APPLCODE>prod permissions - 750 (ID = RW, Group should be Read Only, No universal - NOTE: If there was any permission or ownership issues, pass to SCM Team to review the deployment script and make these adjustments as well. Code Area Verification: - Once complete with vbadfp owner ls -laR /export/appl/<APPLCODE>prod/<code area> o Example: /export/appl/rpsprod/bin * The code directory should only be 750 Recursive and NOT contain Universal Read/Write or Execute. - cd to path /export/appl/<APPLCODE>prod - all sub directory permissions should be 750 unless otherwise noted. ***3b.) DATA Unix Praqniques (Practical Techniques)by Dale C Clarke Page 16 PRAQNIQUES o o In General: Input data file Settings: ID = OPS Transfer ID (badftpp), chgroup should be <APPLCODE>prod or <APPLCODE>acpt in acceptance permissions – 640 Output and result data file Settings: ID = OPS EUC Execution ID (rpsprod), chgroup should be <APPLCODE>prod or <APPLCODE>acpt in acceptance permissions – 640 Instructions ==================== Log in to pwh-sas17lp2 using rpsprod and cd /export/appl/<APPLCODE>_data_p or <APPLCODE>_data_a in acceptance Data Area Verification: - Ownership of the data input directory should be the OPS ID (badftpp). If OPS ID is not the owner of the input directory, sudo to the ID that owns the directory and make a ".d" copy of the current one (ex. d20090928) then exit out of sudo and sudo into the OPS ID. Once sudoed with badftpp, mkdir a new inputs directory and chgrp to <APPLCODE>prod NOTE Ensure you copy all previous files in the backups directory back to the new inputs directory. - Once complete ls -ltr (ls -laR?) while in path /export/appl/<APPLCODE>_data_p/input/200909 - Permissions should be 640 for all files, same should be for output as well - cd to path /export/appl/<APPLCODE>_data_p - FS permissions on ALL data directories should be 750 - Ownership of the data input directory should be the OPS ID (badftpp) with group-<APPLCODE>prod. - Owner of output directory should be the Appl ID (<APPLCODE>prod) with group = <APPLCODE>prod 4.) Confirm accessibility of the stages input files in both rps_curr and prev_mth BADOps will confirm that all of the required input files can be transferred: *** Ensure that the files can be read Permissions 644 or group readable and available for transfer during transfers. 4a.) As necessary confirm data can be written to the locked down input directory by verifying the availability and successful transfer of the Input Data. Please consult BADOps Manager for step 3a and 3b Unix to Unix <See pre-execution Runbook steps> Unix Praqniques (Practical Techniques)by Dale C Clarke Page 17 PRAQNIQUES NOTE EE will drop a foo test file as necessary on UNIX file destination path Unix to AD < If applicable> <See pre-execution Runbook steps> NOTE Alias for the AD path will be required and needs to be verified. EE will drop a foo test file as necessary 4b). if necessary: Save a copy of the current input file 3c) Assure access and that the Unix and AD accesses are correct through all lines of the Runbook. Due to the aggressive schedule this may not be possible until the execution. Mitigation should then be recorded in the Errata column of the Runbook. The validating of the transfer lines and will take 4-6 hours or more 4) Create/verify the execution code for all the transfers to be done and add them to the final draft runbook. This is VERY resource intensive. Due to the aggressive schedule this may not be possible until the execution. Code development should then be recorded in the Errata column of the runbook *** Ensure that the files can be read (Permissions 640 or group readable) and available for transfer during executions. 5.) Notification: Once all validation steps are complete, BADOps Rep will notify CreditSupp Manager (Jim Elsinger) and BADOps Mailbox. - OPS to coordinate with RPS as well as input source systems and Downstream systems like FAVM Business NOTE: Keep this change ticket open until consumer has verified the data and structure is correct. Shakeout: ======== ls -la <filename> Unix Praqniques (Practical Techniques)by Dale C Clarke Page 18 PRAQNIQUES Backout: ======= rm <filename> Test / Verification Result: ============ All verification results and adjustments will be noted in the worklog. Also all adjust to Runbook steps will be noted in the Runbook Errata Column. ^^^^^^================================================================================================================= ================================================================================================================= Please note, CORELOGIC CLIENT CONTACT LOGIN NAME: P455W0RD Fannie Mae James Elsinger jelsinger u85EGJ9a Fannie Mae Dale C Clarke dclarke HJKcXH96 Fannie Mae Anthony H Perera aperera 85qq9WAe Fannie Mae Gary Rogensues Unix Praqniques (Practical Techniques)by Dale C Clarke Page 19 PRAQNIQUES grogensues bL5SqtSk From Anthony 6/17/11 ================================================================================================================= Get a LUN or SAN free space <s8ujsm@pcredit-sas01lp3> 1: lsvg rootvg oscopyvg1 DG01_T1 DG01_T0 DG01_T3 DG01_T2 oscopyvg2 <s8ujsm@pcredit-sas01lp3> 2: lsvg DG01_T2 | grep FREE MAX LVs: 4096 FREE PPs: 1010 (64640 megabytes) ^^^^^^================================================================================================================= From Dale Password ldap check password must be on solaris xcredit01 Key is ldapdcc login as: g8udoc g8udoc@dcredit01's password: Last login: Fri Jul 8 12:33:21 2011 from 10.136.197.253 FANNIE MAE MANAGEMENT APPROVED PURPOSES ONLY **** SYSTEM ACTIVITIES MAY BE MONITORED **** <g8udoc@dcredit01> 2: uname -a SunOS dcredit01 5.10 Generic_142900-03 sun4u sparc SUNW,Netra-T12 <g8udoc@dcredit01> 3: bash bash-3.00$ ldaplist -l passwd auqprod |grep userPassword userPassword: {crypt}*NOLOGININITIAL* bash-3.00$ ldaplist -l passwd badftpp |grep userPassword userPassword: {crypt}YM2U7.ddJ8R5g bash-3.00$ Unix Praqniques (Practical Techniques)by Dale C Clarke Page 20 PRAQNIQUES ================================================================================================================= 17. MD5 perl for files hash sudo-sh[badftpp@pwarehouse-rsas17lp1]$ csum -h MD5 ./md5sum.tst 00ba45f9cbd9dba978eea6ce97dfd82a ./md5sum.tst sudo-sh[badftpp@pwarehouse-rsas17lp1]$ ./md5sum.pl ./md5sum.tst 00ba45f9cbd9dba978eea6ce97dfd82a ./md5sum.tst sudo-sh[badftpp@pwarehouse-rsas17lp1]$ cat ./md5sum.tst somethingsomething sudo-sh[badftpp@pwarehouse-rsas17lp1]$ cat ./md5sum.pl #! /usr/bin/perl use Digest::MD5; use IO::File; my $chk = Digest::MD5->new(); foreach my $file (@ARGV) { $chk->addfile(IO::File->new($file)); print "",$chk->hexdigest," $file \n"; } sudo-sh[badftpp@pwarehouse-rsas17lp1]$ ================================================================================================================= 18. Here are the command lines to test SAS 1. Running with auqprod UID > login /usr/bin/ssh to cwh-sas17lp2 > sudo -H -u auqprod sudo-sh > cd /export/appl/auqprod > /export/apps/sas_9.1/sas -noterminal -is 'proc setinit; run ;' /dev/null -log /tmp/test.log > cat /tmp/test.log > rm -f /tmp/test.log Unix Praqniques (Practical Techniques)by Dale C Clarke Page 21 PRAQNIQUES 2. Running with any other UID > login /usr/bin/ssh to cwh-sas17lp2 > /export/apps/sas_9.1/sas -noterminal -work /tmp -sasuser /tmp -is 'proc setinit; run ;' /dev/null -log /tmp/test.log > cat /tmp/test.log > rm -f /tmp/test.log a) If you see following error, SAS does not have execute permission The file access permissions do not allow the specified action.. b) Change SAS command as below if we agree to use contingency SAS installation as it is (not changing to match production) /export/apps/sas_9.1.3_sp4/sas ================================================================================================================= 19. Eval this from John on 8/26 ================================================================================================================= sudo-sh[root@pcredit-sas01lp2]# ps -ef |egrep '450720|598016|610452|725028|761996|958636|1126516|1159178|1581270' root 450720 508016 0 Aug 14 - 8:56 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_481272/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 483346 610452 0 Aug 19 - 1:05 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 528608 1581270 0 Aug 21 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 573674 450720 0 Aug 14 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 598016 1163346 0 16:46:46 pts/4 0:00 sudo-sh root 610452 1941684 0 Aug 19 - 8:50 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_622246/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 655576 761996 0 Aug 15 - 1:05 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 680144 958636 0 06:00:28 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" Unix Praqniques (Practical Techniques)by Dale C Clarke Page 22 PRAQNIQUES root 725028 442586 0 Aug 23 - 9:05 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1150717/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 761996 393404 0 Aug 15 - 8:39 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_769173/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 958636 1691750 0 06:00:28 - 8:47 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1966601/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 971004 1126516 0 Aug 17 - 1:06 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" root 1126516 892972 0 Aug 17 - 8:55 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1128165/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 1143014 725028 0 Aug 23 - 1:09 /usr/fnma/bin/ssh ccredit-sas01lp2 "/export/appl/uxprod/sodc/sbin/rsync" --server -vlHogDtpAre.i --delete . "/export/appl/badops" badftpp 1159178 1872024 0 15:04:10 pts/3 0:00 sudo-sh root 1581270 995518 0 Aug 21 - 8:46 /export/appl/uxprod/sodc/sbin/rsync --rsync-path "/export/appl/uxprod/sodc/sbin/rsync" -vaHA --blocking-io --delete -rsh=/usr/fnma/bin/ssh --exclude-from /tmp/.queue/RSYNC_1596486/files/00008.EXCLUDE_FILE /export/appl/badops/ ccredit-sas01lp2:"/export/appl/badops" root 1736720 598016 0 16:54:14 pts/4 0:00 egrep 450720|598016|610452|725028|761996|958636|1126516|1159178|1581270 root 1822966 598016 7 16:54:14 pts/4 0:00 ps -ef sudo-sh[root@pcredit-sas01lp2]# kill 450720 483346 528608 573674 610452 655576 680144 725028 761996 958636 971004 1126516 1143014 1581270 sudo-sh[root@pcredit-sas01lp2]# ps -ef |egrep '450720|598016|610452|725028|761996|958636|1126516|1159178|1581270' root 442590 598016 1 16:55:20 pts/4 0:00 egrep 450720|598016|610452|725028|761996|958636|1126516|1159178|1581270 root 598016 1163346 1 16:46:46 pts/4 0:00 sudo-sh root 725046 598016 7 16:55:20 pts/4 0:00 ps -ef badftpp 1159178 1872024 0 15:04:10 pts/3 0:00 sudo-sh sudo-sh[root@pcredit-sas01lp2]# df -k |grep badop /dev/badops 31457280 31450428 1% 100 1% /export/appl/badops sudo-sh[root@pcredit-sas01lp2]# ================================================================================================================================================ ================================================================================== 24. sudo-sh[lwftpp@pwh-sas17lp2]$ sudo-sh[lwftpp@pwh-sas17lp2]$ date; uname -a ; df -gs /export/appl/lw_data_p3 Mon Oct 3 16:33:09 EDT 2011 AIX pwh-sas17lp2 3 5 00C371A04C00 Filesystem GB blocks Free* %Used Iused %Iused Mounted on /dev/lw_data_p3 500.00 148.06 71% 3641 1% /export/appl/lw_data_p3 sudo-sh[lwftpp@pwh-sas17lp2]$ ================================================================================================================= Unix Praqniques (Practical Techniques)by Dale C Clarke Page 23 PRAQNIQUES 25. When you get a dialog box when connecting to Outlook the has a User prompt looking like EXSW01NS/<your userID> requesting a password (which it then rejects) try the fillowing. Change the EXSW01NS/<your userID> to fanniemae/<your userID>, enter your correct password. It should then connect. This works when opening Communicator as well. This happens most often when your connection has been lost or when changing from wired to wireless but may have other causes. ================================================================================================================= TOOLS: sudo-sh[badftpp@pwarehouse-rsas17lp1]$ pwd; ls -ltr *.ksh /export/appl/badops/tools/devl -rwx------ 1 badftpp badftpp 2706 Oct 12 18:17 ckperlver.ksh -rwx------ 1 badftpp badftpp 2195 Oct 12 18:20 cktoolsize.ksh sudo-sh[badftpp@pwarehouse-rsas17lp1]$ ================================================================================================================= 34. grep examples: SPECIFIC ONE LINERS: ######## Find if there are ANY files in a directory, recursively that contain data: Where file/dir, - not owned or grouped by lwftpp or lwprod - not an laR total line, colon title, or blank ls -laR /export/appl/badops/export|grep -v '[lwftpp|lwprod]'|grep -v total|grep -v ":"|grep -v '^$'|wc % Tells if there are ANY non-zero size directories: du /export/appl/badops/export|grep -v '^0'|wc ######## An easy, one liner command to replace a test string in multiple files in a sub-directory using recursion. % grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<chgword>/g' ***CAUTION BACK UP YOUR DIRECTORY BEFORE MAKING THE ERROR YOU CERTAINLY WILL ((-8 OR Use the perl oneliner that provides the option of creating a backup of each file changed replacing the sed: % perl -p -i.bak -e ’s/<oldword>/<chgword>/g’ * Note: In both cases some characters require negation with the backslash (experiment) GENERAL: % grep -v '^$' - grep OUT blank lines % Search vivek in any case (i.e. case insensitive search): grep -i -w vivek /etc/passwd % Search vivek or raj in any case: grep -E -i -w 'vivek|raj' /etc/passwd % Match Vivek or vivek:grep '[vV]ivek' filename Unix Praqniques (Practical Techniques)by Dale C Clarke Page 24 PRAQNIQUES OR grep '[vV][iI][Vv][Ee][kK]' filename for any upper/lower in all positions % You can also match digits (i.e match vivek1 or Vivek2 etc): grep -w '[vV]ivek[0-9]' filename OR You can match two numeric digits (i.e. match foo11, foo12 etc): grep 'foo[0-9][0-9]' filename % You are not limited to digits, you can match at least one letter: grep '[A-Za-z]' filename % Display all the lines containing either a "w" or "n" character: grep [wn] filename % Display any lines starting with a dot and digit: grep '^\.[0-9]' filename % Match a character "v" two times in a line: egrep "v{2}" filename % Match both "col" and "cool": egrep 'co{1,2}l' filename % Match any row of at least three letters 'c': egrep 'c{3,}' filename % Search directories recursively: grep -r 'hello' /home/gigi % The -l option print will only the filename (e.g., /etc/crontab, /etc/group) if the file contains a hit: grep -l root /etc/* Can be used to create an array for a command to process. CLASS brackets: Within a bracket expression, the name of a character class enclosed in "[:" and ":]" stands for the list of all characters belonging to that class. Standard character class names are: [:alnum:] - Alphanumeric characters. [:alpha:] - Alphabetic characters [:blank:] - Blank characters: space and tab. [:digit:] - Digits: '0 1 2 3 4 5 6 7 8 9'. [:lower:] - Lower-case letters: 'a b c d e f g h i j k l m n o p q r s t u v w x y z'. [:space:] - Space characters: tab, newline, vertical tab, form feed, carriage return, and space. [:upper:] - Upper-case letters: 'A B C D E F G H I J K L M N O P Q R S T U V W X Y Z'. In this example match all upper case letters: grep '[:upper:]' filename WILDCARDS (Dot): You can use the "." for a single character match. In this example match all 3 character word starting with "b" and ending in "t": % grep '\<b.t\>' filename Where, The Dot matches ANY character \< Match the empty string at the beginning of word \> Match the empty string at the end of word. % Print all lines with exactly two characters: grep '^..$' filename ESCAPING (backslash before a character): % In an IP search all three dots need to be escaped: grep '192\.168\.1\.254' /etc/hosts % The following example will only match an IP address: egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' filename Unix Praqniques (Practical Techniques)by Dale C Clarke Page 25 PRAQNIQUES % The following will match word Linux or UNIX in any case: egrep -i '^(linux|unix)' filename % Search for all lines matching '--test--' using -e option. Without -e, grep would attempt to parse '--test--' as a list of options: grep -e '--test--' filename Anchors: % grep -v '^$' - grep OUT blank lines % Match line only containing foo: grep '^foo$' filename % Find lines ending with word foo: grep 'foo$' filename % Display only lines starting with the word vivek only i.e. do not display vivekgite, vivekg etc: grep -w ^vivek /etc/passwd % % ls -laR /export/appl/lw_archive_p2|grep -v lwftpp|grep -v lwprod|head grep -v total|grep -v ":"|wc ================================================================================================================= 35. How to get a server or dns name for and IP address - Dougherty, Kate [11:46 PM]: just run nslookup <IP Address> Example: bash-2.03$ nslookup 10.204.64.21 Server: psysadm-dn07.fanniemae.com Address: 158.137.218.75 Name: pwarehouse-rsas17lp1.fanniemae.com Address: 10.204.64.21 bash-2.03$ ================================================================================================================= 35a. $ cp filename{,.bak} --> In this one-liner filename{,.bak} gets copied filename.bak This one-liner copies the file named filename to a file named filename.bak. Here is how it works. It uses brace expansion to construct a list of arguments for the cp command. Brace expansion is a mechanism by which arbitrary strings may be generated. In this one-liner filename{,.bak} gets brace expanded to filename filename.bak and puts in place of the brace expression. The command becomes cp filename filename.bak and file gets copied. ================================================================================================================= 36. Print out unique lines in a file - Print out unique lines in a file cat LOGS/CMDSRUN.txt| sort -n | uniq |head Unix Praqniques (Practical Techniques)by Dale C Clarke Page 26 PRAQNIQUES - Print out unique lines in a file for csv selecting 2nd field cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -nr | uniq |head cat LOGS/CMDSRUN.txt| cut -d',' -f 2 | sort -n | uniq |head Simplified: cut -d',' -f 2 <file> | sort -nu MORE: bash$ cat list-1 list-2 list-3 | sort | uniq > final.list # Concatenates the list files, # sorts them, # removes duplicate lines, # and finally writes the result to an output file. bash$ cat testfile This line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times. bash$ uniq -c testfile 1 This line occurs only once. 2 This line occurs twice. 3 This line occurs three times. bash$ sort testfile | uniq -c | sort -nr 3 This line occurs three times. 2 This line occurs twice. 1 This line occurs only once. SCRIPT: #!/bin/bash # wf.sh: Crude word frequency analysis on a text file. # This is a more efficient version of the "wf2.sh" script. Unix Praqniques (Practical Techniques)by Dale C Clarke Page 27 PRAQNIQUES # Check for input file on command-line. ARGS=1 E_BADARGS=85 E_NOFILE=86 if [ $# -ne "$ARGS" ] # Correct number of arguments passed to script? then echo "Usage: `basename $0` filename" exit $E_BADARGS fi if [ ! -f "$1" ] # Check if file exists. then echo "File \"$1\" does not exist." exit $E_NOFILE fi ######################################################## # main () sed -e 's/\.//g' -e 's/\,//g' -e 's/ /\ /g' "$1" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr # ========================= # Frequency of occurrence # Filter out periods and commas, and #+ change space between words to linefeed, #+ then shift characters to lowercase, and #+ finally prefix occurrence count and sort numerically. # Arun Giridhar suggests modifying the above to: # . . . | sort | uniq -c | sort +1 [-f] | sort +0 -nr # This adds a secondary sort key, so instances of #+ equal occurrence are sorted alphabetically. # As he explains it: Unix Praqniques (Practical Techniques)by Dale C Clarke Page 28 PRAQNIQUES # "This is effectively a radix sort, first on the #+ least significant column #+ (word or string, optionally case-insensitive) #+ and last on the most significant column (frequency)." # # As Frank Wang explains, the above is equivalent to #+ . . . | sort | uniq -c | sort +0 -nr #+ and the following also works: #+ . . . | sort | uniq -c | sort -k1nr -k ######################################################## exit 0 bash$ cat testfile This line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times. bash$ ./wf.sh testfile 6 this 6 occurs 6 line 3 times 3 three 2 twice 1 only 1 once 1. cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename 2. It is even possible to specify a linefeed as a delimiter if apropos. The trick is to actually embed a linefeed (RETURN) in the command sequence. Unix Praqniques (Practical Techniques)by Dale C Clarke Page 29 PRAQNIQUES bash$ cut -d' ' -f3,7,19 testfile This is line 3 of testfile. This is line 7 of testfile. This is line 19 of testfile. 37. ================================================================================================================= bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc Could not create directory '/home/g8udoc/.ssh'. dn: uid=g8udoc,ou=people,ou=corporate,dc=fanniemae,dc=com shadowLastChange: 15555 fmpasswordresetdate: 20120803005933Z userPassword: {crypt}d5Q9tVme3qYb2 fmDepartment: Enterprise Technology Svcs fmdivision: Technology fmbusinessunit: Ent Data Technology fmcostcenter: 485 fmProfileOwner: 1 roomNumber: 2450B physicalDeliveryOfficeName: 2h-2n/14 fmManagerLevel: Associates fmservicedate: 02/28/2011 12:00:00 AM EST fmjoblevel: 5 title: Applctn Support Analyst III fmlegalname: Dale C Clarke fmemployeestatus: A fmPrivilegedID: No fmSharedID: No employeeNumber: 07887 employeeType: Employee fmofficelocation: 4000 Wisconsin Avenue fmfloor: Second telephoneNumber: 202-752-3465 shadowMin: -1 shadowFlag: 0 Unix Praqniques (Practical Techniques)by Dale C Clarke Page 30 PRAQNIQUES shadowExpire: -1 shadowInactive: -1 shadowMax: -1 shadowWarning: 7 homeDirectory: /home/g8udoc uidNumber: 46496 gidNumber: 5069 gecos: Dale C Clarke loginShell: /bin/csh mailAlternateAddress: g8udoc@fanniemae.com mailRoutingAddress: dale_c_clarke@exchange.fanniemae.com mail: dale_c_clarke@fanniemae.com mailHost: exchange.fanniemae.com cn: Dale C Clarke fmmarketroom: No fmpeoplemanager: 0 fmContractorType: Regular fmbusinesscritical: No fmkiosk: No fmmanageruid: g8ujxe fmstatus: Active manager: James Elsinger initials: C fmisterminated: No fmregion: dc uid: g8udoc givenName: Dale objectClass: inetorgperson objectClass: organizationalPerson objectClass: person objectClass: top objectClass: fannieMaePerson objectClass: mailRecipient objectClass: posixAccount objectClass: shadowAccount objectClass: account sn: Clarke Unix Praqniques (Practical Techniques)by Dale C Clarke Page 31 PRAQNIQUES bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep UID Could not create directory '/home/g8udoc/.ssh'. bash-2.05b$ ssh dcredit01 /usr/bin/ldaplist -l passwd g8udoc|grep uid Could not create directory '/home/g8udoc/.ssh'. dn: uid=g8udoc,ou=people,ou=corporate,dc=fanniemae,dc=com uidNumber: 46496 fmmanageruid: g8ujxe uid: g8udoc bash-2.05b$ 38. ================================================================================================================= Find in a directory all occurrences of "paul_obrecht“ and print; then replace all occurrences, then print again to show there are none found. //////////////////////////////// example of the first file in the print below before it is edited sudo-sh[badftpp@pwh-sas17lp2]$ cat ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES #|gnum|G9999| #|dsesrc|RTI| #|dsesrcsrvr|pwh-sas17lp2| #|dsedest|"M&A RESEARCH"| #|dsedestsrvr|pcredit-sas01lp3| #|dsedistro|"paul_obrecht daniel_dausman dean_m_henderson af-c_operations ba-d_operations"| #|dsesubject|"TRANSFERS COMPLETED SUCCESSFULLY"| #|dseerrorsubject|"TRANSFERS FAILED"| #|dsebody|"PLEASE SEE THE ATTACHED FILE FOR COMPLETE TRANSFER DETAILS."| #|dseendnote|"Please contact af-c_operations@fanniemae.com, if you have any questions or concerns regarding this transfer."| #|dseattachmentT|TRANSFER.log| #|dseattachmentE|ERROR.log| sudo-sh[badftpp@pwh-sas17lp2]$ ////////////////////////////////////////// sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print ß find - found 5 occurrences #|dsedistro|"paul_obrecht daniel_dausman dean_m_henderson af-c_operations ba-d_operations"| ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES #|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"| ./VAL/RTI.d20120913DCCbackup/rtit2300.1_g6941_rti-ldm.listN #|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"| Unix Praqniques (Practical Techniques)by Dale C Clarke Page 32 PRAQNIQUES ./VAL/RTI.d20120913DCCbackup/rtit2300.2_g6941_rti-ldm.listN #|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"| ./VAL/RTI.d20120913DCCbackup/rtit2300.3_g6941_rti-ldm.listN #|dsedistro|"stephen_c_schwartz lina_m_gomez udit_mahensaria dmitry_zontov paul_obrecht af-c_operations ba-d_operations"| ./VAL/RTI.d20120913DCCbackup/rtit2300.4_g6941_rti-ldm.listN sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ | xargs perl -pi -e 's/paul_obrecht/ /g' ßfind and change in place perl command Can't do inplace edit: ./VAL/RTI.d20120913DCCbackup/ is not a regular file. sudo-sh[badftpp@pwh-sas17lp2]$ find ./VAL/RTI.d20120913DCCbackup/ -type f -exec grep "obrecht" {} \; -print ß find – found none sudo-sh[badftpp@pwh-sas17lp2]$ ß None found //////////////////////////////// example of the first file after edited sudo-sh[badftpp@pwh-sas17lp2]$ cat ./VAL/RTI.d20120913DCCbackup/rtit1300_g9999_rti-marsrch.listN-MULTIPLE-FILES #|gnum|G9999| #|dsesrc|RTI| #|dsesrcsrvr|pwh-sas17lp2| #|dsedest|"M&A RESEARCH"| #|dsedestsrvr|pcredit-sas01lp3| #|dsedistro|" daniel_dausman dean_m_henderson af-c_operations ba-d_operations"| ß Notice the extra space – could be a replacement name #|dsesubject|"TRANSFERS COMPLETED SUCCESSFULLY"| #|dseerrorsubject|"TRANSFERS FAILED"| #|dsebody|"PLEASE SEE THE ATTACHED FILE FOR COMPLETE TRANSFER DETAILS."| #|dseendnote|"Please contact af-c_operations@fanniemae.com, if you have any questions or concerns regarding this transfer."| #|dseattachmentT|TRANSFER.log| #|dseattachmentE|ERROR.log| sudo-sh[badftpp@pwh-sas17lp2]$ 39. ================================================================================================================= GET A NAME FROM A USERID ON UNIX ===================================== Solaris: bash-3.00$ ldaplist -l passwd <userID> |grep cn: cn: Dale C Clarke bash-3.00$ AIX: bash-2.05b$ getable -a passwd <userID> g8udoc:*NP*:46496:5069:Dale C Clarke:/home/g8udoc:/bin/csh Unix Praqniques (Practical Techniques)by Dale C Clarke Page 33 PRAQNIQUES Name...........: Dale C Clarke Title..........: Employee Type..: Employee Telephone No...: 202-752-3465 Manager........: James Elsinger bash-2.05b$ Linux: Not yet found 42. ================================================================================================================= 50. TRANSFER A FILE WITH SCP THAT HAS A SAS GDG //////////////////////// sudo-sh[ainprod@pwh-sas17lp2]$ cd /export/appl/ain_archive_p/LASD/ sudo-sh[ainprod@pwh-sas17lp2]$ ls -il total 34331200 18 -rwxr-xr-x 1 ainprod ainprod 1539776512 Jul 02 2010 dstrs_clm_draft.sas7bdat_bad100702 10 -rw-r--r-- 1 ainprod ainprod 2587697152 Aug 03 11:05 lasdbkp#048.sas7bdat 11 -rw-r--r-- 1 ainprod ainprod 2622562304 Sep 03 09:46 lasdbkp#049.sas7bdat 12 -rw-r--r-- 1 ainprod ainprod 2658934784 Oct 03 09:47 lasdbkp#050.sas7bdat 13 -rw-r--r-- 1 ainprod ainprod 2690850816 Nov 03 09:58 lasdbkp#051.sas7bdat 15 -rw-r--r-- 1 ainprod ainprod 2723553280 Dec 03 09:56 lasdbkp#052.sas7bdat 17 -rw-r--r-- 1 ainprod ainprod 2754224128 Jan 03 08:35 lasdbkp.sas7bdat sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec cksum {} \; 1009243769 2690850816./lasdbkp#051.sas7bdat sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \; badftpp@dcredit-sas02lp1's password: sudo-sh[ainprod@pwh-sas17lp2]$ find . -inum 13 -exec scp {} badftpp@dcredit-sas02lp1:/export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ \; badftpp@dcredit-sas02lp1's password: Could not chdir to home directory /appl/badftpp: A file or directory in the path name does not exist. lasdbkp#051.sas7bdat 100% 2597MB 40.6MB/s 01:04 sudo-sh[ainprod@pwh-sas17lp2]$ RENAME ON THE TARGET SERVER: bash-3.2$ cd /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD/ Unix Praqniques (Practical Techniques)by Dale C Clarke Page 34 PRAQNIQUES bash-3.2$ ls -il total 5255568 77921 -rw-r--r-- 1 badftpp badftpp 2690850816 Jan 11 13:24 lasdbkp#051.sas7bdat bash-3.2$ uname -a; pwd;find . -inum 77921 -exec cksum {} \; AIX dcredit-sas02lp1 1 6 00F62A394C00 /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD 1009243769 2690850816 ./lasdbkp#051.sas7bdat bash-3.2$ find . -inum 77921 -exec mv {} dstrs_clm_draft.sas7bdat bash-3.2$ ls -ltr total 5255568 -rw-r--r-- 1 badftpp badftpp 2690850816 Jan 11 13:24 dstrs_clm_draft.sas7bdat bash-3.2$ uname -a; pwd;cksum dstrs_clm_draft.sas7bdat \; AIX dcredit-sas02lp1 1 6 00F62A394C00 /export/appl/a2z_data4_d/lwprod/loan_performance/201210/LASD 1009243769 2690850816 dstrs_clm_draft.sas7bdat bash-3.2$ 43. ================================================================================================================= HOW-2 find man pages sudo-sh[ainprod@pwh-sas17lp2]$ apropos acl aclconvert(1) - Converts the access control information of a file system object from onetype to another. acledit(1) - Edits the access control information of a file. aclget(1) - Displays the access control information of a file. aclgettypes(1) - Gets ACL types supported by a file system path. aclput(1) - Sets the access control information of a file. ctaclfck(1) - Verifies the contents of a clustersecurity services ACL file. sudo-sh[ainprod@pwh-sas17lp2]$ apropos aclput aclput(1) - Sets the access control information of a file. sudo-sh[ainprod@pwh-sas17lp2]$ man aclput ================================================================================================================= ================================================================================================================= Unix Praqniques (Practical Techniques)by Dale C Clarke Page 35