MultiStore Hands-On Labs Lab 5: vFiler DR Scott Gelb, Insight Integrated Systems Roger Weeks, NetApp February 2011 SUMMARY Disaster recovery is an important aspect of MultiStore. In this lab you will see how the vfiler dr command works and set up standby DR vfiler units using both async and sync SnapMirror. STEP 1 SETUP DR VFILER FAS6280B_VFILER1 FROM FAS6280B TO FAS6280A AND FAS3270 vFiler DR fas6280b_vfiler1 from simulator node fas6280b to both fas6280a and fas3270 FAS6280A fas6280a_vfiler1 FAS3270 fas6280b_vfiler1 FAS6280B SnapMirror ASYNC to Primary D.R. fas6280b_vfiler1 fas6280a_vfiler1 SnapMirror SYNC to Secondary D.R. 2 MultiStore Hands-On Lab 5: vFiler DR fas6280b_vfiler1 D.R. STEP 1A - FAS6280B_VFILER1 ON FAS6280B TO FAS6280A DR VFILER Source/target volumes and IPSpaces must exist with the same name – pre-set already in the sims) Do not protect volumes used by vFiler DR with Protection Manager Verify that SnapMirror is running on the source controller o fas6280a> snapmirror on fas6280a> vfiler status -a # no vfiler running since we migrated in lab5 fas6280a> vfiler dr configure -c secure -l root:netapp01 fas6280b_vfiler1@192.168.150.101 o “y” to keep the same IP address and subnet (you can change for DR) o use the same 255.255.255.0 subnet o “n” to use different dns (keep the same DNS since you can change for DR) o use “ns1” interface on the dr vfiler fas6280a> vfiler dr status fas6280b_vfiler1@192.168.150.101 fas6280a> snapmirror status fas6280a> vfiler status -a # you will see the stopped dr vfiler STEP 1B - FAS6280B_VFILER1 ON FAS6280B TO FAS3270 DR VFILER 3 Use “-s” option for snapmirror sync (ignore “too many active transfers at once errors” from the simulator) fas3270> vfiler status -a fas3270> vfiler dr configure -s -c secure -l root:netapp01 fas6280b_vfiler1@192.168.150.101 o enter nothing for “alternate” IP twice when it asks each time o “y” to keep the same IP address and subnet (you can change for DR) o use the same 255.255.255.0 subnet o “n” to use different dns (keep the same DNS since you can change for DR) o use “ns1” interface on the dr vfiler fas3270> vfiler dr status fas6280b_vfiler1@192.168.150.101 fas3270> snapmirror status fas3270> vfiler status -a Delete the DR vfiler (this was an example to create two, but for the lab we will failover locally to the FAS6280A node.)… the volumes will still be available to vfiler0. o fas3270> vfiler dr delete fas6280b_vfiler1@192.168.150.101 # this will take some time o fas3270> vfiler status -a MultiStore Hands-On Lab 5: vFiler DR STEP 2 – FAILOVER VFILER DR Failover vFiler by activating the DR vfiler on FAS6280A after a disaster on the source FAS6280B FAS6280B_vfiler1 will be shut down to simulate a disaster and FAS6280A_vfiler1 DR will be brought online (STOP on node B and dr on node A) *** NOTE THE FILER NAMES TO ISSUE COMMANDS *** The elegance of vfiler DR is that is breaks ALL mirrors automatically and it also has all the shares, exports and iSCSI mappings from the source which automates the dr environment. Without vFiler DR, you would have to manually break mirrors, manually create shares, exports, igroups, mappings. vFiler DR also migrates the vfiler etc configuration which is how it automates matching the source settings. fas6280b> vfiler stop fas6280b_vfiler1 # simulate a disaster fas6280b> ifconfig ns1 0.0.0.0 # Bug in 7.3.3/7.3.4/7.3.5 - The sim on nodeb leaves the IP on ns1 even the vfiler is stopped, so remove the .104 to eliminate the redundant IP on the network. Stopping the source vfiler should disable the IP but it does not in these releases – pending BURT 4 fas6280a> vfiler dr activate fas6280b_vfiler1@192.168.150.101 fas6280b> vfiler status -a fas6280a> vfiler status -a MultiStore Hands-On Lab 5: vFiler DR # confirm fas6280b_vfiler1 is stopped # confirm fas6280b_vfiler1 is running STEP 3 – RESYNC VFILER DR BACK TO PRODUCTION AND RESYNC DR FAS6280A fas6280a_vfiler1 fa6280b_vfiler1 fas6280b_vfiler1 D.R. FAS6280B SnapMirror SnapMirror Resync D.R. Resync D.R. fas6280b_vfiler1 FAS3270 fas6280a_vfiler1 STEP 3B - RESYNC THE DR SITE FROM THE 6280A DR NODE TO THE 6280B PROD NODE. o fas6280b> vfiler dr resync -c secure -l root:netapp01 o o o fas6280b_vfiler1@192.168.150.100 fas6280b> vfiler dr status fas6280b_vfiler1@192.168.150.100 fas6280b> snapmirror status fas6280b> vfiler status -a STEP 3C - STOP THE DR VFILER ON FAS6280A AND ACTIVATE THE PROD VFILER ON FAS6280B *** NOTE THE FILER NAMES TO ISSUE COMMANDS *** this is the opposite of what we did in step 2 to activate the dr side. o o o fas6280a> vfiler stop fas6280b_vfiler1 o fas6280b> ifconfig ns1 192.168.150.104 netmask 255.255.255.0 up # have to o o 5 fas6280a> ifconfig ns1 0.0.0.0 # similar to before (pending BURT) fas6280b> vfiler dr activate fas6280b_vfiler1@192.168.150.100 ifconfig since we turned it off before… we could use “setup -e” from the vfiler but that would whack setup files fas6280b> vfiler status -a # confirm fas6280b_vfiler1 is running fas6280a> vfiler status -a # confirm fas6280b_vfiler1 is stopped MultiStore Hands-On Lab 5: vFiler DR STEP 3D - RESYNC THE PROD VFILER ON FAS6280B TO THE DR VFILER ON FAS6280A o fas6280a> vfiler dr resync -c secure -l root:netapp01 o o o fas6280b_vfiler1@192.168.150.101 fas6280a> vfiler dr status fas6280b_vfiler1@192.168.150.101 fas6280a> snapmirror status fas6280a> vfiler status -a STEP 3E - RELEASE SNAPMIRROR RELATIONSHIPS LEFT OVER FROM THE RESYNC OF THE ORIGINAL SOURCE o fas6280a> snapmirror release fas6280b_vfiler1_root o o 6 fas6280b:fas6280b_vfiler1_root fas6280a> snapmirror release fas6280b_vfiler1_san fas6280b:fas6280b_vfiler1_san fas6280a> snapmirror release fas6280b_vfiler1_nas fas6280b:fas6280b_vfiler1_nas MultiStore Hands-On Lab 5: vFiler DR STEP 4 – ADD A VOLUME AND IP TO THE SOURCE VFILER AND ADD THOSE RESOURCES TO THE DR VFILER NOTE: FAS6280B is the Source and FAS6280A is the Destination (DR) NOTE: “vfiler dr resync” updates snapmirror.conf to 0-59/3 (every 3 minutes) so it is recommended to save the snapmirror.conf file prior to a dr resync so that you can reset to prior settings after the resync. Adding VOLUMES STEP 4A SOURCE VFILER – CREATE A NEW VOLUME AND ADD IT TO FAS6280B_VFILER1 ON FAS6280B o fas6280b> vol create new_vol -s none aggr1 20m o o fas6280b> vfiler add fas6280b_vfiler1 /vol/new_vol fas6280b> vfiler status -a fas6280b_vfiler1 (or -r) # confirm volume added to vfiler STEP 4B DR VFILER – ADD THE NEW SOURCE VOLUME TO THE TARGET VFILER BY CREATING THE VOLUME THEN USE DR RESYNC ON FAS6280A o fas6280a> vol create new_vol -s none aggr1 20m o fas6280a> vfiler dr resync -c secure -l root:netapp01 o o o fas6280b_vfiler1@192.168.150.101 # this will mirror the new volume (and also update snapmirror.conf so edit for prior schedule) fas6280a> vfiler dr status fas6280b_vfiler1@192.168.150.101 fas6280a> snapmirror status fas6280a> vfiler status -a # confirm new_vol is now owned by the stopped dr vfiler Adding IPs (same as changing IPs – KB1011371) STEP 4C SOURCE VFILER – CREATE A NEW IP AND ADD IT TO FAS6280B_VFILER1 ON FAS6280B o fas6280b> vfiler add fas6280b_vfiler1 -i 192.168.150.110 7 o fas6280b> ifconfig ns1 alias 192.168.150.110 # this will link the vfiler IP to the o ns1 interface. This can be done with “setup -e” but that would whack some vfiler etc filers. fas6280b> vfiler status -a fas6280b_vfiler1 # confirm .110 is configured MultiStore Hands-On Lab 5: vFiler DR STEP 4D DR VFILER – CREATE A NEW IP AND ADD IT TO FAS6280B_VFILER1 ON FAS6280A NOTE: this is not done with DR resync but is done on the DR vfiler itself, so adding or changing an IP is different than adding or changing a volume. Changing/adding/removing the DR vfiler IP is done on the DR vfiler itself. The initial IPs are given at the vfiler dr configure command and are not modified with a wizard, but by activating the DR vfiler. Be careful of DUPLICATE IPs if it matches the source. The ONLY way to modify this is to activate the dr vfiler. o o o o o o o o o fas6280a> vfiler dr activate fas6280b_vfiler1@192.168.150.101 fas6280a> vfiler add fas6280b_vfiler1 -i 192.168.150.110 # duplicate IP will occur – disconnect or use no-vfiler-ip boot option if production if using the same ip fas6280a> ifconfig ns1 192.168.150.110 netmask 255.255.255.0 # we could run “vfiler run fas6280b_vfiler1 setup” and whack setup files too fas6280b> vfiler stop fas6280b_vfiler1 # stop vfiler so we can resync fas6280a> vfiler status -a # confirm fas6280b_vfiler1 is stopped fas6280a> vfiler dr resync -c secure -l root:netapp01 fas6280b_vfiler1@192.168.150.101 fas6280a> vfiler dr status fas6280b_vfiler1@192.168.150.101 fas6280a> snapmirror status fas6280a> vfiler status -a # confirm .104 and .110 are attached to the vFiler *** SIMILARLY you can use the method above to change IPs and remove IPs on the DR vFiler since IP settings are not used from the source vFiler. *** 8 MultiStore Hands-On Lab 5: vFiler DR STEP 5 – VFILER DR WHEN MIRRORS ALREADY EXIST ON THE TARGET This step must be completed to continue on with labs 6-9 ONTAP 7.3.5 *** With ONTAP 7.3.5, we have a new “-u” option in vfiler dr configure that will not reinitialize mirrors already initialized. *** Prior to this, we had to manually create the vfiler then stop and resync it. All mirrors in the vfiler must be initialized prior to using “-u”. Procedure (7.3.5) This step will show how to create a DR vfiler if mirrors already exist before vfiler DR. vfiler dr configure initializes mirrors from scratch which may not be desired if mirrors already exist for volumes. This procedure will create the dr vfiler by manually creating it on the target (all mirrors done manually) then doing a dr resync instead of a dr configure. If a customer has TeraBytes of Mirrors already running, then don’t want to run “vfiler dr configure” which will re-initialize those target mirrors. This procedure is also useful for when we ship drives between sites, snapmirror to tape, or use a third in the middle FAS controller to initialize mirrors.** Using the fas6280b_vfiler1 source vfiler to the FAS3270 already has mirrors running. At a customer this might be a 10TB usable volume and you would not want to re-init with the “vfiler dr configure” method. So, mirror remaining volumes manually and manually create the dr vfiler, then run “vfiler dr resync” to create the destination DR vfiler. Mirror any remaining volumes NOT mirrored that are in the source vfiler (make sure to use the same volumes names on source and dest) – REPEAT for ALL volumes NOT mirrored yet – simulate mirrors already exist o fas3270> vol restrict fas6280b_vfiler1_root o fas3270> vol restrict fas6280b_vfiler1_nas o fas3270> vol restrict fas6280b_vfiler1_san o fas3270> snapmirror initialize -S fas6280b:fas6280b_vfiler1_root fas6280b_vfiler1_root o fas3270> snapmirror initialize -S fas6280b:fas6280b_vfiler1_nas fas6280b_vfiler1_nas o fas3270> snapmirror initialize -S fas6280b:fas6280b_vfiler1_san fas6280b_vfiler1_san o fas3270> vol create new_vol -s none aggr1 20m # create new missing volume o fas3270> vol restrict new_vol 9 MultiStore Hands-On Lab 5: vFiler DR o fas3270> snapmirror initialize -S fas6280b:new_vol new_vol NOW CREATE THE vFILER DR USING the ‘-u’ option.. much easier than resync and the extra steps below to get the same result. fas3270> vfiler dr configure -u -c secure -l root:netapp01 fas6280b_vfiler1@192.168.150.101 10 fas3270> snapmirror status fas3270> vfiler status # note vfiler is now a dr vfiler Update/Fix /etc/snapmirror.conf for 3 minute interval update that was automatically updated by resync MultiStore Hands-On Lab 5: vFiler DR SKIP *** Prior to ONTAP 7.3.5 *** SKIP TO NEXT STEP “CLEANUP” - SINCE OUR SIM is 7.3.5 – REFERENCE ONLY This step will show how to create a DR vfiler if mirrors already exist before vfiler DR. vfiler dr configure initializes mirrors from scratch which may not be desired if mirrors already exist for volumes. This procedure will create the dr vfiler by manually creating it on the target (all mirrors done manually) then doing a dr resync instead of a dr configure. If a customer has TeraBytes of Mirrors already running, then don’t want to run “vfiler dr configure” which will re-initialize those target mirrors. This procedure is also useful for when we ship drives between sites, snapmirror to tape, or use a third in the middle FAS controller to initialize mirrors.** Procedure (pre-7.3.5) o Using the fas6280b_vfiler1 source vfiler to the FAS3270 already has mirrors running. At a customer this might be a 10TB usable volume and you would not want to re-init with the “vfiler dr configure” method. So, mirror remaining volumes manually and manually create the dr vfiler, then run “vfiler dr resync” to create the destination DR vfiler. o Mirror any remaining volumes NOT mirrored that are in the source vfiler (make sure to use the same volumes names on source and dest) – REPEAT for ALL volumes NOT mirrored yet – simulate mirrors already exist fas3270> vol restrict fas6280b_vfiler1_root fas3270> vol restrict fas6280b_vfiler1_nas fas3270> vol restrict fas6280b_vfiler1_san fas3270> snapmirror initialize -S fas6280b:fas6280b_vfiler1_root fas6280b_vfiler1_root fas3270> snapmirror initialize -S fas6280b:fas6280b_vfiler1_nas fas6280b_vfiler1_nas fas3270> snapmirror initialize -S fas6280b:fas6280b_vfiler1_san fas6280b_vfiler1_san o Quiese and Break all mirrors on target – REPEAT for ALL volumes fas3270> snapmirror status # wait until mirrors are initialized – REPEAT COMMAND UNTIL Mirrors are IDLE fas3270> snapmirror quiesce fas6280b_vfiler1_root fas3270> snapmirror break fas6280b_vfiler1_root fas3270> snapmirror quiesce fas6280b_vfiler1_nas fas3270> snapmirror break fas6280b_vfiler1_nas fas3270> snapmirror quiesce fas6280b_vfiler1_san fas3270> snapmirror break fas6280b_vfiler1_san o Manually create the vfiler on the destination on the broken mirrors (you only need the root volume of the vfiler (using -r option) from the source since it will 11 MultiStore Hands-On Lab 5: vFiler DR pick up member volumes automatically from the root configuration – the network will be unconfigured) fas3270> vfiler create fas6280b_vfiler1 -r /vol/fas6280b_vfiler1_root o Check the Status and Stop the vfiler fas3270> vfiler status -a fas3270> vfiler stop fas6280b_vfiler1 fas3270> vfiler status -a o If the vFiler IP Address(es) are not correct, perform the next 3 steps to add the correct IP and remove the wrong IP…. Add .105 fas3270> vfiler add fas6280b_vfiler1 -i 192.168.150.105 fas3270> vfiler status -a Remove .104 fas3270> vfiler remove fas6280b_vfiler1 -i 192.168.150.104 Bring up the .105 interface so it is configured (using an alias since ns1 is already used for fas6280a_vfiler1 with the .103 IP).. you would also edit /etc/rc.. see below. fas3270> ifconfig ns1 alias 192.168.150.105 fas3270> vfiler status -a # note that the interface is now configured Edit /etc/rc on fas3270 (.102) to add the ifconfig for this interface Optionally – use vfiler setup FAS3270> vfiler run vfilername setup -e ns1:ip:subnet - **** NOTE: You can ifconfig a new interface or ifconfig alias an existing interface from vfiler0, or run “vfiler run vfilername setup -e interface:ip:subnet”. Setup will wack several setup files (hosts, hosts.equiv, resolv.conf, exports, nsswitch.conf) so be careful to restore from .bak if needed. Setup -e will create an alias if the interface is already in use. Also note that the vfiler route add default is not in the 3270 and should be added if the controller were to reboot and need the route to a different subnet. CIFS to \\192.168.150.102\c$ Open \etc\rc … add the line below the ifconfig of ns1 o ifconfig ns1 alias 192.168.150.105 fas3270> vfiler status -a o Backup /etc/snapmirror.conf first since the file will be modified so that the mirror schedules for the vFiler’s volumes are replicated every 3 minutes. Save the file so you can change back to the prior schedule. fas3270> rdfile /etc/snapmirror.conf o Run a DR resync from source to target *** THIS IS THE TRICK to RESYNC even though DR CONFIG was NEVER done, but resync still works as long as all names, ipspaces, active mirrors and volumes are the same and exist. 12 MultiStore Hands-On Lab 5: vFiler DR fas3270> vfiler stop fas6280b_vfiler1 # if not stopped already fas3270> vfiler status # note vfiler is stopped fas3270> vol create new_vol -s none aggr1 20m # create new missing volume fas3270> vol restrict new_vol fas3270> snapmirror initialize -S fas6280b:new_vol new_vol *** WAIT FOR VOLUME to show online (console message) from the vFiler *** fas3270> snapmirror quiesce new_vol fas3270> snapmirror break new_vol fas3270> vfiler dr resync -c secure -l root:netapp01 fas6280b_vfiler1@192.168.150.101 # instead of dr configure, we resync the manually created vfiler. fas3270> snapmirror status fas3270> vfiler status # note vfiler is now a dr vfiler o Update/Fix /etc/snapmirror.conf for 3 minute interval update that was automatically updated by resync 13 MultiStore Hands-On Lab 5: vFiler DR Cleanup 14 Stop the DR vfiler on FAS6280B and activate the vFiler on the FAS3270 o fas6280b> vfiler stop fas6280b_vfiler1 o fas6280b> vfiler status -a # confirm fas6280b_vfiler1 is stopped o fas3270> vfiler dr activate fas6280b_vfiler1@192.168.150.101 o fas3270> vfiler status -a # confirm fas6280b_vfiler1 is running Halt the cluster simulator nodes since we will do the rest of the labs on the FAS3270 o fas6280a> halt -f o fas6280b> halt -f MultiStore Hands-On Lab 5: vFiler DR NetApp provides no representations or warranties regarding the accuracy, reliability or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer’s responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document. © Copyright 2011 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, xxx, and xxx are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. <<Insert third-party trademark notices here.>> All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. 15 MultiStore Hands-On Lab 5: vFiler DR