Support Knowledge Base Article Article Number: 000066107 Print Email English Audience Level: Customer ARTICLE CONTENT ARTICLE PROPERTIES Article Content Symptoms RATE THIS ARTICLE Performance issues can take a long time to solve as they are often intermittent or have vague symptoms that make them difficult to pinpoint. In order to accurately troubleshoot we need a wide variety of data and it is best if the bulk of this data is gathered while the issue is actually occurring. This may make it difficult with intermittent issues, but it provides the best chance of narrowing down the issue. Resolution We need answers to these questions: Are you receiving any errors? If so, what is the exact syntax of the error and what application is producing the error? What is slow? Transfer speed to clients? Transfer speed to cluster? Reads? Writes? File/Folder open? Share browsing/file listing? Are there CPU usage issues? What performance numbers are you getting? What performance numbers are you expecting? How many clients are experiencing this issue? Is it intermittent or constant? Does it change if you connect to a different node? Does it change if the client is moved to a different port on the switch? Does it change if the client uses a different cable? Does it happen when connecting to another share/mount on other equipment using the same protocol (another computer or NAS)? What is your authentication method? Client OS Version/Service Pack/Kernel? What protocols are you using? If SMB, are you using UNC paths or mapped drives? If NFS, what are your mount options? If NFS, are you using a standard mount or autofs? Are you using smartconnect? Do you connect via the short-name, fully qualified domain name, or ip? Contact Us Isilon Performance Info Gathering What is the MTU of the cluster set at? What is the MTU on your clients set at? What is the MTU on your switches set at? What is the Link rate of the client? What is the Link rate of the cluster? What are your cable lengths? How many hops between client and cluster? When the issue is occurring, go through this sequence: 1)From the cluster, enable SMB logging and create a directory to gather information: isi smb log-level --set=debug mkdir /ifs/data/Isilon_Support/gather 2)From the client, begin a tcpdump: tcpdump -w client.pcap -i <interface> host <nodeIP> 3)From the cluster, begin a tcpdump: tcpdump -w cluster.pcap -i <interface> host <nodeIP> 4)From the client, go to the mount directory for the cluster, run a dd: (time dd if=/dev/zero of=1GBfile bs=1024k count=1024 > ddwrite) 1> /dev/null 2>> ddwrite; (time dd if=1GBfile of=/dev/null bs=1024 > ddread) 1> /dev/null 2>> ddread; NOTE: You should run any other commands indicative of the issue here as well, such as ls on a large directory, cat on a large file, etc 5)End both tcpdumps 6)From the same mount directory on the cluster, pull info and upload everything to the Isilon: cp ddwrite /ifs/data/Isilon_Support/gather/; cp ddread /ifs/data/Isilon_Support/gather/; date > /ifs/data/Isilon_Support/gather/date; isi job status -v > /ifs/data/Isilon_Support/gather/job_status; isi statistics query --nodes=all --stats=node.disk.xfers.rate.sum > /ifs/data/Isilon_Support/gather/disk_xfers; isi statistics client > /ifs/data/Isilon_Support/gather/clients; isi_for_array -s sysctl hw.iosched | grep total_inqueue > /ifs/data/Isilon_Support/gather/hw.iosched; isi statistics system --nodes > /ifs/data/Isilon_Support/gather/system; isi statistics protocol --orderby=TimeAvg > /ifs/data/Isilon_Support/gather/protocol; isi statistics query --nodes=all --stats=node.clientstats.connected.cifs,node.clientstats.active.cifs > /ifs/data/Isilon_Support/gather/clients_cifs; isi statistics protocol --nodes=all --protocols=smb1 > /ifs/data/Isilon_Support/gather/protocol_smb1; isi statistics protocol --nodes=all --protocols=smb2 > /ifs/data/Isilon_Support/gather/protocol_smb2; isi statistics query --nodes=all --stats=node.clientstats.connected.nfs,node.clientstats.active.nfs > /ifs/data/Isilon_Support/gather/clients_nfs; isi statistics protocol --nodes=all --protocols=nfs3 > /ifs/data/Isilon_Support/gather/protocol_nfs3; isi statistics protocol --nodes=all --protocols=nfs4 > /ifs/data/Isilon_Support/gather/protocol_nfs4; isi statistics drive -nall --orderby=busy > /ifs/data/Isilon_Support/gather/drive_busy; isi statistics pstat > /ifs/data/Isilon_Support/gather/pstat; isi perfstat > /ifs/data/Isilon_Support/gather/perfstat; nfsstat > /ifs/data/Isilon_Support/gather/nfsstat; sysctl -aN hw.iosched | grep bios_inqueue > /ifs/data/Isilon_Support/gather/bios_inqueue; sysctl efs.gmp.infos > /ifs/data/Isilon_Support/gather/efs.gmp.infos; sysctl efs.gmp.group > /ifs/data/Isilon_Support/gather/efs.gmp.group; Contact Us What is the model name and number of all your switches/routers? sysctl net.inet > /ifs/data/Isilon_Support/inet; isi_for_array sysctl kern.openfiles > /ifs/data/Isilon_Support/gather/kern.openfiles; isi networks list pools -v > /ifs/data/Isilon_Support/gather/flexnet_pools; isi networks list subnets -v > /ifs/data/Isilon_Support/gather/flexnet_subnets; isi networks list interface -v > /ifs/data/Isilon_Support/gather/flexnet_interfaces; cp /var/log/audit/smb.log /ifs/data/Isilon_Support/gather/smb.log; cd /ifs/data/Isilon_Support/gather && tar -czvf /ifs/data/Isilon_Support/gather/performance.tgz *; isi_gather_info --nologs -f /ifs/data/Isilon_Support/gather/performance.tgz -s "isi_hw_status -i" -s "ifconfig" -s "uname -a" -s "df" -s "uptime" --group protocol --group network --group usage 7)Turn off SMB logging and delete gather information: isi smb log-level --set=error rm -rf /ifs/data/Isilon_Support/gather Article Properties Affected Product Isilon Product PowerScale OneFS Last Published Date 20 Nov 2020 Version 2 Article Type Solution Rate This Article All fields are required unless marked otherwise. Accurate Useful Easy to Understand Was this article helpful? Yes No Contact Us cp /var/log/lsassd.log /ifs/data/Isilon_Support/gather/lsassd.log; cp /var/log/lwiod.log /ifs/data/Isilon_Support/gather/lwiod.log; Additional Information (optional) Submit Feedback Your Recently Viewed Articles Isilon Intermittent slow performance seen while accessing files for the first time over SMB Isilon On-Cluster Analysis - Boot Disks Life Remaining - Event ID: 400120001 Isilon OneFS: SSD pool is nearing 100% or significantly more full than the associated HDD pool - Event ID: 100010018 Dell EMC VxRail: ESXi host shows 'not responding' or 'disconnected' in vCenter (Customer Correctable) PowerStore: PowerStore alerts are not reported on the Remote Syslog Server Load more results Contact Us 0/3000 characters
