Webinar Series
In-depth Troubleshooting on NetScaler
using Command Line Tools
27 March 2014
Andrew Sandford
Senior Readiness Specialist, Worldwide Support Readiness EMEA
Agenda
> show
>add
>remove
>set
>enable
>disable
>force
>bind
>unbind
>help <command>
>man <command>
#
© 2014 Citrix | Confidential – Do Not Distribute
# tar
# head
# less
# more
# cat
# zcat
# ls
# find
>stat ssl
SSL Summary
System
Transactions Rate (/s) Total
SSL transactions 0 301
SSLv2 transactions 0 0
SSLv3 transactions 0 0
TLSv1 transactions 0 301
# sysctl -a netscaler
netscaler.descr:
netscaler.developer: 0 NetScaler Virtual Appliance 3G
netscaler.recovery: 0
netscaler.num_pe_running:
netscaler.sysid: 450000 1
netscaler.version: NetScaler NS10.1: Build
netscaler.serial: 98310000cb254307ee78
netscaler.descr: NetScaler Virtual Appliance 3G
124.13.nc, Date: Feb 20 2014, 18:53:27
netscaler.num_pe_running: 1
netscaler.model:
netscaler.version: NetScaler3000
NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
netscaler.model: 3000
netscaler.vmpe_max_cpus: 2
netscaler.vmpe_max_cpus: 2
netscaler.nCore:
netscaler.Classic: 0 1
netscaler.nCore: 1
> show ip
> show ip
Ipaddress TD Type Mode Arp Icmp Vserver State
--------- -- ---- ---- --- ---- ------- ------
1) 192.168.196.45 0 NetScaler IP Active Enabled Enabled NA
Enabled
2) 192.168.196.146 0 SNIP Active Enabled Enabled NA
Enabled
3) 192.168.196.147 0 VIP Active Enabled Enabled Enabled
Enabled
4) 22.22.22.2 0 VIP Active Enabled Enabled Enabled Enabled
5) 192.168.100.44 0 SNIP Active Enabled Enabled NA Enabled
6) 192.168.1.2 0 SNIP Active Enabled Enabled NA Enabled
7) 192.168.196.148 0 VIP Active Enabled Enabled Enabled
Enabled
#uptime
PID USERNAME
50185 root
THR PRI NICE
1 44 -52
SIZE
814M
100%
RES STATE
815M CPU1
C
1
TIME WCPU COMMAND
20.2H 100.00% NSPPE-00
11834 nobody 1 4 0 26448K 18276K accept 0 0:02 0.05% httpd
50206 root 1 4 0 61364K 11632K kqread 0 1:03 0.00% nsaggregator
50251 root 1 4 0 30504K 7964K kqread 0 0:47 0.00% nsconfigd
44 root 1 4 0 15880K 1828K kqread 0 0:21 0.00% pitboss
995 root 1 8 0 25660K 17672K nanslp 0 0:13 0.00% httpd
987 root 1 96 0 3668K 876K select 0 0:11 0.00% syslogd
50188 root 1 4 0 59184K 9016K kqread 0 0:10 0.00% nsnetsvc
> show ip
# ping
# traceroute
# telnet
Persistence: NONE
New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
Warning: Feature(s) not enabled [LB]
Warning:
Vserver Feature(s)
IP and not OFF
Port insertion: enabled [LB]
© 2014 Citrix | Confidential – Do Not Distribute
show lb vserver [vservername]
> sh persistentSessions
Type SRC-IP DST-IP PORT VSNAME TIMEOUT PERSISTENCE-
PARAMETER
SOURCEIP 192.168.119.81 192.168.196.61 80 LB_RGB 118 192.168.119.81
# cat /tmp/aaad.debug
# ls -la
total 8
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .
drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..
drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1
-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds
# ls –la 1/
total 10292
drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..
-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz
-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz
Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask
is 0x0 0x0 0x0 0x0
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00
(1077) unexpectedly died due to receiving signal
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may
be a delay restarting process while collecting core dump on NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc
NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to
policy setting (0x29ac)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler
restart may be delayed if collecting core dump for NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss
declaring system failure: NSPPE-00 (1077) exited
Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting
watch on NSPPE-00 (1077) for ()
# cd /var/core/
# ls -la
total 14
drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .
drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..
drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1
-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds
root@vpx1# ls 1/
NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz
aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz
imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz
monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz
nsaaad-1131.gz nsconfigd-1156.gz
nsaggregatord-1107.gz nsfsyncd-1110.gz
NodeHello
State: STAYSECONDARY
Interval: 200 msecs
Dead Interval: 3 secs
Local node information:
Critical Interfaces: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Master State: Secondary
Node in this Master State for: 0:15:2:13
(days:hrs:min:sec) >
>stat ns
>stat cpu
>stat interface
System
>stat lb vserver
>stat cs vserver
>stat service
Entities
>stat ssl
>stat dns
>stat http
Protocols
© 2014 Citrix | Confidential – Do Not Distribute
stat ns
> stat ns
System overview
CPU statistics
ID Usage
1 0
7 0
6 0
5 0
4 0
3 1
2 0
Interface Summary
ID IntfState IntfAlias Rx Bytes Tx Bytes Rx Pkts Tx Pkts
1/8 DOWN 0 0 0 0
1/7 DOWN 0 0 0 0
1/6 DOWN 0 0 0 0
1/5 DOWN 0 0 0 0
1/4 DOWN 0 0 0 0
1/3 DOWN 0 0 0 0
1/2 DOWN 0 0 0 0
1/1 UP 14476M 21813M 442178k 56718611
10/2 DOWN 0 0 0 0
10/1 DOWN 0 0 0 0
0/1 UP 32027M 18048M 292060k 67610607
0/2 DOWN 0 0 0 0
LO/1 UP 831255M 1218G 6624M 13125M
LA/1 DOWN LA2 0 0 0 0
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0c 286M 245M 35M 88% /
devfs 1.0K 1.0K 0B 100% /dev
procfs 4.0K 4.0K 0B 100% /proc
/dev/ad0s1a 1.4G 965M 368M 72% /flash
/dev/ad0s1e 14G 3.1G 9.5G 24% /var
/var/log/ns.log
/var/nslog/newnslog/newnslog.ppe.x Logs
/var/nsinstall/build-10.1-124.13_nc.tgz
/flash/ns-10.1-124.13.gz Firmware
/var/core/NSPPE-00-353.gz
/var/crash/vmcore.0
Crash Files
/nsconfig/ns.conf NetScaler Configuration
/nsconfig/ssl/ns-root.cert SSL Certificates
/nsconfig/monitors/nssmtp.pl Monitor Scripts
/nsconfig/license/FID__b0d70c6_13b16ab7034_573f.lic License Files
/netscaler/nsconmsg NetScaler
NetScaler Binaries
Binaries
/var/nstrace/nstrace1.cap Packet
Packet Trace
Trace Files
Files
root@ns# dmesg
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-NETSCALER-10.1 #0: Thu Feb 20 18:54:22 PST 2014
root@sjcpbldbsd6301.eng.citrite.net:/usr/obj/amd64/usr/home/build/rs_101_124_8/usr.src/sys/NS64
Preloaded elf kernel "/ns-10.1-124.13" at 0xffffffff930b3000.
Calibrating clock(s) ... i8254 clock: 1189606 Hz
CLK_USE_I8254_CALIBRATION not specified - using default frequency
Timecounter "i8254" frequency 1193182 Hz quality 0
Calibrating TSC clock ... TSC clock: 3325066248 Hz
CPU: Intel(R) Core(TM)2 Duo CPU E8600 @ 3.33GHz (3325.07-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x1067a Stepping = 10
Features=0x789fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x81282201<SSE3,SSSE3,CX16,<b19>,<b21>,<b24>,<b31>>
AMD Features=0x20000800<SYSCALL,LM>
AMD Features2=0x1<LAHF>
real memory = 2143289344 (2044 MB)
# ls
auth.log httperror.log ns.log
auth.log.0.gz httperror.log.0 ns.log.0
callhome.log lastlog nscollect.log
callhomedebug.log license.log nscollect_cl.log
cron lpd-errs nsvpn.log
cron.0.gz maillog nsvpnd.log
ctxslsboc.log maillog.0.gz ntpd.log
db messages security
httpaccess.log messages.0 snmpd.log
httpaccess.log.0.gz nitro.log wicmd.log
httpd.scoreboard nitro.log.0.gz
# cd /var/nslog/
# ls
asl lspci_tv.last nsagg.conf
aslearn.log lspci_tv.prev nsagg.log
aslearn_old_db.tar.gz lspci_vvvxxx.boot nslog.nextfile
conmsg.log lspci_vvvxxx.last nsumond
dmesg.boot lspci_vvvxxx.prev nsumond.log
dmesg.last newnslog snmpdebug.log
dmesg.prev newnslog.0.gz
lspci_tv.boot ns.log
# cd newnslog
# ls
newnslog.ppe.0
View events
View console messages
Use View statistics
NOT -k
# nsconmsg –K newnslog -d consmsg
# nsconmsg –K newnslog -d oldconmsg
# nsconmsg –K newnslog -s ConLb=2 –d oldconmsg
# nsconmsg –K newnslog -s ConDebug=1 –d oldconmsg
# cd /var/nslog/newnslog
# nsconmsg -K newnslog.ppe.0 -d current -g pol_hits
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
# cat /var/log/ns.log
Aug 14 12:39:11 <local0.info> 192.168.47.1
ipConflict (ipConflictAddr = 192.168.47.1,
sysIpAddress = 192.168.47.1)
Aug 14 12:39:42 <local0.info> 192.168.47.1 last
message repeated 169 times
NetScaler software version including build – (from the ‘show version’ NetScaler
Command Line Interface (NSCLI) command) – example: 10.1.124.13
Production setup or new installation
Whether an application/service that was working is now broken or whether the
user wants to configure an application/service
Network topology information
What changes were performed on the NetScaler appliance prior to the issue
Any change(s) on the connected Switches, upstream Router, or backend
server prior to experiencing the issue
ns.conf after saving the configuration - (from the save configuration NSCLI
command)
If this node is part of HA pair, please run it on the other node also!!
© 2014 Citrix | Confidential – Do Not Distribute
techsupport archive structure
/etc
/flash
/nsconfig
/shell
/var
/cfsynclog
/clusterd
/core
/crash
/download
/log
/netscaler
/nslog
/nsproflog
/nssynclog
nstrace nstcpdump
Nstcpdump.sh Nstrace.sh
Useful if traces are to be viewed on Useful for offline collection
standard output nstrace.sh –sz 0
nstcpdump.sh –w <filename> option Saves traces in /var/nstrace in cap format
helps writing output to file
can be used with expressions so that you get useful for collection of traces in separate log
to see filtered traffic files based on NICs
# nstrace.sh --?
-h - prints this message - exclusive option
nstrace - utility to start NetScaler packets trace
-nf
usage: - number
nstrace.sh of <number_of_files]
[-h] [-nf files to be generated
[-time <time>] in
[-m cycle
<mode>] (def. 24)
[-nic <boolean>]
-h - prints this message - exclusive option
-time - seconds per file (def. 3600) (could be an expression)
-nf - number of files to be generated in cycle (def. 24)
-sz -
-time size of the
- seconds captured
per file data
(def. 3600) (bytes
(could from 60 to 1514)
be an expression)
-sz - size of the captured data (bytes from 60 to 1514)
-tcpdump
-m - 0=nstrace-format (default)
- Capturing mode: sum of or 1=tcpdump-format
the values (def. 18):
-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format
-nic -nic - use separate trace files for each interface (only works if -tcpdump
- use separate trace files for each interface
option is set) (only works if -tcpdump option is set)
-name -name
- - name
name ofofthe
the trace
trace file
file
-filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of
-filter - Filter
following format: expression
<expression> [<relop>for nstrace. The maximum length of filter expression
<expression>]
is 255- and
-link it canconnection's
Log filtered of following
peer's format: <expression>
(linked connection's) [<relop>
traffic. Works only <expression>]
with -filter option
-id - ID for the trace file name for uniqueness. Should be used only with -name option
-link - Log filtered connection's peer's (linked connection's) traffic. Works
-stop - can be used to disable tracing (when 'nstrace.sh' is run in the background)
only
#
with -filter option
nstrace.sh nstcpdump.sh
Proprietary capture format TCPdump PCAP capture format
Native format captures more Useful for live capture from CLI
information
Files are stored in /var/nstrace Option to write to a file
Needs custom dissector in Most TCPdump options supported
Wireshark (1.6+)
Works in standard Wireshark
The trace capture runs for 1 hr. (3600 sec) if not interrupted
Files are cyclically numbered from 1-24. Trace mode is 6 (that is the TXB
and RX packets are captured)
Command Purpose
# nstrace.sh Stores the traces in default (proprietary) format
# nstrace.sh Begins to save the traces in the TCPDUMP format for a default 3600
-tcpdump 1 seconds
# nstrace.sh Logs the traces (in TCPDUMP format) into separate log files based on
-tcpdump 1 -nic 1 the NIC IDs
# nstrace.sh –nf No of files to be generated in cycle by default is 24
<value>
# nstrace.sh –time Seconds per file by default 3600 seconds
<value>
# nstrace.sh -sz 0 Size of the captured data( by default it is 164), -sz 0 is the entire
packet length
# nstrace.sh –m Capturing mode: sum of the values (def. 6):
1-Transmitted packets (TX)
2 - Packets buffered for transmission (TXB)
© 2014 Citrix | Confidential – Do Not Distribute
4 - Received packets (RX)
Trace Analysis
nstcpdump.sh example syntax
Command Purpose
# nstcpdump.sh Possible types are host, net and port .If there is no type
<type> qualifier host assumed
# nstcpdump.sh Qualifiers specify a particular transfer direction to and/or
<dir> from id. Possible directions are src, dst, src or dst and
src and dst. If there is no dir qualifier, src or dst is
assumed. src foo', `dst net 128.3', `src or dst port ftp-
data'
# nstcpdump.sh ether, fddi, ip, arp, rarp, decnet, tcp and udp.
<proto>
# nstcpdump.sh –c Exit after receiving ‘value’ number of packets
<value>
# nstcpdump.sh –F –I Not supported and not to give them as options to the
–r script
© 2014 Citrix | Confidential – Do Not Distribute
# nstcpdump.sh –w Write the raw packets to file rather than parsing and
nstcpdump.sh
# nstcpdump.sh --?
Setting 1000 pages (8000 KB) of trace buffers ... Done.
Enabling all nic trace mode=6 ... Done.
Changing trace packet length from 0 to 0 ... Done.
tcpdump version 3.9.4
libpcap version 0.9.4
Usage: tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
[ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
[ -W filecount ] [ -y datalinktype ] [ -Z user ]
[ expression ]
Saving current trace data in file 'pipe' ... in TCPDUMP format
Disabling all nic trace ... Done.
# nstcpdump.sh port 21
# ls -la
total 8
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .
drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..
drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1
-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds
# ls –la 1/
total 10292
drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..
-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz
-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz
Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask
is 0x0 0x0 0x0 0x0
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00
(1077) unexpectedly died due to receiving signal
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may
be a delay restarting process while collecting core dump on NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc
NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to
policy setting (0x29ac)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler
restart may be delayed if collecting core dump for NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss
declaring system failure: NSPPE-00 (1077) exited
Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting
watch on NSPPE-00 (1077) for ()
# cd /var/core/
# ls -la
total 14
drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .
drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..
drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1
-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds
root@vpx1# ls 1/
NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz
aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz
imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz
monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz
nsaaad-1131.gz nsconfigd-1156.gz
nsaggregatord-1107.gz nsfsyncd-1110.gz
# cd /nsconfig/
# ls -lath ns*
-rw-r--r-- 1 root wheel 16K Oct 30 14:01 ns.conf
-rw------- 1 root wheel 16K Oct 30 14:01 ns.conf.0
-rw------- 1 root wheel 12K Oct 25 09:34 ns.conf.1
-rw------- 1 root wheel 12K Oct 25 08:09 ns.conf.NS10.1-
124.13
-rw------- 1 root wheel 12K Oct 25 07:52 ns.conf.2
-rw------- 1 root wheel 12K Oct 24 15:17 ns.conf.3
-rw------- 1 root wheel 13K Oct 24 08:56 ns.conf.4
Check it out