Anda di halaman 1dari 90

Citrix Support Secrets

Webinar Series
In-depth Troubleshooting on NetScaler
using Command Line Tools
27 March 2014

Andrew Sandford
Senior Readiness Specialist, Worldwide Support Readiness EMEA
Agenda

NetScaler CLI recap


What’s my NetScaler?
How is my NetScaler configured?
What’s my NetScaler doing?
Time for Technical Support?
Q&A
© 2014 Citrix | Confidential – Do Not Distribute
NetScaler CLI
Recap
NetScaler Native CLI

> show
>add
>remove
>set
>enable
>disable
>force
>bind
>unbind

© 2014 Citrix | Confidential – Do Not Distribute


NetScaler Native CLI

>help <command>

>man <command>

>set cli prompt %u@%h-%T

>set cli mode -color ON

© 2014 Citrix | Confidential – Do Not Distribute


NetScaler BSD Shell

#
© 2014 Citrix | Confidential – Do Not Distribute
# tar
# head
# less
# more
# cat
# zcat
# ls
# find

© 2014 Citrix | Confidential – Do Not Distribute


What’s my NetScaler?
What’s my NetScaler?

> show version

> show ns hostname

> show hardware

> show interface –summary

>stat ssl

#sysctl –a netscaler | more

© 2014 Citrix | Confidential – Do Not Distribute


show version/show ns hostname

> show version


NetScaler NS10.1: Build 124.13.nc, Date: Feb 20
2014, 18:53:27

> show ns hostname


Hostname: nstipster

© 2014 Citrix | Confidential – Do Not Distribute


show hardware

> show hardware


Platform: NSMPX-10500 8*CPU+2*E1K+8*E1K+2*IX+8*CVM
1620 760100
Manufactured on: 10/7/2010
CPU: 2832MHZ
Host Id: 1234567890
Serial no: M123456789
Encoded serial no: M123456789

© 2014 Citrix | Confidential – Do Not Distribute


show interface -summary

> show interface -summary


--------------------------------------------------------------------------------
Interface MTU MAC Suffix
--------------------------------------------------------------------------------
1 0/1 1500 00:25:90:12:eb:5a Gig Ethernet 10/100/1000 MBits
2 0/2 1500 00:25:90:12:eb:5b Gig Ethernet 10/100/1000 MBits
3 1/1 1500 00:e0:ed:1a:24:97 Gig Ethernet, copper SFP
4 1/2 1500 00:e0:ed:1a:24:96 Gig Ethernet, copper SFP
5 1/3 1500 00:e0:ed:1a:24:95 Gig Ethernet, no SFP found
6 1/4 1500 00:e0:ed:1a:24:94 Gig Ethernet, no SFP found
7 1/5 1500 00:e0:ed:1a:24:a3 Gig Ethernet, copper SFP
8 1/6 1500 00:e0:ed:1a:24:a2 Gig Ethernet, no SFP found
9 1/7 1500 00:e0:ed:1a:24:a1 Gig Ethernet, no SFP found
10 1/8 1500 00:e0:ed:1a:24:a0 Gig Ethernet, no SFP found
11 10/1 1500 00:1b:21:77:c0:35 10G Ethernet,...SFP+/SFP found
12 10/2 1500 00:1b:21:77:c0:34 10G Ethernet,...SFP+/SFP found
13 LO/1 1500 00:25:90:12:eb:5a NetScaler Loopback interface

© 2014 Citrix | Confidential – Do Not Distribute


stat ssl

> stat ssl

SSL Summary

# SSL cards present 8


# SSL cards UP 8
SSL engine status 1
SSL sessions (Rate) 0

System
Transactions Rate (/s) Total
SSL transactions 0 301
SSLv2 transactions 0 0
SSLv3 transactions 0 0
TLSv1 transactions 0 301

© 2014 Citrix | Confidential – Do Not Distribute


sysctl

# sysctl -a netscaler

netscaler.descr:
netscaler.developer: 0 NetScaler Virtual Appliance 3G
netscaler.recovery: 0
netscaler.num_pe_running:
netscaler.sysid: 450000 1
netscaler.version: NetScaler NS10.1: Build
netscaler.serial: 98310000cb254307ee78
netscaler.descr: NetScaler Virtual Appliance 3G
124.13.nc, Date: Feb 20 2014, 18:53:27
netscaler.num_pe_running: 1
netscaler.model:
netscaler.version: NetScaler3000
NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
netscaler.model: 3000
netscaler.vmpe_max_cpus: 2
netscaler.vmpe_max_cpus: 2
netscaler.nCore:
netscaler.Classic: 0 1
netscaler.nCore: 1

© 2014 Citrix | Confidential – Do Not Distribute


How is my NetScaler
configured?
How is my NetScaler configured?

> show ip

> show feature

> show ns mode

> show info

> show license

> show run | more

© 2014 Citrix | Confidential – Do Not Distribute


show ip

> show ip
Ipaddress TD Type Mode Arp Icmp Vserver State
--------- -- ---- ---- --- ---- ------- ------
1) 192.168.196.45 0 NetScaler IP Active Enabled Enabled NA
Enabled
2) 192.168.196.146 0 SNIP Active Enabled Enabled NA
Enabled
3) 192.168.196.147 0 VIP Active Enabled Enabled Enabled
Enabled
4) 22.22.22.2 0 VIP Active Enabled Enabled Enabled Enabled
5) 192.168.100.44 0 SNIP Active Enabled Enabled NA Enabled
6) 192.168.1.2 0 SNIP Active Enabled Enabled NA Enabled
7) 192.168.196.148 0 VIP Active Enabled Enabled Enabled
Enabled

© 2014 Citrix | Confidential – Do Not Distribute


show feature

> show feature


Feature Acronym Status
------- ------- ------
1) Web Logging WL ON
2) Surge Protection SP OFF
3) Load Balancing LB ON
4) Content Switching CS ON
5) Cache Redirection CR OFF
6) Sure Connect SC ON
8) Priority Queuing PQ ON
9) SSL Offloading SSL ON
10) Global Server Load Balancing GSLB ON
11) Http DoS Protection HDOSP OFF
12) Content Filtering CF ON
13) Integrated Caching IC OFF
19) Rewrite REWRITE ON

© 2014 Citrix | Confidential – Do Not Distribute


show ns mode

> show ns mode


Mode Acronym Status
------- ------- ------
1) Fast Ramp FR ON
2) Layer 2 mode L2 OFF
3) Use Source IP USIP OFF
4) Client Keep-alive CKA OFF
5) TCP Buffering TCPB OFF
6) MAC-based forwarding MBF OFF
7) Edge configuration Edge ON
8) Use Subnet IP USNIP ON
9) Layer 3 mode (ip forwarding) L3 ON
10) Path MTU Discovery PMTUD ON
16) Bridge BPDUs BridgeBPDUs OFF

© 2014 Citrix | Confidential – Do Not Distribute


show info

> show info


NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014,
18:53:27
NetScaler IP: 192.168.47.6 (mask: 255.255.240.0)
NW FWMODE: NOFIREWALL
Number of MappedIP(s): 0
Node: Secondary (Primary is 192.168.47.1)
System Time: Thu Mar 27 08:52:06 2014
Last Config Changed Time: Thu Mar 27 07:52:50 2014
Last Config Saved Time: Tue Mar 25 13:28:21 2014

© 2014 Citrix | Confidential – Do Not Distribute


show run | more

#NS10.1 Build 124.13


# Last modified Thu Mar 27 10:08:20 2014
set ns config -IPAddress 192.168.47.1 -netmask 255.255.240.0
enable ns feature LB CS SSL SSLVPN OSPF REWRITE AppFw RESPONDER HTMLInjection AppFlow
enable ns mode FR L3 Edge USNIP SRADV DRADV PMTUD
set system parameter -natPcbForceFlushLimit 4294967295
set system user nsroot -----------------SNIP---------------- -encrypted
set rsskeytype -rsstype ASYMMETRIC
set lacp -sysPriority 32768 -mac 11:22:33:aa:bb:cc
set ns hostName nstipster
set interface 1/1 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/1
set interface 1/2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/2
set interface 1/3 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/3
set interface LO/1 -haMonitor OFF -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype Loopback
-ifnum LO/1
add ns ip6 fe80::40da:5dff:fe08:296b/64 -scope link-local -type NSIP -vlan 1 -vServer DISABLED -mgmtAccess
ENABLED -dynamicRouting ENABLED

© 2014 Citrix | Confidential – Do Not Distribute


What’s my NetScaler
doing?
Processes and Uptime

> shell top

#ps –ax | more

#uptime

© 2014 Citrix | Confidential – Do Not Distribute


#top/nsppe

> shell top


last pid: 13825; load averages: 1.00, 1.04, 1.02 up 1+19:48:58 08:38:17
60 processes: 2 running, 58 sleeping
CPU states: 18.0% user, 0.0% nice, 32.1% system, 0.0% interrupt, 49.9% idle
Mem: 99M Active, 51M Inact, 1492M Wired, 15M Cache, 165M Buf, 2564K Free
Swap: 4500M Total, 12K Used, 4500M Free

PID USERNAME
50185 root
THR PRI NICE
1 44 -52
SIZE
814M
100%
RES STATE
815M CPU1
C
1
TIME WCPU COMMAND
20.2H 100.00% NSPPE-00
11834 nobody 1 4 0 26448K 18276K accept 0 0:02 0.05% httpd
50206 root 1 4 0 61364K 11632K kqread 0 1:03 0.00% nsaggregator
50251 root 1 4 0 30504K 7964K kqread 0 0:47 0.00% nsconfigd
44 root 1 4 0 15880K 1828K kqread 0 0:21 0.00% pitboss
995 root 1 8 0 25660K 17672K nanslp 0 0:13 0.00% httpd
987 root 1 96 0 3668K 876K select 0 0:11 0.00% syslogd
50188 root 1 4 0 59184K 9016K kqread 0 0:10 0.00% nsnetsvc

© 2014 Citrix | Confidential – Do Not Distribute


#ps –ax | more

root@ns# ps -ax | more


PID TT STAT TIME COMMAND
0 ?? WLs 0:00.44 [swapper]
1 ?? ILs 0:00.09 /sbin/init --
2 ?? DL 0:02.35 [g_event]
3 ?? DL 0:09.00 [g_up]
4 ?? DL 0:02.21 [g_down]
5 ?? DL 0:00.00 [xpt_thrd]
6 ?? DL 0:00.00 [acpi_task_0]
7 ?? DL 0:00.00 [acpi_task_1]
8 ?? DL 0:00.00 [acpi_task_2]
9 ?? DL 0:00.00 [kqueue taskq]
10 ?? RL 0:10.39 [idle: cpu1]
--More—(byte 933)

© 2014 Citrix | Confidential – Do Not Distribute


IP related

> show route

> show ip

> show dns addrec -type proxy

# ping

# traceroute

# telnet

© 2014 Citrix | Confidential – Do Not Distribute


> show route

> show route


Network Netmask Gateway/OwnedIP State TD
Type
------- ------- --------------- ----- --
----
1) 0.0.0.0 0.0.0.0 192.168.32.1 UP 0
STATIC
2) 127.0.0.0 255.0.0.0 127.0.0.1 UP 0
PERMANENT
3) 192.168.32.0 255.255.240.0 192.168.47.1 UP 0
DIRECT
4) 172.16.200.0 255.255.255.0 192.168.47.2 UP 0
STATIC|ADV

© 2014 Citrix | Confidential – Do Not Distribute


Load Balancing

> show lb vserver

> show service

> show connectiontable | grep <IP Address|port>

> show persistentSessions

© 2014 Citrix | Confidential – Do Not Distribute


show lb vserver

> show lb vserver


LB_RGB
1) LB_RGB (192.168.47.3:80)
(192.168.47.3:80) - HTTP - HTTP
Type: ADDRESS Type: ADDRESS
State: UP
State: UP change was at Wed Aug 14 09:17:14 2013
Last state
Time since last state change: 0 days, 00:30:42.140
Client Idle Timeout: 180 sec
Effective State: UP
Down state flush:
Client Idle Timeout: ENABLED
180 sec
Down state flush: ENABLED
Disable Primary
Disable Vserver
Primary Vserver On: DISABLED
On Down Down : DISABLED
Appflow logging: ENABLED
Appflow logging: ENABLED
Port Rewrite : DISABLED
Port Rewrite
No. of Bound : DISABLED
Services : 1 (Total) 1 (Active)
Configured Method: ROUNDROBIN
No. of Bound Services :
Mode: IP
1 (Total) 1 (Active)
Configured Method:
Persistence: NONE ROUNDROBIN
Vserver IP and Port insertion: OFF
Mode:IcmpResponse:
IP PASSIVE

Persistence: NONE
New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
Warning: Feature(s) not enabled [LB]
Warning:
Vserver Feature(s)
IP and not OFF
Port insertion: enabled [LB]
© 2014 Citrix | Confidential – Do Not Distribute
show lb vserver [vservername]

> sh lb vserver LB_RGB


LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS
State: UP
Last state change was at Wed Aug 14 11:56:19 2013
Time since last state change: 0 days, 20:33:37.60
Effective State: UP
1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1
Client Idle Timeout: 180 sec
Down state flush: ENABLED
No. of Bound Services : 1 (Total) 1 (Active)
Configured Method: ROUNDROBIN
Mode: IP
Persistence: NONE

1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1

© 2014 Citrix | Confidential – Do Not Distribute


show service [servicename]

> show service svc_blue


State:svc_blue
UP (192.168.196.62:80) - HTTP
State: UP
Last state change was at Wed Aug 14 11:56:19 2013
1) Monitor Name: mon-http-ecv
Time since last state change: 0 days, 20:38:31.880
State:
Server Name: Blue UP Weight: 1 Passive: 0
Use Source IP:Probes:
NO 14887 Failed [Total: 124 Current: 0]
Idle timeout: Client: 180 sec Server: 360 sec
Client IP:
Last response: Success - Pattern found in response.
ENABLED ClientIP
Response Time: 10.220 millisec
1) Monitor Name: mon-http-ecv
State: UP Weight: 1 Passive: 0
Probes: 14887 Failed [Total: 124 Current: 0]
Last response: Success - Pattern found in response.
Response Time: 10.220 millisec

© 2014 Citrix | Confidential – Do Not Distribute


show persistentSessions

> sh persistentSessions
Type SRC-IP DST-IP PORT VSNAME TIMEOUT PERSISTENCE-
PARAMETER
SOURCEIP 192.168.119.81 192.168.196.61 80 LB_RGB 118 192.168.119.81

© 2014 Citrix | Confidential – Do Not Distribute


Authentication

> show vpn vserver

> show aaa session

> show aaa stats

# cat /tmp/aaad.debug

© 2014 Citrix | Confidential – Do Not Distribute


How do I troubleshoot
deeper?
Time for Technical
Support?
Core dump directories

# ls -la
total 8
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .
drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..
drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1
-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds
# ls –la 1/
total 10292
drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..
-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz
-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz

© 2014 Citrix | Confidential – Do Not Distribute


Packet Engine (nsppe) Process Crashing

Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask
is 0x0 0x0 0x0 0x0
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00
(1077) unexpectedly died due to receiving signal
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may
be a delay restarting process while collecting core dump on NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc
NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to
policy setting (0x29ac)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler
restart may be delayed if collecting core dump for NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss
declaring system failure: NSPPE-00 (1077) exited
Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting
watch on NSPPE-00 (1077) for ()

© 2014 Citrix | Confidential – Do Not Distribute


/var/core

# cd /var/core/
# ls -la
total 14
drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .
drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..
drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1
-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds
root@vpx1# ls 1/
NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz
aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz
imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz
monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz
nsaaad-1131.gz nsconfigd-1156.gz
nsaggregatord-1107.gz nsfsyncd-1110.gz

© 2014 Citrix | Confidential – Do Not Distribute


Show commands
System show node
show info
show license
Vserver/Service show lb vserver
show cs vserver
show service
show persistencesession
show connectiontable
IP related show route
show ip
show dns addrec -type proxy
Diagnostic show techsupport

© 2014 Citrix | Confidential – Do Not Distribute


show node

> show node 2) Node ID: 1


Node
1) State:
Node ID: 0 NOT UP IP: 192.168.1.45
IP: 192.168.1.145 (NS145) Node State: STAYSECONDARY
Master State: Secondary
Node State: NOT UP Master State: Secondary
Master State: Secondary Fail-Safe Mode: OFF
Master Enabled Interfaces : 1/8 1/7 1/6INC 1/5
Fail-Safe Mode: OFF State: 1/4
DISABLED1/3 1/2 1/1
INC State: DISABLED Sync State: ENABLED
Disabled Interfaces : None
Sync State: ENABLED
Propagation: ENABLED
Propagation: ENABLED
Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 Disabled
1/4 1/3 1/2
Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces 1/1
: None
Disabled Interfaces : None
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces on which
HA MON ON Interfaces : 1/8 1/7 heartbeats are not Interfaces
1/6 1/5 1/4 1/3 1/2 1/1 seen : 1/8heartbeats
on which 1/6 1/5 are not1/4
seen : 1/3
1/8 1/6
Interfaces on which heartbeats are not seen : 1/8 1/6
1/2
1/5 1/4 1/1
1/3 1/2 1/1
1/5 1/4 1/3 1/2 1/1
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3
Interfaces
1/2 1/1 causing Partial Failure:1/2 1/8
1/1
1/6 1/5 1/4 1/3 1/2 1/1
SSL Card Status: UP
SSL Card Status: UP

NodeHello
State: STAYSECONDARY
Interval: 200 msecs
Dead Interval: 3 secs
Local node information:
Critical Interfaces: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Master State: Secondary
Node in this Master State for: 0:15:2:13
(days:hrs:min:sec) >

© 2014 Citrix | Confidential – Do Not Distribute


NetScaler Processes
Process Description Process Description
Sync bookmarks and
nsppe NetScaler Packet Engine nsfsyncd
SSL certificates
RBA and SSL VPN Used by the GUI for
nsaaad nsnetsvc
External Auth config changes
Runs the scriptable
nsconf Writes the ns.conf file nsumond
monitors
Controls Logging for Controls writing of
nslog.sh nsconmsg
newnslog newnslog
Statistics gathering for
nssync HA sync nscollect
historical reporting
Used to read SSL Cert imi/ripd/
nsreadfile Routing processes
Files ospfd/bgpd
nscrlrefresh SSL CRL list update
© 2014 Citrix | Confidential – Do Not Distribute
stat commands

>stat ns
>stat cpu
>stat interface
System
>stat lb vserver
>stat cs vserver
>stat service
Entities
>stat ssl
>stat dns
>stat http
Protocols
© 2014 Citrix | Confidential – Do Not Distribute
stat ns

> stat ns
System overview

Up since Wed Aug 14 11:53:33 2013


CPU usage (%) 0.60
Packet CPU usage (%) 0.60
Management CPU usage (%) 0.80
Memory usage (MB) 190
InUse Memory (%) 18.73
Last Transition time We...013
System state UP
Master state Primary
# SSL cards UP 0
# SSL cards present 0

© 2014 Citrix | Confidential – Do Not Distribute


stat cpu

> stat cpu

CPU statistics
ID Usage
1 0
7 0
6 0
5 0
4 0
3 1
2 0

© 2014 Citrix | Confidential – Do Not Distribute


stat interface

> stat interface

Interface Summary
ID IntfState IntfAlias Rx Bytes Tx Bytes Rx Pkts Tx Pkts
1/8 DOWN 0 0 0 0
1/7 DOWN 0 0 0 0
1/6 DOWN 0 0 0 0
1/5 DOWN 0 0 0 0
1/4 DOWN 0 0 0 0
1/3 DOWN 0 0 0 0
1/2 DOWN 0 0 0 0
1/1 UP 14476M 21813M 442178k 56718611
10/2 DOWN 0 0 0 0
10/1 DOWN 0 0 0 0
0/1 UP 32027M 18048M 292060k 67610607
0/2 DOWN 0 0 0 0
LO/1 UP 831255M 1218G 6624M 13125M
LA/1 DOWN LA2 0 0 0 0

© 2014 Citrix | Confidential – Do Not Distribute


stat interface [interfacename]
> stat interface 1/1 LACP Statistics
Rate (/s) Total
Interface [1/1]: LACPDUs received 0 0
LACPDUs transmitted 0 0
Interface State UP Error Statistics
Link uptime 00:40:21 Rate (/s) Total
Link downtime 00:00:00 Error packets received (hw) 0 0
Error packets transmitted (hw)0 0
Throughput Statistics Inbound packets discarded (hw) 0 0
Rate (/s) Total Outbound packets discarded (hw) 0 0
Bytes received 42393 54497294 Packets dropped in Rx (sw) 539 599904
Packets dropped in Tx (sw) 0 0
Bytes transmitted2584 20222135
NIC hangs -- 0
Packets received 629 710246 Status stalls -- 0
Packets transmitted 47 69066 Transmit stalls -- 0
Packet Statistics Receive stalls -- 0
Rate (/s) Total Error-disables -- 0
Multicast packets 18 41219 Duplex mismatches -- 0
NetScaler packets 85 98954 Link re-initializations -- 0
MAC moves registered 0 0
Times NIC became muted -- 0

© 2014 Citrix | Confidential – Do Not Distribute


stat dns

> stat dns Non-authoritative entries 0


Authoritative entries 98
DNS Statistics
Error Statistics
Runtime Statistics Nonexistent domain 102359
Dns queries 106983 Response class unsupported 0
Multi queries 0 Invalid query format 0
Dns responses 0 Stray answers 0
Server responses 102334 Incorrect RD length 0
Total Record updates 0 Requests refused 0
Auth answers 0 Response type unsupported 0
Server queries 102347 Query class unsupported 0
Cache flush called 0 Invalid response format 0
Cache entries flushed 0 No answer responses 102334
Multi queries disabled 0
Configuration Statistics Other errors 0

© 2014 Citrix | Confidential – Do Not Distribute


stat http

> stat http

HTTP Statistics - Summary


Rate (/s) Total
Total requests 1 6251
Total responses 1 5885
Request bytes received 27 293191
Response bytes received 286 1744835

© 2014 Citrix | Confidential – Do Not Distribute


stat lb vserver [name]

> stat lb vserver LB_RGB


Vserver hits 0 0
Virtual Server Summary
Bound
RequestsService(s) Summary 0 0
vsvrIP port Protocol State Health actSvcs
Responses
LB_RGB 192.168.47.3 80IP port
HTTP Type
0DOWN 0 State
0 0 Hits
Hits/s
Request bytes
inactSvcs 0 65
svc_andrews
Response
LB_RGB bytes 1 192.168.33.130 80 0 HTTP DOWN
188 0
0/sPackets rcvd
Total 0 5
VirtualPackets
Total Server Statistics
sent 0 4
Rate (/s) Total
Current client connections -- 0
Vserver hits Req Req/s Rsp
0 Rsp/s Throughp
0 ClntConn SurgeQ
Current Client Est connections -- 0
Requests
svc_andrews
Current server 0
connections 0/s 00 -- 0/s
0
0 0 0 0
Responses 0 0
Requests in surge queue
Request bytes 0 -- 65 0
Requests in vserver's
Response bytes surgeQReuseP
SvrConn MaxConn --
0 ActvTran SvrTTFB
188 0 Load
Requests in service's
Total Packets rcvd surgeQs -- 0
00 5
svc_andrews 0 0 0 0 0
Total Packets
Spill sent
Over Threshold 0 -- 4 0

© 2014 Citrix | Confidential – Do Not Distribute


stat service [servicename]

> stat service svc_blue


Requests
Service Summary 0 865
IP port Type State
Responses 0 855
svc_blue 192.168.196.62 80 HTTP UP
Request bytes
Service Stats: 22 67683
Rate (/s) Total
Response bytes 767 2304810
Requests 0 865
Current
Responses client connections 0 855 -- 0
Request bytes 22 67683
Requests in surge queue -- 0
Response bytes 767 2304810
Current server
Current client connections
connections -- 0 -- 9
Requests in surge queue -- 0
Current Server Est connections -- 0
Current server connections -- 9
Connections
Current Server Estin reuse pool
connections -- 0 -- 0
Connections server
Maximum in reuse pool
connections -- 0 -- 0
Maximum server connections -- 0
Average server
Average server TTFB TTFB -- 0 -- 0
Current
Current loadload
on the on the service
service -- 0 -- 0

© 2014 Citrix | Confidential – Do Not Distribute


NetScaler Disk Partitions
NetScaler File System

# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0c 286M 245M 35M 88% /
devfs 1.0K 1.0K 0B 100% /dev
procfs 4.0K 4.0K 0B 100% /proc
/dev/ad0s1a 1.4G 965M 368M 72% /flash
/dev/ad0s1e 14G 3.1G 9.5G 24% /var

© 2014 Citrix | Confidential – Do Not Distribute


NetScaler File System examples

/var/log/ns.log
/var/nslog/newnslog/newnslog.ppe.x Logs
/var/nsinstall/build-10.1-124.13_nc.tgz
/flash/ns-10.1-124.13.gz Firmware
/var/core/NSPPE-00-353.gz
/var/crash/vmcore.0
Crash Files
/nsconfig/ns.conf NetScaler Configuration
/nsconfig/ssl/ns-root.cert SSL Certificates
/nsconfig/monitors/nssmtp.pl Monitor Scripts
/nsconfig/license/FID__b0d70c6_13b16ab7034_573f.lic License Files
/netscaler/nsconmsg NetScaler
NetScaler Binaries
Binaries
/var/nstrace/nstrace1.cap Packet
Packet Trace
Trace Files
Files

© 2014 Citrix | Confidential – Do Not Distribute


dmesg

root@ns# dmesg
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-NETSCALER-10.1 #0: Thu Feb 20 18:54:22 PST 2014
root@sjcpbldbsd6301.eng.citrite.net:/usr/obj/amd64/usr/home/build/rs_101_124_8/usr.src/sys/NS64
Preloaded elf kernel "/ns-10.1-124.13" at 0xffffffff930b3000.
Calibrating clock(s) ... i8254 clock: 1189606 Hz
CLK_USE_I8254_CALIBRATION not specified - using default frequency
Timecounter "i8254" frequency 1193182 Hz quality 0
Calibrating TSC clock ... TSC clock: 3325066248 Hz
CPU: Intel(R) Core(TM)2 Duo CPU E8600 @ 3.33GHz (3325.07-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x1067a Stepping = 10
Features=0x789fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x81282201<SSE3,SSSE3,CX16,<b19>,<b21>,<b24>,<b31>>
AMD Features=0x20000800<SYSCALL,LM>
AMD Features2=0x1<LAHF>
real memory = 2143289344 (2044 MB)

© 2014 Citrix | Confidential – Do Not Distribute


/var/log

# ls
auth.log httperror.log ns.log
auth.log.0.gz httperror.log.0 ns.log.0
callhome.log lastlog nscollect.log
callhomedebug.log license.log nscollect_cl.log
cron lpd-errs nsvpn.log
cron.0.gz maillog nsvpnd.log
ctxslsboc.log maillog.0.gz ntpd.log
db messages security
httpaccess.log messages.0 snmpd.log
httpaccess.log.0.gz nitro.log wicmd.log
httpd.scoreboard nitro.log.0.gz

© 2014 Citrix | Confidential – Do Not Distribute


/var/nslog

# cd /var/nslog/
# ls
asl lspci_tv.last nsagg.conf
aslearn.log lspci_tv.prev nsagg.log
aslearn_old_db.tar.gz lspci_vvvxxx.boot nslog.nextfile
conmsg.log lspci_vvvxxx.last nsumond
dmesg.boot lspci_vvvxxx.prev nsumond.log
dmesg.last newnslog snmpdebug.log
dmesg.prev newnslog.0.gz
lspci_tv.boot ns.log
# cd newnslog
# ls
newnslog.ppe.0

© 2014 Citrix | Confidential – Do Not Distribute


Troubleshooting Techniques

View events
View console messages
Use View statistics

cases Debug system counters


Debug load balancing issues
Debug CPU/Memory utilization

© 2014 Citrix | Confidential – Do Not Distribute


Troubleshooting Techniques: nsconmsg
nsconmsg Common Syntax

Make sure to use –K,


# cd /var/nslog
# nsconmsg –K newnslog -d event

NOT -k
# nsconmsg –K newnslog -d consmsg
# nsconmsg –K newnslog -d oldconmsg
# nsconmsg –K newnslog -s ConLb=2 –d oldconmsg
# nsconmsg –K newnslog -s ConDebug=1 –d oldconmsg

© 2014 Citrix | Confidential – Do Not Distribute


nsconmsg examples

# nsconmsg -d current -g cpu_use Live CPU related stats


# nsconmsg -K <newnslog-filename> -d event Archived events
# nsconmsg -d current -g ha_cur_master_state HA Failover cause
# nsconmsg -s ConLb=2 -d oldconmsg LB stats
# nsconmsg -s ConCSW=2 -d oldconmsg CS related counters
# nsconmsg -d current -g pol_hits Real-time policy hits
# nsconmsg -s ConSSL=2 -d oldconmsg SSL related counters
# nsconmsg -s ConCMP=2 -d oldconmsg Compression related counters

© 2014 Citrix | Confidential – Do Not Distribute


Policy Hits

# cd /var/nslog/newnslog
# nsconmsg -K newnslog.ppe.0 -d current -g pol_hits
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27

reltime:mili second between two records Fri Oct 25 09:45:52 2013


Index rtime totalcount-val delta rate/sec symbol-name&device-no
0 3038060 2 2 0 pol_hits Policy(CTX-LDAP)
1 3500159 15 15 2 pol_hits
Policy(__ESNS_PREBODY_POLICY)
2 0 15 15 2 pol_hits
Policy(__ESNS_POSTBODY_POLICY)

© 2014 Citrix | Confidential – Do Not Distribute


nsconmsg –K newnslog -d event

# nsconmsg -K newnslog -d event | more


Displaying event information
1906
NetScaler3910 PPE-0 'interface(1/1)'
V20 Performance Data has been disabled Wed Aug 14 12:27:16 2013
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013
rtime: Relative time between two records in milliseconds
1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)'
seqno rtime event-message event-time DOWN Wed
Aug
1906 14 12:27:25
3910 2013
PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013
1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013
1913 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.51:53(SVC_CTXANG_DNS2)' DOWN Wed
1908 7 PPE-0 MonServiceBinding_192.168.47.16:4739_(ping-default)(service_192.168.47.16_33554): DOWN; Last
Aug 14 12:27:25
response: 2013
Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1909 0 PPE-0 MonServiceBinding_192.168.224.5:53_(ping-default)(SVC_CTX_DNS1): DOWN; Last response: Failure - Probe
timed out. Wed Aug 14 12:27:25 2013
1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping-default)(SVC_CTXANG_DNS2):
1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping-default)(SVC_CTXANG_DNS2): DOWN; Last response: Failure -
DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
Probe timed out. Wed Aug 14 12:27:25 2013
1911
1911 0 PPE-0
0 PPE-0 MonServiceBinding_192.168.204.50:53_(ping-default)(SVC_CTXANG_DNS1):
MonServiceBinding_192.168.204.50:53_(ping-default)(SVC_CTXANG_DNS1): DOWN; Last response: Failure -
Probe timed out. Wed Aug 14 12:27:25 2013
DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)' DOWN Wed Aug 14 12:27:25 2013
1913 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.51:53(SVC_CTXANG_DNS2)' DOWN Wed Aug 14 12:27:25 2013

© 2014 Citrix | Confidential – Do Not Distribute


nsconmsg –K newnslog –d consmsg

# nsconmsg -K newnslog -d consmsg


Displaying console message information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27

current time is Wed Aug 14 12:53:43 2013


LSM 1/1: Mon->Up on Link_Up
platform: NetScaler Virtual Appliance 450000 (9), manufactured at 2/17/2009
platform: serial HE2H91SCZ6
platform: unknown platform - using defaults!
LSM LO/1: Mon->Up on Link_Up
NetScaler: 420Mb of memory allocated
cell_pool: 0, (2 MB, 2 MB)
cell_pool: 0, map_addr 0xd2200000

© 2014 Citrix | Confidential – Do Not Distribute


nsconmsg –K newnslog –d oldconmsg

# nsconmsg -K newnslog.ppe.0 -d oldconmsg


Displaying debug performance information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27

current time is Wed Aug 14 10:17:10 2013


CPU:2.4% MEM:199419576 UP:00.00:00:14 since:Wed Aug 14 10:16:57 2013

current time is Wed Aug 14 10:17:17 2013


CPU:2.4% MEM:199462204 UP:00.00:00:21 since:Wed Aug 14 10:16:57 2013

current time is Wed Aug 14 10:17:24 2013


CPU:2.4% MEM:199481988 UP:00.00:00:28 since:Wed Aug 14 10:16:57 2013

© 2014 Citrix | Confidential – Do Not Distribute


nsconmsg –K newnslog –s ConLb=2 –d
oldconmsg
# nsconmsg -K /var/nslog/newnslog -s ConLb=2 -d oldconmsg
Displaying debug performance information
NetScaler V20 Performance Data
VIP(127.0.0.2:53:DOWN:WEIGHTEDRR):
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27 Hits(0, 0/sec)
Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024
current time is Wed Aug 14 10:17:10 2013

Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE)


-------------------------------------------------------
NATSession : Free(6552)A(6553)InUse(1)
override(0) newlyUP(0)
NATSession: Cur(Tcp[0] Udp[1] Icmp[0] Other[0])
NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])

Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0


Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0
SSF: Conn (Srvr 0 Clnt 0) U:0
OnVserver:
CM: Conn (Srvr 0 Clnt 0 OnServices:
0) Sessions PCB 0 NATPCB 0 0)
Z(SIP[0], C[0], SSL[0] Server[0] SIPDIP[0] DIP[0] SO[0])
slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex:
Mon: Probes: 0, Failed: 0
VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024
0 Borrowed: 0 actSvc(0)
Pkt(0/sec, 0 bytes) TotActiveConn: 0]newlyUP(0)
DefPol(NONE) override(0) Available: 0-
Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0 OnVserver: 0 OnServices: 0)
slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex: 0 Borrowed: 0 TotActiveConn: 0] Available: 0

© 2014 Citrix | Confidential – Do Not Distribute


IP Conflict

# cat /var/log/ns.log
Aug 14 12:39:11 <local0.info> 192.168.47.1
ipConflict (ipConflictAddr = 192.168.47.1,
sysIpAddress = 192.168.47.1)
Aug 14 12:39:42 <local0.info> 192.168.47.1 last
message repeated 169 times

© 2014 Citrix | Confidential – Do Not Distribute


Troubleshooting Techniques
Common NetScaler issues/resolutions

HDD Issues Logging fails


/var missing
Flash Issues Config fails to save
Config saves partially
Sync fails
Device fails to boot
Memory starvation Dropped sessions
CPU starvation All services failing
All VIPS down
Degraded performance
© 2014 Citrix | Confidential – Do Not Distribute
NetScaler Data Collection

NetScaler software version including build – (from the ‘show version’ NetScaler
Command Line Interface (NSCLI) command) – example: 10.1.124.13
Production setup or new installation
Whether an application/service that was working is now broken or whether the
user wants to configure an application/service
Network topology information
What changes were performed on the NetScaler appliance prior to the issue
Any change(s) on the connected Switches, upstream Router, or backend
server prior to experiencing the issue
ns.conf after saving the configuration - (from the save configuration NSCLI
command)

© 2014 Citrix | Confidential – Do Not Distribute


show techsupport I
> show techsupport

showtechsupport data collector tool - $Revision: #1 $!


NetScaler version 10.1
Creating /var/tmp/support ....
The NS IP of this box is 192.168.47.1
Current HA state: Primary (or this is not part of HA pair!)
All the data will be collected under
/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12
Copying selected configuration files from nsconfig ....
Copying WebInterface configuration files (if WI is installed)...
... WI is not installed on this system. Nothing to copy.
Running shell commands ....
Running CLI show commands ....
Running CLI stat commands ....
Running vtysh commands ....
Determining newnslog files to archive....

© 2014 Citrix | Confidential – Do Not Distribute


show techsupport II
... copied 6 files from this directory.
Copying core files from /var/core ...(last 5 files created within the last week)
NSPPE core (2/NSPPE-00-1077.gz) file present! Skipping this file because of size
restrictions..
... copied 5 files from this directory.
Copying core files from /var/crash ...(last 5 files created within the last week)
... Nothing to copy...No files created within the last one week
Copying messages,ns.log,dmesg and other log files ....
Copying imported files and mapping files ...
Copying GSLB location database files ....
Archiving all the data into
"/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12.tar.gz"....Done.
Created a symbolic link for the archive with /var/tmp/support/support.tgz
/var/tmp/support/support.tgz ---- points to --->
/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12.tar.gz

If this node is part of HA pair, please run it on the other node also!!
© 2014 Citrix | Confidential – Do Not Distribute
techsupport archive structure

/etc
/flash
/nsconfig
/shell
/var
/cfsynclog
/clusterd
/core
/crash
/download
/log
/netscaler
/nslog
/nsproflog
/nssynclog

© 2014 Citrix | Confidential – Do Not Distribute


nstrace & nstcpdump

nstrace nstcpdump

Common syntax: Common syntax:


Nstrace.sh –sz 0 Nstcpdump.sh –X tcp port 80
Nstrace.sh –sz 0 –filter “SOURCEIP Nstcpdump.sh –w testcapture.cap
= 10.198.4.10” –link enabled –X src host 10.198.4.10 tcp port 80
Filter qualifiers and operators: Filter qualifiers and operators:
SOURCEIP, SOURCEPORT, DESTIP, tcpdump standard
DESTPORT, SVCNAME, VSVRNAME,
STATE
==, eq, !=, neq, >, gt, <, lt, >=,
ge, <=, le, BETWEEN
Compound filters using || and &&

© 2014 Citrix | Confidential – Do Not Distribute


Packet Tracing
Differences between nstrace.sh & nstcpdump.sh

Nstcpdump.sh Nstrace.sh
Useful if traces are to be viewed on Useful for offline collection
standard output nstrace.sh –sz 0
nstcpdump.sh –w <filename> option Saves traces in /var/nstrace in cap format
helps writing output to file

can be used with expressions so that you get useful for collection of traces in separate log
to see filtered traffic files based on NICs

nstcpdump.sh host <IP> nstrace.sh –tcpdump1 –nic 1


nstcpdump.sh port 21

© 2014 Citrix | Confidential – Do Not Distribute


nstrace.sh

# nstrace.sh --?
-h - prints this message - exclusive option
nstrace - utility to start NetScaler packets trace
-nf
usage: - number
nstrace.sh of <number_of_files]
[-h] [-nf files to be generated
[-time <time>] in
[-m cycle
<mode>] (def. 24)
[-nic <boolean>]
-h - prints this message - exclusive option
-time - seconds per file (def. 3600) (could be an expression)
-nf - number of files to be generated in cycle (def. 24)
-sz -
-time size of the
- seconds captured
per file data
(def. 3600) (bytes
(could from 60 to 1514)
be an expression)
-sz - size of the captured data (bytes from 60 to 1514)
-tcpdump
-m - 0=nstrace-format (default)
- Capturing mode: sum of or 1=tcpdump-format
the values (def. 18):
-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format
-nic -nic - use separate trace files for each interface (only works if -tcpdump
- use separate trace files for each interface
option is set) (only works if -tcpdump option is set)
-name -name
- - name
name ofofthe
the trace
trace file
file
-filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of
-filter - Filter
following format: expression
<expression> [<relop>for nstrace. The maximum length of filter expression
<expression>]
is 255- and
-link it canconnection's
Log filtered of following
peer's format: <expression>
(linked connection's) [<relop>
traffic. Works only <expression>]
with -filter option
-id - ID for the trace file name for uniqueness. Should be used only with -name option
-link - Log filtered connection's peer's (linked connection's) traffic. Works
-stop - can be used to disable tracing (when 'nstrace.sh' is run in the background)
only
#
with -filter option

© 2014 Citrix | Confidential – Do Not Distribute


Packet Tracing

nstrace.sh nstcpdump.sh
Proprietary capture format TCPdump PCAP capture format
Native format captures more Useful for live capture from CLI
information
Files are stored in /var/nstrace Option to write to a file
Needs custom dissector in Most TCPdump options supported
Wireshark (1.6+)
Works in standard Wireshark

© 2014 Citrix | Confidential – Do Not Distribute


Trace analysis
nstrace.sh

By Default if nstrace.sh is executed from shell prompt

Trace files are stored in NetScaler proprietary (.cap) format

The trace capture runs for 1 hr. (3600 sec) if not interrupted

Files are cyclically numbered from 1-24. Trace mode is 6 (that is the TXB
and RX packets are captured)

The size of the captured data for each packet is 164


© 2014 Citrix | Confidential – Do Not Distribute
Trace Analysis
nstrace syntax examples

Command Purpose
# nstrace.sh Stores the traces in default (proprietary) format
# nstrace.sh Begins to save the traces in the TCPDUMP format for a default 3600
-tcpdump 1 seconds
# nstrace.sh Logs the traces (in TCPDUMP format) into separate log files based on
-tcpdump 1 -nic 1 the NIC IDs
# nstrace.sh –nf No of files to be generated in cycle by default is 24
<value>
# nstrace.sh –time Seconds per file by default 3600 seconds
<value>
# nstrace.sh -sz 0 Size of the captured data( by default it is 164), -sz 0 is the entire
packet length
# nstrace.sh –m Capturing mode: sum of the values (def. 6):
1-Transmitted packets (TX)
2 - Packets buffered for transmission (TXB)
© 2014 Citrix | Confidential – Do Not Distribute
4 - Received packets (RX)
Trace Analysis
nstcpdump.sh example syntax

Command Purpose
# nstcpdump.sh Possible types are host, net and port .If there is no type
<type> qualifier host assumed
# nstcpdump.sh Qualifiers specify a particular transfer direction to and/or
<dir> from id. Possible directions are src, dst, src or dst and
src and dst. If there is no dir qualifier, src or dst is
assumed. src foo', `dst net 128.3', `src or dst port ftp-
data'
# nstcpdump.sh ether, fddi, ip, arp, rarp, decnet, tcp and udp.
<proto>
# nstcpdump.sh –c Exit after receiving ‘value’ number of packets
<value>
# nstcpdump.sh –F –I Not supported and not to give them as options to the
–r script
© 2014 Citrix | Confidential – Do Not Distribute
# nstcpdump.sh –w Write the raw packets to file rather than parsing and
nstcpdump.sh

# nstcpdump.sh --?
Setting 1000 pages (8000 KB) of trace buffers ... Done.
Enabling all nic trace mode=6 ... Done.
Changing trace packet length from 0 to 0 ... Done.
tcpdump version 3.9.4
libpcap version 0.9.4
Usage: tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
[ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
[ -W filecount ] [ -y datalinktype ] [ -Z user ]
[ expression ]
Saving current trace data in file 'pipe' ... in TCPDUMP format
Disabling all nic trace ... Done.

© 2014 Citrix | Confidential – Do Not Distribute


Trace analysis

# nstcpdump.sh host 10.102.12.250

# nstcpdump.sh port 21

# nstcpdump.sh src host 10.102.12.250 and dst host 10.102.12.204

# nstcpdump.sh ‘(host 10.102.12.250 and host 10.102.12.204) or


(host 10.102.12.204 and host 10.102.12.31)’

# nstcpdump.sh host storefront.nstipster.lab

© 2014 Citrix | Confidential – Do Not Distribute


Core Dumps/Crashes
Core dump directories

# ls -la
total 8
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .
drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..
drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1
-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds
# ls –la 1/
total 10292
drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..
-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz
-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz

© 2014 Citrix | Confidential – Do Not Distribute


Packet Engine (nsppe) Process Crashing

Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask
is 0x0 0x0 0x0 0x0
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00
(1077) unexpectedly died due to receiving signal
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may
be a delay restarting process while collecting core dump on NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc
NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to
policy setting (0x29ac)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler
restart may be delayed if collecting core dump for NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss
declaring system failure: NSPPE-00 (1077) exited
Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting
watch on NSPPE-00 (1077) for ()

© 2014 Citrix | Confidential – Do Not Distribute


/var/core

# cd /var/core/
# ls -la
total 14
drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .
drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..
drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1
-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds
root@vpx1# ls 1/
NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz
aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz
imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz
monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz
nsaaad-1131.gz nsconfigd-1156.gz
nsaggregatord-1107.gz nsfsyncd-1110.gz

© 2014 Citrix | Confidential – Do Not Distribute


show techsupport

> show techsupport


----SNIP----
Copying core files from /var/core ...(last 5 files created within the
last week)
NSPPE core (1/NSPPE-00-1077.gz) file present! Skipping this
file because of size restrictions..
... copied 5 files from this directory.
Copying core files from /var/crash ...(last 5 files created within the
last week)
... Nothing to copy...No files created within the last one
week
----SNIP—---

© 2014 Citrix | Confidential – Do Not Distribute


Backup conf files

# cd /nsconfig/
# ls -lath ns*
-rw-r--r-- 1 root wheel 16K Oct 30 14:01 ns.conf
-rw------- 1 root wheel 16K Oct 30 14:01 ns.conf.0
-rw------- 1 root wheel 12K Oct 25 09:34 ns.conf.1
-rw------- 1 root wheel 12K Oct 25 08:09 ns.conf.NS10.1-
124.13
-rw------- 1 root wheel 12K Oct 25 07:52 ns.conf.2
-rw------- 1 root wheel 12K Oct 24 15:17 ns.conf.3
-rw------- 1 root wheel 13K Oct 24 08:56 ns.conf.4

© 2014 Citrix | Confidential – Do Not Distribute


Useful Links

CTX1093 Data Collection Procedure to Troubleshoot NetScaler


04 Related Issues
http://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-n
et-scaler-for-the-first-time
http://support.citrix.com/search/basic?searchQuery=counters&refin
ement=Content+Type,Technotes&refinement=Product+Family,NetScaler
CTX1149 How to Troubleshoot Authentication with Aaad.debug
99
https://taas.citrix.com/

© 2014 Citrix | Confidential – Do Not Distribute


Maximize your knowledge.
Continue your journey with Citrix Education.
Recommended next step for hands-on technical
training:

CNS-205 Citrix NetScaler 10 Essentials and Networking


Identify the capabilities and functionality of the NetScaler 
Explain basic NetScaler network architecture
Obtain, install, and manage NetScaler licenses
Explain how SSL is used to secure the NetScaler
Implement NetScaler TriScale Technology, including Clustering

Visit bit.ly/NSCOURSE to save 10% now through April 30.*


*Not valid with any other promotions, packages or discounts. Applies only to new purchases. Regional limitations may apply.
© 2014 Citrix | Confidential – Do Not Distribute
Educate | Guide | Support | Succeed

Citrix Services How we can help


make sure
About
you succeed with Citrix Education – The fastest, most
efficient way to
your
Citrix Services
virtualization
get your team the virtualization skills they
need. Online,
on-site or in class.
programs. citrix.com/training

Citrix Consulting – Intensive engagements


for
complex, critical or just plain massive
projects.
citrix.com/consulting

Citrix Support – Always-on support


© 2014 Citrix | Confidential – Do Not Distribute
Secrets of the Citrix Support Ninjas

• 40 insider troubleshooting tips


• Covering XenDesktop, XenServer, XenApp and NetScaler
• Citrix Support top engineers
• FREE eBook
• Citrix Auto Support
• Now available!
© 2014 Citrix | Confidential – Do Not Distribute
Premier Support Calculator

Check it out

© 2014 Citrix | Confidential – Do Not Distribute


Work better. Live better. Use
NetScaler

Anda mungkin juga menyukai