SESSION SEC-3020
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 1
Agenda
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 2
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 3
Private Public
Adaptive
Security
Algorithm
7
Drop Drop
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 5
CSCdv33495
6.3(2) 6.3(3)
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 6
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 7
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 8
• Archival methods
• Primary mechanism to record traffic
TO and THROUGH the PIX
• One of the best troubleshooting tools
Debugging Methods
Archival Methods
Console
SSH Client
Buffered
SNMP Server
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 9
0 Emergencies 0
1 Alerts 41 (41)
2 Critical 21 (62)
3 Errors 74 (136)
4 Warnings 56 (192)
5 Notifications 21 (213)
6 Informational 95 (308)
7 Debugging 15 (323)
Define internal
syslog server
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 11
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 12
SOLUTION:
• Lower syslog message 111009 to level 3 (Error)
pix(config)# logging message 111009 level 3
- or –
pix(config)# logging message 111009 level error
• Now our syslog looks as follows:
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 13
Notes on ICMP
2 DMZ
Ping
1 Ping Ping
Internet
Inside Outside
3
Bob
Private
Internet
Network
http://www.cisco.com
undebug all
pix(config)# un all
pix(config)# show debug
pix(config)#
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 16
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 17
Packet Capture
Inside Outside
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 19
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 21
Displays Current
and Past Info
Show Related to the PIX Show
Perfmon Blocks
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 23
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 25
Xlate Flags
FLAG DESCRIPTION
s Static Translation Slot
d Dump Translation Slot on Next Clearing Cycle
r Port Map Translation
n No Randomization of TCP Sequence Number
o Outside Address Translation
i Inside Address Translation
D DNS A RR Rewrite
I Identity Translation from NAT 0
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 26
Connection Flags
SYN+ACK
SYN
ACK
Data
5
1
3 42 U A
saA
UI
UIO
Inside Outside
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved.
Server Client 27
Connection Flags
FIN+ACK
ACK
FIN
1
3 2 UfUfFR
UfFRr
Inside Outside
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved.
Server Client 28
An Example
pixfirewall# show cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1%
A Note
The Percentage Usage Prints as NA (Not Applicable) if the Usage is Unavailable
For the Specified Time Interval; this Can Happen if the User Asks For CPU
Usage Before the 5-second, 1-minute, or 5-minutes
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 29
Show Traffic
• The show blocks command, along with the show cpu usage
command, are useful in determining whether the PIX is being
overloaded
• The blocks are internal storage locations, similar to queues on
a router; a packet is stored in a block until the PIX can process
it and place it on the outbound interface xmit queue
An Example
pixfirewall# show blocks What are these blocks used for?
256 = Stateful Failover, Syslog, TCP module
SIZE MAX LOW CNT 1550 = Ethernet Packets, buffering url filtered packets
4 1600 1597 1600 16384 = Only used for the Gig Ethernet cards
80 400 399 400
256 500 495 499
1550 1444 1170 1188
16384 2048 1532 1538
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 31
Show Local-Host
Conn(s):
UDP out 209.165.201.1:8943 in 10.1.2.74:63556 idle 0:01:31 flags -
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 32
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 33
Output Filters Have Been Added to PIX 6.3, Similar to the Ones
in IOS; to Use them, at the End of Show <command>, Use the Pipe
Character ‘|’ Followed By:
begin|include|exclude|grep [-v] <regular_exp>
to Filter the Show Output
Begin—Start Displaying the Output Beginning at the First
Match of the RegEx, and Continue to Display the
Remaining Output
include – Display any line that matches the RegEx
exclude – Display any line that does not match the RegEx
grep – same as include
grep –v – same as exclude
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 34
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 35
Output Interpreter
GREAT TOOL TO CATCH COMMON CONFIGURATION ERRORS
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 36
Snip of Output
Example
Of Messages
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 37
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 38
Device Information
Interface Statistics
Current System
Utilization: CPU,
Memory, Traffic
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 39
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 40
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 41
Common Issues
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 42
Problem: Troubleshooting:
• Accessing the Internet • P ermissions
• T ranslation
• R outing
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 43
Permissions (Access-Lists)
Note
If you Have an Access-list Applied on the Inside Interface, Check to Make Sure
Traffic is Permitted Outbound; Remember, there is an Implicit Deny at the End
of an Access-list
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 44
1 Verified Permissions
Verify Translation Commands
global (outside) 1 63.1.1.10–63.1.1.50 netmask 255.255.255.0
nat (inside) 1 10.10.10.0 255.255.255.0
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 45
Routing
INSIDE OUTSIDE
.1 .2
.2 .3 .1 .1 .2
Internet
192.168.1.x 10.10.10.x 63.1.1.x
1 Verified Permissions
2 Verified Translation
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 46
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 47
http://www.xyz.com
Problem: Troubleshooting:
• Internal web server not • P ermissions
accessible to users on the • T ranslation
Internet
• R outing
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 48
http://www.xyz.com
Traffic has to be Explicitly Allowed into the PIX from a Lower Security to a
Higher Security
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 49
Translation
Web Server 10.10.10.x INSIDE OUTSIDE
.1 .1 .2
Internet
.1 63.1.1.x
http://www.xyz.com
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 50
http://www.xyz.com
• Check to make sure the PIX has the correct default gateway
pixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 51
Common Issues
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 52
DMZ (30)
DMZ 1 (40)
Common Issues
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 54
INSIDE OUTSIDE
.2 .1 .2 .1 .1
Internet
172.16.171.x 63.1.1.x
10.10.10.x
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 55
Common Issues
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 56
• Active/Standby vs.
Primary/Secondary
• Serial Failover/LAN Failover
• Stateful Failover
• A Failover ONLY occurs
Stateful
when either PIX determines
the Standby PIX is healthier Serial
than the Active PIX Lan FO
Active Standby
• Both PIXes swap MAC and Unit Unit
IP Addresses when a
failover occurs
• Level 1 syslogs will give
reason of failover
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 57
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
This host: Primary - Standby (Failed)
Active time: 7140 (sec)
Interface 0 (192.168.1.1): Normal
Interface 1 (172.16.171.54): Normal
Other host: Secondary - Active
Active time: 30 (sec)
Interface 0 (192.168.1.3): Normal
Interface 1 (172.16.171.55): Normal
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 58
Failover
Ping Test Cable Failure
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 59
1 No Failover Hello Seen on Serial Cable for 30 + Seconds; This Ensures that
Failover is Running Properly on the Other PIX
2 An Interface Did not Pass One of the 4 Failover Tests
(Link up, Interface Traffic, ARP Test, Broadcast Ping)
3 No Proper ACK for 15+ Seconds After a Command has Been Sent on the
Serial Cable
Example of Syslogs when Primary Was Active, and It Lost One of Its Interfaces
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 61
http://www.cisco.com/warp/public/110/34.shtml
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 62
Do you want to remove the commands listed above from the configuration? [yn] y
Passwords and aaa commands have been erased.
Rebooting..
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 63
Common Issues
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 64
Number of msec
The Name of the
this Process Has
Process
been on the CPU
For more information on the output of the “show processes” command, see:
http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 65
The Following Output Was a Diff of the Processes Taken 1 Minute Apart
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 66
. . .
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 68
SOLUTION:
• Bring back up syslog service on server
• Take server offline
• Configure PIX to not log IDS ICMP Unreachable
messages
ip audit signature 2001 disable
- or -
no logging message 400011
pixfirewall# show run | grep signature
ip audit signature 2001 disable
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 70
SUMMARY:
• Examine the DIFF of two show processes taken
over a one minute interval
• Find the process taking up the highest amount of
CPU (excluding the polling processes)
• Take actions to lower that processes’ CPU time
• Re-examine the CPU output, and repeat as
necessary
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 71
Common Issues
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 72
PROBLEM:
• Users are unable to access the Internet
• No new connections are working;
• All old (long lived) connections continue to work
Out of Memory
Varied Source
IPs
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 74
Out of Memory
outside:
received (in 25.000 secs):
1475 packets 469050 bytes
59 pkts/sec 18762 bytes/sec
transmitted (in 25.000 secs):
167619 packets 9654480 bytes
Traffic Flow
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 77
Out of Memory
Out of Memory
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 80
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 81
Out of Memory
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 82
• At a minimum include:
Detailed problem description
Output from “show tech”
• Optionally include:
Syslogs captured during time of problem
Sniffer traces from both interfaces using the “capture”
command (capturing only the relevant packets, and saved
in pcap format)
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 83
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 85
Feature Releases
Port Features
Bug Fixes
Feature Releases
FWSM 1.1(1) 2.2(1)
1.1(2) 1.1(3)
New Features Maintenance New Features
Releases
OSPF, Virtual FW,
crashinfo, etc. Transparent FW, etc.
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 86
Feature Releases
Port Features
Cloned to FWSM Bug Fixes
CSCeb16339
1.1(2) 1.1(3)
Maintenance
Releases CSCeb76295
Bug Fixes Committed
to Multiple Trains
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 87
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 88
Slot 5
OUTSIDE INSIDE
Vlan 30 Vlan 60
10.30.1.0/24 10.30.1.1 10.60.1.1 10.60.1.0/24
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 89
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 90
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 91
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 92
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 93
What Is CBAC
INSIDE OUTSIDE
ip inspect name foo tcp access-list 101 deny ip any any
interface ethernet0 interface ethernet1
ip inspect foo in ip access-group 101 in
SYN
A:a B:b
SYN + ACK
A:a B:b
ACK
A:a B:b
RST
A:a B:b
RST
A:a B:b
e0 e1
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 95
Established Sessions
Session 814063CC (192.168.1.116:32955)=>(192.168.101.115:23) tcp SIS_OPEN
Created 00:00:10, Last heard 00:00:06
Bytes sent (initiator:responder) [140:298]
In SID 192.168.101.115[23:23]=>192.168.1.117[32955:32955] on ACL 101 (15 matches)
Out SID 192.168.101.115[23:23]=>192.168.1.116[32955:32955] on ACL 102
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 96
State Tables:
• Maintains session state information
• Updated when a packet is inspected at the firewall's interface
• State table updates will permit return traffic through the firewall for packets
traveling within a permissible session
• The state table is updated to remove the temporary opening in the access
list when the session terminates
Session Maintenance
• Use of timeouts and threshold values to manage session state information
• Timeouts used to prevent DoS attacks by dropping half-open connections
and freeing up network resources
• Threshold values used to prevent DoS attacks by controlling the number of
half-open sessions
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 97
Thresholds
• Set for the total number of half-open TCP and UDP sessions
• Set for the total number of half-open session based on time
• Set for the total number of half-open TCP-only sessions per host
Configure Thresholds
• ip inspect max-incomplete {high|low}
This will modify the number of allowed max-incomplete sessions from the defaults of
500/400
• ip inspect one-minute {high|low}
This will modify the number of unestablish sessions allowed in one minute from the
defaults of 500/400
• ip inspect tcp max-incomplete host <value>
This will modify the number of incomplete sessions one particular host is allowed to have
at any time from the default of 50
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 98
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 99
Show Commands
show ip access-list
show ip inspect name inspection-name
show ip inspect config
show ip inspect config
show ip inspect interfaces
show ip inspect session [detail]
show ip inspect all
show ip inspect stat
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft
/120t/120t5/iosfw2/iosfw2_2.htm#12583
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 100
Debugging CBAC
Audit Trails
ip inspect audit-trail
Generic Debug
debug ip inspect object-creation
debug ip inspect object-deletion
debug ip inspect events
debug ip inspect timers
debug ip inspect detail
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 102
Feb 14 12:41:17 10.31.1.52 56: 3d05h: CBAC* sis 258488 pak 16D0DC TCP P ack
3195751223 seq 3659219376(2) (10.31.1.5:11109) => (12.34.56.79:23)
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 103
Configuration Issues
e0 e1
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 104
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 105
PIX Firewall
http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/index.htmhttp://www.cisco.com/cgi
-bin/tablebuild.pl/pix
http://www.cisco.com/pcgi-bin/Support/browse/psp_view.pl?p=Hardware:PIX&s=Software_Configuration
http://www.cisco.com/pcgi-
bin/Support/browse/psp_view.pl?p=Hardware:PIX&s=Troubleshooting#Known_Problems
FWSM
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/78_14450.htm
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/fwsm/index.htm
http://www.cisco.com/en/US/products/hw/modules/ps2706/products_qanda_item09186a0080
0c4fee.shtml
IOS FW
http://www.cisco.com/pcgi-
bin/Support/browse/psp_view.pl?p=Software:Cisco_IOS_Firewall&s=Implementation_and_Configuration
http://www.cisco.com/warp/partner/synchronicd/cc/pd/iosw/ioft/iofwft/prodlit/fire_qa.htm
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/iosfw2/iosfw2_2.htm#
xtocid135950
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 107
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 108