Anda di halaman 1dari 55

TROUBLESHOOTING FIREWALLS

SESSION SEC-3020

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 1

Agenda

• Understanding the Concepts


• PIX® and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS® Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 2

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Agenda

• Understanding the Concepts


• PIX® and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS® Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 3

How the PIX Processes a Packet

Private Public

Entering PIX Leaving PIX


Packet Processed
Against ASA

Adaptive
Security
Algorithm

Randomize Sequence Numbers


Xlate and Connection Objects
Stateful Inspection
Security Levels
SEC-3020
Other rules
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 4

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
PIX Packet Processing
1 2 3 1. Receive packet
ACL No 2. Existing connection?
Recv Existing No
Pkt Permit Drop 3. Permit by Inbound ACL on
Conn
interface?
Yes Yes 4. Match translation rule
(nat, static)
Match
4 xlate 5. NAT embedded IP and
perform security
Yes checks/randomize sequence
number
L7 NAT 5
Sec Checks 6. NAT IP header
7. Pass packet to outgoing
interface
8. Layer 3 route lookup?
NAT IP 6
Header
9. Layer 2 next hop?
10. Transmit packet
8 9 10
Egress L3 Yes L2 Yes Xmit
Interface Route Addr Pkt

7
Drop Drop
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 5

PIX Release Process

Interim Images 6.0(2.100) 6.0(2.101) 6.0(2.122) Bug Fixes Committed


to Multiple Trains

CSCdv33495

6.0(2) 6.0(3) 6.0(4)

Maintenance 6.1(2) 6.1(3) 6.1(4) 6.1(5)


Trains
6.2(2) 6.2(3)

6.3(2) 6.3(3)

6.0(1) 6.1(1) 6.2(1) 6.3(1)

Major Feature Releases

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 6

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Agenda

• Understanding the Concepts


• PIX and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 7

PIX and FWSM Troubleshooting Tools

• Syslog and ICMP


• Packet Capture
• Show Commands
• Output Interpreter
• PDM (3.0)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 8

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
What Are Syslogs?

• Archival methods
• Primary mechanism to record traffic
TO and THROUGH the PIX
• One of the best troubleshooting tools
Debugging Methods
Archival Methods

Console

Syslog Server SNMP


Syslog
Internet

SSH Client

Buffered
SNMP Server
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 9

Syslog Level vs. Number of


Messages Logged

Log Level Description # of Messages (Sum)

0 Emergencies 0
1 Alerts 41 (41)
2 Critical 21 (62)
3 Errors 74 (136)
4 Warnings 56 (192)
5 Notifications 21 (213)
6 Informational 95 (308)
7 Debugging 15 (323)

* Based on PIX 6.3


SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 10

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Configuring and Using Syslog

Define internal
syslog server

Log error (level 3) pixfirewall (config) # logging host inside 10.1.1.5


messages and
below to syslog
server pixfirewall (config) # logging trap 3
Log alerts (level 1)
to the internal pixfirewall (config) # logging buffered 1
buffer

pixfirewall (config) # logging on


You must enable
logging
pixfirewall (config) # logging monitor 4
Log to a
SSH/Telnet
session as
necessary

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 11

What Are Modifiable Syslog Levels

[no] logging message <syslog_id> level <level>

• Modifiable Syslog Levels


Levels:
Allows one to move any syslog message
to any level 0–Emergency
1–Alert
• Example:
You want to record what commands are being 2–Critical
executed on the PIX; message 111009 records 3–Errors
this information, but by default it is at level 7
(Debug) 4–Warnings

%PIX-7-111009: User ‘johndoe’ executed 5–Notifications


cmd: show run 6–Informational
The problem is we don’t want to log all 322 other 7–Debugging
syslogs that are generated at debug level

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 12

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
How to Create Modifiable Syslog Levels

[no] logging message <syslog_id> level <level>

SOLUTION:
• Lower syslog message 111009 to level 3 (Error)
pix(config)# logging message 111009 level 3
- or –
pix(config)# logging message 111009 level error
• Now our syslog looks as follows:

%PIX-3-111009: User ‘johndoe’ executed cmd: show run


• To restore the default syslog level:
pix(config)# no logging message 111009 level error
- or –
pix(config)# logging message 111009 level 7

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 13

Notes on ICMP

2 DMZ
Ping
1 Ping Ping
Internet
Inside Outside
3
Bob

1 User Can Only Ping the Local Interface of the PIX

2 User Cannot Ping Remote Interface of the PIX

3 ICMP Echo-Replies Must be Explicitly Permitted thru the PIX


SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 14

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Debug ICMP Trace

Private
Internet
Network

http://www.cisco.com

1 User Able to Access Internet? No


2 Can Aser Access Private Network? Yes
3 Check Syslog Messages?
4 Test IP Connectivity—Ping
5 pixfirewall (config) # debug icmp trace

EXAMPLE OF DEBUG ICMP TRACE TO TEST SUCCESSFUL IP CONNECTIVITY


ICMP echo-request from inside:10.1.1.2 to 209.165.201.2 ID=3239 seq=4369 length=80
ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22

ICMP echo-reply from outside:209.165.201.2 to 209.165.201.22 ID=3239 seq=4369 length=80


ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 15

Disabling Debug Commands

undebug all

In 6.3 You Can Now Turn Off All Debugs Globally


by Issuing “no debug all" and "undebug all” or
“un all” for Short

pix(config)# show debug


debug icmp trace
debug sip

pix(config)# un all
pix(config)# show debug
pix(config)#
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 16

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
PIX and FWSM Troubleshooting Tools

• Syslog and ICMP


• Packet Capture
• Show Commands
• Output Interpreter
• PDM (3.0)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 17

Packet Capture

capture <capture-name> [access-list <acl-name>] [buffer <buf-size>]


[ethernet-type <type>] [interface <if-name>] [packet-length <bytes>]

• Capture command first introduced in PIX 6.2;


it depreciates the “debug packet” command
• Capture sniffs packets on an interface that match
an ACL
• Key steps:
Create an ACL that will match interesting traffic
Define the capture and bind it to an access-list and
interface
View the capture on the PIX, or copy it off in pcap format
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 18

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Packet Capture (Cont.)

• Traffic can be captured both before and after it


passes through the PIX; one capture on the Inside
interface, one capture on the Outside interface
• Capture buffer saved in RAM (default size 512kb)
• Default is to stop capturing when buffer is full
• Default packet-length is 68 bytes
(typically set to 1500)
• Copy captures off via TFTP or HTTPS

Capture In Capture Out

Inside Outside

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 19

Packet Capture: Example


No. Time Source Destination Protocol Info
15 •148.701751
Example: User on198.133.219.25
209.165.201.4 the Inside with
TCP an4511>http[SYN]
IP of 10.1.1.12 is havingAck=0
Seq=27007623614
a problem
16 148.704086 accessing
198.133.219.25 Cisco.com
209.165.201.4 TCP(198.133.219.25);
http>4511[SYN,ACK]the user is
Seq=979356760..
getting209.165.201.4
17 148.705398 NATed to 198.133.219.25
209.165.201.4.
TCP 4511>http[ACK] Seq=2707623615………
18 148.701751 209.165.201.4 198.133.219.25 HTTP GET /HTTP/1.1

Step 1: Create ACL for both Inside and Outside Interface


Access-list 100 permit tcp host 209.165.201.4 host 198.133.219.25 eq 80
Access-list 100 permit tcp host 198.133.219.25 eq 80 host 209.165.201.4
Access-list 101 permit tcp host 10.1.1.12 host 198.133.219.25 eq 80
Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.1.12
Step 2: Create Captures on Both Inside and Outside Interface
capture out access-list 100 interface outside packet-length 1500
capture in access-list 101 interface inside packet-length 1500
Step 3: Have Inside User Access www.cisco.com
Step 4: Copy the Captures off to a TFTP Server
copy capture:out-web tftp://10.1.1.10 pcap
copy capture:in-web tftp://10.1.1.10 pcap
OR copy using https:
https://<PIX_IP>/capture/out-web/pcap
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 20

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
PIX and FWSM Troubleshooting Tools

• Syslog and ICMP


• Packet Capture
• Show Commands
• Output Interpreter
• PDM (3.0)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 21

An Introduction to Show Commands

Displays Current
and Past Info
Show Related to the PIX Show
Perfmon Blocks

CLI Tool Used to Used to Monitor the


Extract Information Health of the PIX and
from the PIX for Draw a Baseline for
Information or your Network
Troubleshooting

Show cpu usage


Show Show xlate <detail> Show
Memory Show conn <detail> Processes
Show interface
Show traffic
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 22

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
“Show Conn” and “Show Conn <detail>”

pixfirewall (config) # show connection


2 in use, 2 most used
TCP out 192.150.49.10:23 in 10.1.1.15:1026 idle 0:00:22 bytes 1774 flags UIO
UDP out 192.150.49.10:31649 in 10.1.1.15:1028 idle 0:00:14 bytes 540 flags d

pixfirewall (config) # show connection detail


2 in use, 2 most used
Flags: A – awaiting inside ACK to SYN, a – awaiting outside ACK to SYN,
B – initial SYN from outside, D – DNS, d – dump,
E – outside back connection, F – outside FIN, f – inside FIN,
G – group, H – H.323, I – inbound data, M – SMTP data,
m – SIP media, O- outbound data, P – inside back connection,
q – SQL*Net data, R – outside acknowledged FIN,
R – UDP RPC, r – inside acknowledged FIN, S – awaiting inside SYN,
s – awaiting outside SYN, T – SIP, t – SIP transient, U – up
TCP outside: 192.150.49.10/23 dmz:10.1.1.15/1026 flags UIO
UDP outside: 192.150.49.10/31649 dmz:10.1.1.15/1028 flags d

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 23

Connection Termination Reasons

%PIX-6-302014: Teardown TCP connection number for interface_name:real_address/real_port to


interface_name:real_address/real_port duration time bytes number [reason] [(user)]

Reset-I Reset was from Inside

Reset-O Reset was from Outside

TCP FINs Normal Close Down Sequence

FIN Timeout Force Termination After 15 Seconds

SYN Timeout Force Termination After 2 Min

Xlate Clear Command Line Removal

Deny Terminate by Application Inspection

SYN Control Back Channel Initiation from Wrong Side

Uauth Deny Deny by URL Filter

Unknown Catch All Error


SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 24

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
“Show Xlate” and “Show Xlate <detail>”

pixfirewall (config) # show xlate


3 in use, 3 most used
PAT Global 192.150.49.1(0) Local 10.1.1.15 ICMP id 340
PAT Global 192.150.49.1 (1024) Local 10.1.1.15(1028)
PAT Global 192.150.49.1 (1024) Local 10.1.1.15(516)

pixfirewall (config) # show xlate detail


3 in use, 3 most used
Flags: D –DNS, d – dump, I – identity, I – inside, n – no random,
o – outside, r – portmap, s – static
TCP PAT from inside:10.1.1.15/1026 to outside:192.150.49.1/1024 flags ri
UDP PAT from inside:10.1.1.15/1028 to outside:192.150.49.1/1024 flags ri
ICMP PAT from inside:10.1.1.15/21505 to outside:192.150.49.1/0 flags ri

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 25

Xlate Flags

FLAG DESCRIPTION
s Static Translation Slot
d Dump Translation Slot on Next Clearing Cycle
r Port Map Translation
n No Randomization of TCP Sequence Number
o Outside Address Translation
i Inside Address Translation
D DNS A RR Rewrite
I Identity Translation from NAT 0

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 26

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Example – Connection through PIX
1. PIX receives an initial SYN packet from the Inside.
The SYN is permitted by the access-list, a translation (xlate) is built up,
and the connection is also created with the flags "saA".
2. The Outside device responds to the SYN packet with a SYN+ACK.
The connection flags are updated to reflect this, and now show "A".
3. The Inside device responds to the SYN+ACK with an ACK and this
completes the TCP 3-way handshake, and the connection is now
considered "up" (U flag).
4. The Outside device sends the first data packet. The connection is
updated and an "I" is added to the flags to indicate the PIX received
Inbound data on that connection.
5. Finally, the Inside device has sent a data packet and the connection
is updated to include the "O" flag.

Connection Flags
SYN+ACK
SYN
ACK
Data
5
1
3 42 U A
saA
UI
UIO
Inside Outside

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved.
Server Client 27

Example – Connection Teardown

1. PIX receives a FIN packet from the Inside.


As the FIN passes through the PIX, it updates the connection flags by
adding an “f” to indicate that the FIN was received on the Inside
interface.
2. The Outside device immediately responds to the FIN packet with a
FIN+ACK. The connection flags are updated to reflect this, and now
show “UfFR".
3. The Inside device responds to the FIN+ACK with a final ACK and the
PIX tears down the connection. Thus, there are no more connection
flags, because the connection no longer exists.

Connection Flags
FIN+ACK
ACK
FIN
1
3 2 UfUfFR
UfFRr
Inside Outside

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved.
Server Client 28

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Show CPU Usage

• First introduced in PIX OS version 6.0(1)


• Under normal conditions the PIX CPU should stay below 30%
(baseline as per Network); if the CPU reaches 100% the PIX will start
dropping packets
• The show cpu usage command displays the CPU over time as a
running average

An Example
pixfirewall# show cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1%

A Note
The Percentage Usage Prints as NA (Not Applicable) if the Usage is Unavailable
For the Specified Time Interval; this Can Happen if the User Asks For CPU
Usage Before the 5-second, 1-minute, or 5-minutes
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 29

Show Traffic

The Show Traffic Command Displays the Traffic, in Packets and in


Bytes, Out Each Interface of the PIX
An Example
pixfirewall# show traffic
outside:
received (in 124.650 secs):
295468 packets 167218253 bytes
2370 pkts/sec 1341502 bytes/sec
transmitted (in 124.650 secs):
260901 packets 120467981 bytes
2093 pkts/sec 966449 bytes/sec
inside:
received (in 124.650 secs):
261478 packets 120145678 bytes
2097 pkts/sec 963864 bytes/sec
transmitted (in 124.650 secs):
294649 packets 167380042 bytes
2363 pkts/sec 1342800 bytes/sec
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 30

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Show Blocks

• The show blocks command, along with the show cpu usage
command, are useful in determining whether the PIX is being
overloaded
• The blocks are internal storage locations, similar to queues on
a router; a packet is stored in a block until the PIX can process
it and place it on the outbound interface xmit queue

An Example
pixfirewall# show blocks What are these blocks used for?
256 = Stateful Failover, Syslog, TCP module
SIZE MAX LOW CNT 1550 = Ethernet Packets, buffering url filtered packets
4 1600 1597 1600 16384 = Only used for the Gig Ethernet cards
80 400 399 400
256 500 495 499
1550 1444 1170 1188
16384 2048 1532 1538
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 31

Show Local-Host

• The show local-host command displays the translation and


connection slots for all local hosts
• The clear local-host command stops traffic on all local hosts
• The clear local-host <ip_address> command stops traffic on the local
host specified by its IP address
An Example
pixfirewall# show local-host
Interface inside: 822 active, 823 maximum active, 0 denied
local host: <10.1.2.74>,
TCP connection count/limit = 0/unlimited
TCP embryonic count = 0
TCP intercept watermark = unlimited
UDP connection count/limit = 63/unlimited
AAA:
Xlate(s):
PAT Global 209.165.201.26(41166) Local 10.1.2.74(39075)

Conn(s):
UDP out 209.165.201.1:8943 in 10.1.2.74:63556 idle 0:01:31 flags -
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 32

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Show Tech-Support Enhancements (6.3)

• The show tech output was show version


enhanced to include some show clock
additional “show” commands that show memory
can be used to troubleshoot show conn count
memory and performance issues show xlate count
show blocks
• On the right are the commands show interface
included in the show tech output; show cpu usage
note: They are in order; (New show process
commands are in red) show failover
show traffic
show perfmon
show running-config

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 33

Show Output Filters

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>

Output Filters Have Been Added to PIX 6.3, Similar to the Ones
in IOS; to Use them, at the End of Show <command>, Use the Pipe
Character ‘|’ Followed By:
begin|include|exclude|grep [-v] <regular_exp>
to Filter the Show Output
Begin—Start Displaying the Output Beginning at the First
Match of the RegEx, and Continue to Display the
Remaining Output
include – Display any line that matches the RegEx
exclude – Display any line that does not match the RegEx
grep – same as include
grep –v – same as exclude

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 34

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
PIX and FWSM Troubleshooting Tools

• Syslog and ICMP


• Packet Capture
• Show Commands
• Output Interpreter
• PDM (3.0)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 35

Output Interpreter
GREAT TOOL TO CATCH COMMON CONFIGURATION ERRORS

Select the Output


In Question

Paste the Output

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 36

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Output Example

Snip of Output

Example
Of Messages

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 37

PIX and FWSM Troubleshooting Tools

• Syslog and ICMP


• Packet Capture
• Show Commands
• Output Interpreter
• PDM (3.0)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 38

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Using PDM for Monitoring

Device Information

Interface Statistics

Active VPN Tunnels

Current System
Utilization: CPU,
Memory, Traffic

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 39

Using PDM for Monitoring

PDM Can Graph Up


to 4 Different
Graphs at Once
Great for
Monitoring Trends

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 40

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Agenda

• Understanding the Concepts


• PIX and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 41

Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 42

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Accessing the Internet

10.10.10.x INSIDE OUTSIDE


.1 .1 .2
Internet
.3 63.1.1.x
http://www.xyz.com
.2
Web Server

Problem: Troubleshooting:
• Accessing the Internet • P ermissions
• T ranslation
• R outing

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 43

Permissions (Access-Lists)

10.10.10.x INSIDE OUTSIDE


.1 .1 .2
Internet
.3 63.1.1.x
http://www.xyz.com
.2
Web Server

Verify if any Access-lists Are Applied


Pixfirewall (config) # access-list acl permit tcp host 10.10.10.3 any eq www
Pixfirewall (config) # access-group acl in interface inside

Note
If you Have an Access-list Applied on the Inside Interface, Check to Make Sure
Traffic is Permitted Outbound; Remember, there is an Implicit Deny at the End
of an Access-list
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 44

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Translation

10.10.10.x INSIDE OUTSIDE


.1 .1 .2
Internet
.3 63.1.1.x
http://www.xyz.com
.2
Web Server

1 Verified Permissions
Verify Translation Commands
global (outside) 1 63.1.1.10–63.1.1.50 netmask 255.255.255.0
nat (inside) 1 10.10.10.0 255.255.255.0

global (outside) 1 interface


nat (inside) 1 10.10.10.0 255.255.255.0

static (inside,outside) 63.1.1.51 10.10.10.3 netmask 255.255.255.255

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 45

Routing
INSIDE OUTSIDE
.1 .2
.2 .3 .1 .1 .2
Internet
192.168.1.x 10.10.10.x 63.1.1.x

1 Verified Permissions
2 Verified Translation

Verify Route Commands on the PIX


• Check to make sure the pix has the correct default gateway
pixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2
• If trying to access the internet from behind a layer 3 device, verify the
PIX has a route to that network
Pixfirewall (config)# route inside 192.168.1.0 255.255.255.0 10.10.10.3

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 46

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 47

Accessing Internal Network From Internet


Web Server 10.10.10.x INSIDE OUTSIDE
.1 .1 .2
Internet
.1 63.1.1.x

http://www.xyz.com

Problem: Troubleshooting:
• Internal web server not • P ermissions
accessible to users on the • T ranslation
Internet
• R outing

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 48

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Permissions (Access-Lists)
Web Server 10.10.10.x INSIDE OUTSIDE
.1 .1 .2
Internet
.1 63.1.1.x

http://www.xyz.com

Traffic has to be Explicitly Allowed into the PIX from a Lower Security to a
Higher Security

CHECK TO MAKE SURE YOU HAVE PERMITTED INTERESTING TRAFFIC EXPLICITLY


Pixfirewall (config) # access-list acl permit tcp any host 63.1.1.10 eq http
Pixfirewall (config) # access-group acl in interface outside
• If you have an access-list applied, check to make sure traffic is permitted inbound
Remember, there is an implicit deny at the end of an access-list

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 49

Translation
Web Server 10.10.10.x INSIDE OUTSIDE
.1 .1 .2
Internet
.1 63.1.1.x

http://www.xyz.com

A Static Translation is Required to Pass Inbound Traffic

Verify Translation Commands


Pixfirewall(config) #static(inside,outside) 63.1.1.10 10.10.10.1 netmask 255.255.255.255

Example of a Syslog Message With No Static Defined


305005: No translation group found for tcp src outside:200.1.1.1/35550 dst inside:63.1.1.10/80

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 50

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Routing Issues
Web Server 10.10.10.x INSIDE OUTSIDE
.1 .1 .2
Internet
.1 63.1.1.x

http://www.xyz.com

• Check to make sure the PIX has the correct default gateway
pixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2

• Verify the PIX has a route to the internal network


pixfirewall(config)# route inside 172.16.171.0 255.255.255 10.10.10.2
• Other Issues to Consider
Confirm default gateway on your Web Server
Verify your layer 3 device is routing correctly

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 51

Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 52

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Issues with Traffic Between Interfaces

DMZ (30)

INSIDE (100) OUTSIDE (0)

DMZ 1 (40)

1 Static and Access-List/Conduit


2 Static and Access-List/Conduit
3 Static or a NAT/Global Statement
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 53

Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 54

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
PIX Is NOT Redirecting

INSIDE OUTSIDE

.2 .1 .2 .1 .1
Internet
172.16.171.x 63.1.1.x
10.10.10.x

1 PIX Will Not Handle Redirects

2 Change User’s Default Gateway to Be the Layer 3 Device

3 Modify the Layer 3 Device’s Default Gateway to Be the PIX

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 55

Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 56

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
PIX Failover

• Active/Standby vs.
Primary/Secondary
• Serial Failover/LAN Failover
• Stateful Failover
• A Failover ONLY occurs
Stateful
when either PIX determines
the Standby PIX is healthier Serial
than the Active PIX Lan FO
Active Standby
• Both PIXes swap MAC and Unit Unit
IP Addresses when a
failover occurs
• Level 1 syslogs will give
reason of failover
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 57

Commands to Verify Failover Is Active

Pixfirewall (config)# show failover

Failover On
Cable status: Normal
Reconnect timeout 0:00:00
This host: Primary - Standby (Failed)
Active time: 7140 (sec)
Interface 0 (192.168.1.1): Normal
Interface 1 (172.16.171.54): Normal
Other host: Secondary - Active
Active time: 30 (sec)
Interface 0 (192.168.1.3): Normal
Interface 1 (172.16.171.55): Normal

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 58

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Why Will Failover Happen?

Power Failure Network Activity


Test

ARP Test Link Test

Failover
Ping Test Cable Failure

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 59

Using Syslogs to Troubleshoot Failover

1 No Failover Hello Seen on Serial Cable for 30 + Seconds; This Ensures that
Failover is Running Properly on the Other PIX
2 An Interface Did not Pass One of the 4 Failover Tests
(Link up, Interface Traffic, ARP Test, Broadcast Ping)
3 No Proper ACK for 15+ Seconds After a Command has Been Sent on the
Serial Cable

Message on Secondary when Primary Did Not Respond


%PIX-1-103001: (Secondary) No response from other firewall (reason code = 3)

Example of Syslogs when Primary Was Active, and It Lost One of Its Interfaces

%PIX-4-411002: Line protocol on Interface intf2, changed state to down


%PIX-1-105007: (Primary) Link status ‘Down’ on interface 1
%PIX-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

%PIX-1-104001: (Secondary) Switching to ACTIVE—mate want me Active


SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 60

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 61

PIX Password Recovery

• Password recovery can only be performed by uploading


the Password Recovery utility to the PIX from Monitor
mode via TFTP
• Password recovery will also remove any AAA
commands
• A password recovery utility is created for each major
PIX release (6.1, 6.2, 6.3…) however, the utilities are
backwards compatible

EX: np63.bin for PIX 6.3 and prior releases

http://www.cisco.com/warp/public/110/34.shtml
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 62

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Example of Password Recovery

monitor> interface 1 Inside


TFTP
monitor> address 14.36.1.88
monitor> file np63.bin
monitor> gateway 14.36.1.1 172.18.108.26 14.36.1.88
monitor> server 172.18.108.26
monitor> tftp
tftp np63.bin@172.18.108.26 via 14.36.1.1............... Received 92160 bytes

Do you wish to erase the passwords? [yn] y


The following lines will be removed from the configuration:
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
aaa authentication serial console LOCAL
aaa authentication telnet console LOCAL
aaa authentication ssh console LOCAL
aaa authentication enable console LOCAL

Do you want to remove the commands listed above from the configuration? [yn] y
Passwords and aaa commands have been erased.

Rebooting..
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 63

Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 64

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
High CPU Usage on the PIX

Problem: PIX CPU Running Very High

A Quick Overview of the “Show Processes” Command

Number of msec
The Name of the
this Process Has
Process
been on the CPU

pixfirewall(config)# show processes

PC SP STATE Runtime SBASE Stack Process


Hsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timer
Lsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGC
Lwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtrace
Lwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 Logger
Crd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557poll
Lsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean

For more information on the output of the “show processes” command, see:
http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 65

High CPU Usage on the PIX

Step 1: Determine What Process Is Eating the CPU


Take the Difference in Output of Two
“Show Processes” over a Period of Time

The Following Output Was a Diff of the Processes Taken 1 Minute Apart

In one minute, these


Process_Name Runtime (msec) Processes Account
for 44 Seconds of
Logger 25940
CPU Time ~ 73%
pix/intf3 18410
557poll 9250
The Interface Polling
i82543_timer 4180 Processes Always
i82542_timer 2230 Run, and Are not
Counted in the CPU
Usage

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 66

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
High CPU Usage on the PIX

Step 2: Focus on the Processes with High CPU Time


Logging Is Taking up Much of the CPU, Let’s Review
What We Have Configured to Log
This is Cumulative
pixfirewall(config)# show log Since the PIX Was
Syslog logging: enabled Last Rebooted
Standby logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: level alerts, 0 messages logged
Trap logging: level warnings, 5919412 messages logged
Logging to lab 172.18.173.123
History logging: disabled
Notice the Change
. . . Over a Few Minutes
pixfirewall(config)# show log
Syslog logging: enabled
Buffer logging: level alerts, 0 messages logged
Trap logging: level warnings, 6172472 messages logged
SEC-3020 Logging to lab 172.18.173.123
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 67

High CPU Usage on the PIX

Syslog Server Is Controlled by a Different Group


Enable Buffered Logging to Same Level as Syslog
Server, and Examine the Buffered Messages

pixfirewall(config)# show log


Buffer logging: level warnings, 31527 messages logged
Trap logging: level warnings, 6453127 messages logged PIX’s Interface
Logging to lab 172.18.173.123 Address

. . .
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab
400011: IDS:2001 ICMP unreachable from 172.18.173.123 to 14.36.1.88 on interface lab

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 68

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
High CPU Usage on the PIX

EXAMINE IDS CONFIGURATION


pixfirewall(config)# show run | grep audit
ip audit name IDS info action alarm
ip audit interface lab IDS

• Syslog service was down on the syslog server


• ICMP Unreachable was generated by syslog server for each
syslog message the PIX sent it
• PIX’s IDS configuration also logged every ICMP Unreachable
message, creating the exponentially increasing problem

Syslog LAB OUTSIDE


Server
Syslog Message
ICMP Unreachable
IDS Syslog Message
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 69

High CPU Usage on the PIX

SOLUTION:
• Bring back up syslog service on server
• Take server offline
• Configure PIX to not log IDS ICMP Unreachable
messages
ip audit signature 2001 disable
- or -
no logging message 400011
pixfirewall# show run | grep signature
ip audit signature 2001 disable

pixfirewall# show cpu usage


CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99%

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 70

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
High CPU Usage on the PIX

SUMMARY:
• Examine the DIFF of two show processes taken
over a one minute interval
• Find the process taking up the highest amount of
CPU (excluding the polling processes)
• Take actions to lower that processes’ CPU time
• Re-examine the CPU output, and repeat as
necessary

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 71

Common Issues

• PIX Common Issues


Accessing the Internet
Accessing internal network from the Internet
Issues with traffic between interfaces
PIX not redirecting
Failover
Password recovery
High CPU
Out-of-memory issues
• FWSM Common Issues
Configuration issues
Passing traffic outbound
Understanding failover

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 72

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Out of Memory

PROBLEM:
• Users are unable to access the Internet
• No new connections are working;
• All old (long lived) connections continue to work

Step 1: Check the Syslogs


%PIX-3-211001: Memory allocation Error
%PIX-3-211001: Memory allocation Error

Step 2: Check the Amount of Free Memory Available


Hardware: PIX-515E, 64 MB RAM

pixfirewall# show memory


Free memory: 714696 bytes
Used memory: 66394168 bytes
------------- ----------------
Total memory: 67108864 bytes
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 73

Out of Memory

Step 3: What Eats Up Memory (RAM) on the PIX?


PIX image (run from RAM)
Configuration
IPSec Database What Can Eat
Xlates (translations) Up 64MB
Connections on a PIX-515 E

Step 4: Let’s Check the Translations A Small Global


pixfirewall# show xlate Pool is Used,
251 in use, 258 most used Overloading to
PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) a PAT Address
PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276)
Global 209.165.201.25 Local 10.1.1.102
PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783)
PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197)
PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)

Varied Source
IPs
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 74

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Out of Memory

Step 5: Check the Connections


pixfirewall# show conn
147456 in use, 147456 most used
TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU
TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU
TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA
TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU
TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU
TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA
TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU
TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU
TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU
TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU
TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU
TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU
TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU
TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA
. . .

Q: WHY IS THE CONNECTION COUNT SO HIGH?


SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 75

Out of Memory

Take a Look at the Traffic Load


pixfirewall# show traffic
Outside

outside:
received (in 25.000 secs):
1475 packets 469050 bytes
59 pkts/sec 18762 bytes/sec
transmitted (in 25.000 secs):
167619 packets 9654480 bytes
Traffic Flow

6704 pkts/sec 386179 bytes/sec


inside:
received (in 25.000 secs):
Inside

180224 packets 10410480 bytes


7208 pkts/sec 416419 bytes/sec
transmitted (in 25.000 secs):
1050 packets 118650 bytes
42 pkts/sec 4746 bytes/sec

Vast Majority of Traffic Is Coming in the Inside


Interface and Going Out the Outside Interface
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 76

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Out of Memory

Step 6: Review What We Know and Take Action


pixfirewall# show conn count
147456 in use, 147456 most used

pixfirewall# show xlate count


251 in use, 258 most used

Conn Count Is Very High, but xlate Count Is Low


• Many connections per xlate
• Probably one, or a few hosts, are generating the vast majority
of connections
• Most likely due to a virus on the host(s)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 77

Out of Memory

Step 7: Find the Host(s) Generating All the Connections

pixfirewall# show local-host | include host|count/limit


local host: <10.1.1.131>,
TCP connection count/limit = 0/unlimited
UDP connection count/limit = 0/unlimited
local host: <10.1.1.51>,
TCP connection count/limit = 2/unlimited
UDP connection count/limit = 0/unlimited Only Show Lines
local host: <10.1.1.236>, that Have the Word
TCP connection count/limit = 0/unlimited “host” OR
UDP connection count/limit = 0/unlimited “count/limit”
. . . In Them
local host: <10.1.1.99>,
TCP connection count/limit = 146608/unlimited
UDP connection count/limit = 0/unlimited

Host 10.1.1.99 Is Eating Up All the Connections,


and They Are TCP Based Connections
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 78

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Out of Memory

Step 8: Now that We Found the Host, Let’s Look at the


Connections it Is Generating
pixfirewall# show local-host 10.1.1.99
Interface inside: 250 active, 250 maximum active, 0 denied
local host: <10.1.1.99>,
TCP connection count/limit = 146608/unlimited
TCP embryonic count = 146606 All Connections
UDP connection count/limit = 0/unlimited Are Embryonic
Xlate(s): (Refer to Slide 23)
Global 209.165.201.21 Local 10.1.1.99
Conn(s):
TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA
TCP out 64.100.205.160:135 in 10.1.1.99:7774 Connections
idle 0:01:43to Bytes 0 flags saA
TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags
Random saA
TCP out 64.102.218.45:135 in 10.1.1.99:16462 Destinations
idle 0:01:43on Bytes 0 flags saA
TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags
TCP/135– saA
TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags
MS Blaster saA
TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA
TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA
TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA
SEC-3020 . . .
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 79

Out of Memory

Step 9: Limit Infected Host(s) Impact on Network

PIX Provides Two Methods to Limit the Number of Connections


per Host:
• TCP Intercept
• Max Connections
Which One can be Used Here?

TCP Intercept Won’t Help Because the Source Address Is VALID


Limiting the Maximum Number of Connections Each Internal Host
Can Have Is the Only Option

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 80

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Out of Memory

Step 9: Limit Infected Host(s) Impact on Network

Configure the MAX TCP Connections for NATed Hosts to be 50:


pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0

NOTE: The Local-host MUST be Cleared Before the New


Connection Limits Are Applied
pixfirewall(config)# clear local-host 10.1.1.99

pixfirewall(config)# show local-host 10.1.1.99


Interface inside: 250 active, 250 maximum active, 0 denied
local host: <10.1.1.99>,
TCP connection count/limit = 50/50
The Infected
TCP embryonic count = 50
Host is Limited
TCP intercept watermark = unlimited
to 50 TCP
UDP connection count/limit = 0/unlimited
Connections
. . .

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 81

Out of Memory

Take One Last Look at the Memory and Connection


Counts After Applying the TCP Connection Limit
pixfirewall# show conn count
126 in use, 147456 most used

pixfirewall# show memory


Free memory: 47716152 bytes
Used memory: 19392712 bytes
------------- ----------------
Total memory: 67108864 bytes

Things Look Much Better Now


Question: How Could we Configure the PIX so the Connection
Limit Was only Applied to the One Host (10.1.1.99) which Was
Infected with the Virus?
nat (inside) 1 10.1.1.99 255.255.255.255 50 0
nat (inside) 1 0.0.0.0 0.0.0.0 0 0

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 82

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Still Can’t Resolve It? Open a TAC Case

If After Using All Your Troubleshooting Tools You Still


cannot Resolve the Problem, Please Open a TAC Case:
http://www.cisco.com/techsupport/servicerequest/

• At a minimum include:
Detailed problem description
Output from “show tech”

• Optionally include:
Syslogs captured during time of problem
Sniffer traces from both interfaces using the “capture”
command (capturing only the relevant packets, and saved
in pcap format)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 83

PIX/FWSM Best Practices

• Enable ip verify reverse-path on all interfaces


• Set Embryonic and Maximum connection counts on static
and nat statements
• Configure logging to syslog server
• Move messages you want to see to lower levels, instead of
raising logging levels and capturing messages you don’t want
to see
• Disable telnet access, use SSH for management access
• Enable Authentication for Management access
(Console/SSH/Telnet/Enable); use TACACS+, RADIUS or
LOCAL database
• Restrict DMZ access inbound to your Internal Networks
• Baseline CPU load, connection counts, xlate counts, and
traffic (per interface)
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 84

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Agenda

• Understanding the Concepts


• PIX and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 85

FWSM Code Base History

Feature Releases

PIX 6.0(1) 6.1(1) 6.2(1) 6.3(1)

Port Features
Bug Fixes
Feature Releases
FWSM 1.1(1) 2.2(1)

1.1(2) 1.1(3)
New Features Maintenance New Features
Releases
OSPF, Virtual FW,
crashinfo, etc. Transparent FW, etc.

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 86

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
FWSM Bug Fix Process

Feature Releases

PIX 6.0(1) 6.1(1) 6.2(1) 6.3(1)

Bug Found and


Fixed in PIX CSCdw27548

Port Features
Cloned to FWSM Bug Fixes

CSCeb16339

FWSM 1.1(1) 2.2(1)


Fix Ported to FWSM

1.1(2) 1.1(3)
Maintenance
Releases CSCeb76295
Bug Fixes Committed
to Multiple Trains
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 87

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 88

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Understanding the Configuration

Catalyst OS 7.6(3) Native IOS—12.2(17a)SX1


6K> (enable) set vlan 30,60 firewall-vlan 5 Router#
vlans 30,60 declared secure for !
firewall module 5 vlan 30,60
firewall vlan-group 1 30,60
----------------- firewall module 5 vlan-group 1
-----------------
6K> (enable) session 5
FWSM# FWSM#
nameif vlan30 outside security0 nameif vlan30 outside security0
nameif vlan60 inside security100 nameif vlan60 inside security100
ip address outside 10.30.1.2 255.255.255.0 ip address outside 10.30.1.1 255.255.255.0
ip address inside 10.60.1.1 255.255.255.0 ip address inside 10.60.1.1 255.255.255.0

Slot 5
OUTSIDE INSIDE
Vlan 30 Vlan 60
10.30.1.0/24 10.30.1.1 10.60.1.1 10.60.1.0/24

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 89

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 90

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Agenda

• Understanding the Concepts


• PIX and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 91

Cisco IOS Firewall Platform Compatibility

• Cisco IOS Firewall was introduced in 12.0(5)T


• Flash and RAM requirements vary depending on
the router platform
• Router platforms include:
Small Office: 800* and uBR900 series
Branch Offices: 1600, 2500, 2600, and 3600 series
WAN and high throughput: 7100, 7200, 7500, and RSM

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 92

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
What Are the Features
of Cisco IOS Firewall?

Stateful Packet DoS Detection and Real Time Alerts and


Inspection Prevention Audit Trail

Dynamic Port Basic and Advanced


Intrusion Detection
Mapping Traffic Filtering

Per-user Policy Based


Authentication and Java Blocking Multi-Interface
Authorization Support

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 93

What Is CBAC

INSIDE OUTSIDE
ip inspect name foo tcp access-list 101 deny ip any any
interface ethernet0 interface ethernet1
ip inspect foo in ip access-group 101 in

SYN
A:a B:b
SYN + ACK
A:a B:b
ACK
A:a B:b

RST
A:a B:b
RST
A:a B:b

• Allow return traffic


• Deny traffic initiating from outside
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 94

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
How Does CBAC Work and Its Effects

Can this Crash My Router Do I Inspect All Interfaces

INTERNAL NETWORK EXTERNAL NETWORK

e0 e1

What Do I Inspect? and Why Access-Lists


or
Why Do I Inspect? and Dynamic ACL’s?

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 95

Didn’t CBAC Create Dynamic ACL’s

Firewall ACL Bypass:


• Introduced in 12.3(4) T
• ACL bypassing subjects the packet to one search—the inspection session
search against the list of session identifiers that already exist on the interface

Established Sessions
Session 814063CC (192.168.1.116:32955)=>(192.168.101.115:23) tcp SIS_OPEN
Created 00:00:10, Last heard 00:00:06
Bytes sent (initiator:responder) [140:298]
In SID 192.168.101.115[23:23]=>192.168.1.117[32955:32955] on ACL 101 (15 matches)
Out SID 192.168.101.115[23:23]=>192.168.1.116[32955:32955] on ACL 102

Router# show access-list


Extended IP access list 101
deny udp any any (20229 matches)
deny tcp any any
permit ip any any (6 matches)
Extended IP access list 102
deny udp any any
deny tcp any any
permit ip any any (1 match)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 96

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
State Tables and Session Maintenance

State Tables:
• Maintains session state information
• Updated when a packet is inspected at the firewall's interface
• State table updates will permit return traffic through the firewall for packets
traveling within a permissible session
• The state table is updated to remove the temporary opening in the access
list when the session terminates

Session Maintenance
• Use of timeouts and threshold values to manage session state information
• Timeouts used to prevent DoS attacks by dropping half-open connections
and freeing up network resources
• Threshold values used to prevent DoS attacks by controlling the number of
half-open sessions

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 97

Thresholds

What Are Thresholds

• Set for the total number of half-open TCP and UDP sessions
• Set for the total number of half-open session based on time
• Set for the total number of half-open TCP-only sessions per host

Configure Thresholds
• ip inspect max-incomplete {high|low}
This will modify the number of allowed max-incomplete sessions from the defaults of
500/400
• ip inspect one-minute {high|low}
This will modify the number of unestablish sessions allowed in one minute from the
defaults of 500/400
• ip inspect tcp max-incomplete host <value>
This will modify the number of incomplete sessions one particular host is allowed to have
at any time from the default of 50

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 98

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Agenda

• Understanding the Concepts


• PIX and FWSM Troubleshooting Tools
• PIX and FWSM Common Issues
• Cisco IOS Firewall Concepts
• Cisco IOS Firewall Common Issues and
Troubleshooting

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 99

Show Commands

show ip access-list
show ip inspect name inspection-name
show ip inspect config
show ip inspect config
show ip inspect interfaces
show ip inspect session [detail]
show ip inspect all
show ip inspect stat
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft
/120t/120t5/iosfw2/iosfw2_2.htm#12583

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 100

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Example of Show Commands
• Router# show ip inspect session
Established Sessions Session 25A3318 (10.0.0.1:20)=>(10.1.0.1:46068) ftp-
data SIS_OPEN Session 25A6E1C (10.1.0.1:46065)=>(10.0.0.1:21)
ftp SIS_OPEN

• Router# show ip inspect name myinspectionrule


Inspection Rule Configuration Inspection name myinspectionrule
tcp timeout 3600
udp timeout 30
ftp timeout 3600

• The following is sample output for the show ip inspect


interfaces command:
Interface Configuration
Interface Ethernet0
Inbound inspection rule is myinspectionrule
tcp timeout 3600
udp timeout 30
ftp timeout 3600
Outgoing inspection rule is not set
Inbound access list is not set
Outgoing access list is not set
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 101

Debugging CBAC

Audit Trails
ip inspect audit-trail

Generic Debug
debug ip inspect object-creation
debug ip inspect object-deletion
debug ip inspect events
debug ip inspect timers
debug ip inspect detail

Transport Level Debugs


debug ip inspect tcp
debug ip inspect udp
Application Protocol Debugs
debug ip inspect protocol

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 102

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Common Debugging Techniques

If access-list may be a suspect:


• Study the logic of your list or try defining an additional
broader list:
access-list # permit tcp any any
access-list # permit udp any any
access-list # permit icmp any any int <interface>
ip access-group # in|out
• Use an extended access-list with a log option at the end:
access-list 101 deny ip host 171.68.118.100 host 10.31.1.161 log
access-list 101 permit ip any any
• If the ip inspect list is suspect, try debug ip inspect
<type_of_traffic> :

Feb 14 12:41:17 10.31.1.52 56: 3d05h: CBAC* sis 258488 pak 16D0DC TCP P ack
3195751223 seq 3659219376(2) (10.31.1.5:11109) => (12.34.56.79:23)

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 103

Configuration Issues

THE MOST COMMON CONFIGURATION ERROR IS


THE DIRECTION OF INSPECTION

PROTECTED NETWORK UNPROTECTED NETWORK

e0 e1

Inspect Inbound and Access List Inbound


on e0 or on e1
Inspect Outbound and Access List Inbound
on e1 on e1

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 104

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Cisco IOS FW Dropping Packets

• Base Line Your Network


• Adjust Your Threshold Values As Needed
• Check Your Access-Lists
• Verify Your Inspect Statements
• Check for Asymmetrical Routing

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 105

Tips for Troubleshooting CBAC

If Traffic is Being Denied:


See if an access-list Is not denying traffic; remove the
access-group and see if traffic in question Is permitted;
If Possible Apply ExtendedAccess-lists

Log Your Deny Statements Temporarily


Router(config)#ip Access-list Extended IOSFW
Router(config-ext-nacl)#deny ip any any Log
Debugs on the Router:
CBAC related debugs will give a lot of information if CBAC Is
working the way it is supposed to be and return traffic Is permitted
Debug IP Packet Detail:
Router(config) # access-list 101 tcp host 10.1.1.1 host 192.168.1.1
Router # debug ip packet detail 101
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 106

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
Helpful Resources

PIX Firewall
http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_sw/v_63/index.htmhttp://www.cisco.com/cgi
-bin/tablebuild.pl/pix
http://www.cisco.com/pcgi-bin/Support/browse/psp_view.pl?p=Hardware:PIX&s=Software_Configuration
http://www.cisco.com/pcgi-
bin/Support/browse/psp_view.pl?p=Hardware:PIX&s=Troubleshooting#Known_Problems

FWSM
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/78_14450.htm
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/fwsm/index.htm
http://www.cisco.com/en/US/products/hw/modules/ps2706/products_qanda_item09186a0080
0c4fee.shtml

IOS FW
http://www.cisco.com/pcgi-
bin/Support/browse/psp_view.pl?p=Software:Cisco_IOS_Firewall&s=Implementation_and_Configuration
http://www.cisco.com/warp/partner/synchronicd/cc/pd/iosw/ioft/iofwft/prodlit/fire_qa.htm
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/iosfw2/iosfw2_2.htm#
xtocid135950

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 107

Complete Your Online Session Evaluation!

WHAT: Complete an online session evaluation


and your name will be entered into a
daily drawing
WHY: Win fabulous prizes! Give us your feedback!
WHERE: Go to the Internet stations located
throughout the Convention Center
HOW: Winners will be posted on the onsite
Networkers Website; four winners per day

SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 108

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2
SEC-3020
9827_05_2004_c2 © 2004 Cisco Systems, Inc. All rights reserved. 109

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.


9827_05_2004_c2

Anda mungkin juga menyukai