Fast IP Routing

Axel Clauberg
Consulting Engineer
Cisco Systems
Axel.Clauberg@cisco.com

AC_055_2000 © 2000, Cisco Systems, Inc. 1

 Agenda

• The Evolution of IP Routing

• Transmission Update: 10GE
• Router Architectures
• So, it‘s all just speed ?

AC_055_2000 © 2000, Cisco Systems, Inc. 2

 The Evolution of IP Routing

AC_055_2000 © 2000, Cisco Systems, Inc. 3

 Heard around the corner ?

• IP Routers are slow, sw-based

• IP Routers cause high latency
• IP Routers are undeterministic
• IP Routers do not support QoS

AC_055_2000 © 2000, Cisco Systems, Inc. 4

 WAN Customer Access Speed
Evolution

• Late 1980s: 9.6 Kb/s .. 64 Kb/s

• Early 1990s: 64 Kb/s .. 2 Mb/s
• Late 1990s: 2 Mb/s .. 155 Mb/s
• Early 2000s: 155 Mb/s .. 10 Gb/s

AC_055_2000 © 2000, Cisco Systems, Inc. 5

 Backbone Evolution

WAN Campus
• Late 1980s: 56/64 Kb/s • Late 1980s: 10 Mb/s
• Early 1990s: 1.5/2 Mb/s • Early 1990s: 100 Mb/s
(FDDI)
• Mid 1990s: 34 Mb/s,
• Mid 1990s: 155 Mb/s
155 Mb/s (ATM)
• Late 1990s: 622 Mb/s, • Late 1990s: nx FE, 155
2,5 Gb/s Mb/s, 622 Mb/s, GE
• Early 2000s: 10 Gb/s, • Early 2000s: 10 Gb/s, n x
40 Gb/s 10 Gb/s

AC_055_2000 © 2000, Cisco Systems, Inc. 6

 Transmission Update: 10GE

AC_055_2000 © 2000, Cisco Systems, Inc. 7

 MAN/WAN IP Transport
Alternatives
IP over IP over SDH IP over IP over
B-ISDN ATM Optical Ethernet
GE
Multiplexing, Protection and Management at every Layer
IP 10GE

ATM IP IP IP

SONET/SDH ATM SONET/SDH IP Ethernet

Optical Optical Optical Optical Optical

Lower Cost and Overhead

AC_055_2000 © 2000, Cisco Systems, Inc. 8
 Ethernet Scaling History

• 1981: Shared 10 Mbit 1x

• 1992: Switched 10 Mbit 10x
• 1995: Switched 100 Mbit 100X
• 1998: Switched 1 Gigabit 1000X
• 200x: Switched 10 Gigabit 10000X

AC_055_2000 © 2000, Cisco Systems, Inc. 9

 Moving the Decimal Point: 10 GbE Performance
and Scalability

10 Gbps STM-64
10 Gbps
Ethernet
Gigabit EtherChannel • LAN applications
• Metro applications
1 Gbps Gigabit Ethernet • WAN applications
10 GbE IEEE
Fast EtherChannel 802.3ae
Standard
100 Mbps Fast Ethernet

1996 1997 1998 1999 2000 2001 2002

AC_055_2000 © 2000, Cisco Systems, Inc. 10

 Why 10 Gigabit Ethernet

• Aggregates Gigabit Ethernet segments

• Scales Enterprise and Service Provider LAN
backbones
• Leverages installed base of 250 million
Ethernet switch ports
• Supports all services (packetized voice and
video, data)
• Supports metropolitan and wide area networks
• Faster and simpler than other alternatives

AC_055_2000 © 2000, Cisco Systems, Inc. 11

 IEEE Goals for 10 GbE
(Partial List)

• Preserve 802.3 Ethernet frame format

• Preserve minimum and maximum frame size of
current 802.3 Ethernet
• Support only full duplex operation
• Support 10,000 Mbps at MAC interface
• Define two families of PHYs
LAN PHY operating at 10 Gbps
Optional WAN PHY operating at a data rate
compatible with the payload rate of OC-
192c/SDH VC-4-64c

AC_055_2000 © 2000, Cisco Systems, Inc. 12

 IEEE 802.3ae Task Force Milestones

First 10GE
deliveries

1999 2000 2001 2002

HSSG
Formed PAR First Working LMSC
Drafted Draft Group Ballot
Ballot
PAR 802.3ae Standard
Approved Formed

HSSG= Higher Speed Study Group

PAR= project authorization request
802.3ae= the name of the project and the name of the sub-committee of IEEE 802.3 chartered with writing the
10GbE Standard
Working group ballot= task force submits complete draft to larger 802.3 committee for technical review and ballot
LMSC: LAN/MAN Standards Committee ballot. Any member of the superset of 802 committees may vote and
comment on draft
AC_055_2000 © 2000, Cisco Systems, Inc. 13
 10 Gigabit Ethernet Media Goals

Type Media Type Max Distance

1550 nm Laser std/dispersion free fiber 40-100 km

extended reach

1300 nm Laser
standard reach single mode fiber 2-10 km

1300 nm Laser
multimode fiber 300 m
CDWM (4x2.5)

780 nm VCSEL
ribbon multimode fiber 200 m
multichannel

AC_055_2000 © 2000, Cisco Systems, Inc. 14

 IEEE Status

• 802.3ae Meeting 10.-14. Juli 2000

• 75% Consensus
1550nm Transceiver 40 Km @ SMF
1300nm Transceiver 10 Km @ SMF
• No Consensus yet
Multimode Support
300m mit 62.5µ 160/500 Mhz*Km MM
50µ 2000/500 MHz*Km MM

AC_055_2000 © 2000, Cisco Systems, Inc. 15

 Router Architectures

AC_055_2000 © 2000, Cisco Systems, Inc. 16

 Components

• Memory Architecture
• Interconnect
• Forwarding Engine

• Scalability
• Stability
• Queueing / QoS

AC_055_2000 © 2000, Cisco Systems, Inc. 17

 Basic Design
Buffer
Interfaces
Memory
Inputs ... Router ... Outputs

Forwarding Route
Engine Processor

• Data are of random sizes

• Arrival is async, unpredictable, independantly on i/f
• Data have to be buffered
• TCP/IP traffic is bursty, but short-term congestion
only

AC_055_2000 © 2000, Cisco Systems, Inc. 18

 How much buffers ?

• Rule of Thumb: RTT x BW

(Villamizer & Song, High Performance TCP in ANSNET,
1994)

• STM-16 @ 200 ms: ~ 60 MB buffering

capacity

AC_055_2000 © 2000, Cisco Systems, Inc. 19

 How to Buffer ?

• SRAM
Fast, Power-hungry, Density 8 Mb -> 16
Mb, Simple Controller Design
• DRAM / SDRAM
Slower, Less Power, Density 64 Mb ->
256 Mb, Complex Controller Design

AC_055_2000 © 2000, Cisco Systems, Inc. 20

 Interconnect

• Switch Fabric / Crossbar

• Shared Memory
• Variations

AC_055_2000 © 2000, Cisco Systems, Inc. 21

 Switch Fabric / Crossbar

Ingress Line Cards Egress Line Cards

• Packet forwarding
Line Line
decision done on each Card 0 Card 0

linecard Switch
Fabric
Line Line
• Ingress and Egress Card 1 Card 1

Buffering on Linecards Scheduler

• Possible Problem: Head Line Line

Card N Card N
of Line Blocking RP RP

• Solution: VOQ

AC_055_2000 © 2000, Cisco Systems, Inc. 22

 Linecard in Detail
Physical Layer 3 Fabric
Layer Engine Interface
(Optics)
Switch
RX To Fabric
Fabric

CPU

Scheduler
TX From Fabric

• HOL Blocking can occur when packet cannot flow off transmit linecard
• Packet will be buffered on receiving linecard
• Packet blocks other packets to other linecards
• Solution: Virtual Output Queues, one per egress linecard
AC_055_2000 © 2000, Cisco Systems, Inc. 23
 GSR Queuing Architecture

Input Transmit Output

Ports Receive Line Card Line Card Ports

Group of 8 CoS
Queues
Virtual
Per Interface
Output (M-DRR)
W-RED DRR

Crossbar Switch Fabric

Queues

CAR

CEF

AC_055_2000 © 2000, Cisco Systems, Inc. 24

 Shared Memory Architecture
Physically Centralized

• One large memory Line Cards 1-8

system, data passing
through it 2.5Gbps
2.5Gbps

• Simple memory Interconnects

Memory Controller
Interconnects
&
&
management 40
Forwarding
Forwarding
Engine
Engine
G

• High speed memory 2.5Gbps

• Simple Linecards 2.5Gbps

• Needs SRAM for high

speeds

AC_055_2000 © 2000, Cisco Systems, Inc. 25

 Shared Memory Architecture
Distributed

• Memory distributed over linecards Line Cards 1-8

• Memory controller treats sum of

2.5Gbps
pieces as shared memory
2.5Gbps
Memory
• Packet forwarding decision in System
central engine(s)
Memory
Memory
• Difficult to maximize interconnect Controller
Controller
&
efficiency &
Forwarding
Forwarding
Engine(s)
Egress line cards simply request 2.5Gbps Engine(s)
packets from shared memory 2.5Gbps
Memory
Causes Head of Line (HOL) blocking System
and high latency, worsening under
moderate-to-heavy system load or
with multicast traffic

AC_055_2000 © 2000, Cisco Systems, Inc. 26

 Switch Fabric vs. Shared Memory

• Shared Memory requires only half the

buffer space
• HOL Blocking in Shared Memory,
especially for Multicast
• Involvement of distributed shared memory
causes more points of failure

AC_055_2000 © 2000, Cisco Systems, Inc. 27

 Forwarding Engine

• Classifying the packet

IPv4, IPv6, MPLS, ...
• Packet validity (TTL, length, ...)
• Next Hop
• Basic Statistics
• Optional:
• Policing, Extended Statistics, RPF check (security, Multicast),
QoS, Tunnel, ...
• Distributed vs. Central

AC_055_2000 © 2000, Cisco Systems, Inc. 28

 Central Forwarding ?

• IP Longest match
Hash vs. TCAM vs. Tree Lookup
• Tree Lookup requires high number of routing
table lookups
Need SRAM
Danger to run out of SRAM
Forwarding speed dependant on depth of routing
table

AC_055_2000 © 2000, Cisco Systems, Inc. 29

 Distributed Forwarding

• One copy of forwarding info per

linecard
• Parallel processing without sync or
communication between linecards
• Able to use TCAMs and SDRAMs

AC_055_2000 © 2000, Cisco Systems, Inc. 30

 So, it’s all just speed ?

AC_055_2000 © 2000, Cisco Systems, Inc. 31

 So, it’s just speed ?

• Services
IP Multicast
IP QoS
Security
• IPv6
• MPLS
• Manageability
• Availability
• Investment protection

AC_055_2000 © 2000, Cisco Systems, Inc. 32

 Multicast Solutions
End-to-End Architecture
ISP A
ISP B

MSDP
RP
Multicast Source
Multicast Source DR Y
X RP
ISP B

ISP A
MBGP
CGMP

DR

IGMP PIM-SM DR

Campus Multicast Interdomain Multicast

• End Stations (hosts-to-routers):
IGMP • Multicast routing across domains
• Switches (Layer 2 Optimization): MBGP
IGMP Snooping • Multicast Source Discovery
• Routers (Multicast Forwarding MSDP with PIM-SM
Protocol):
PIM Sparse Mode
AC_055_2000 © 2000, Cisco Systems, Inc. 33
 Summary

• IP Routers have evolved during the past

years
• Line rate up to 10 Gb/s
• Crossbar architectures with distributed
forwarding seem to scale better than
shared memory architectures
• Services remain the most decisive factor

AC_055_2000 © 2000, Cisco Systems, Inc. 34

 Outlook

• 10 Gb/s Interfaces supported in 2000

10GE, STM-64/OC-192
• High density of 10 Gb/s interfaces soon in a
PoP
• Next step will be STM-256/OC-768 = 40 Gb/s
• Will these routers be „Palm-Size“ ?
Probably not...

AC_055_2000 © 2000, Cisco Systems, Inc. 35

www.cisco.com