A best-in-class architecture enables the widest range of efficiency, performance, and flexibility advantages.
The Brocade 48000 Director is the industrys highest-performing platform for supporting enterprise-class Storage Area Network (SAN) operations. With its intelligent fifth-generation ASICs and new hardware and software capabilities, the Brocade 48000 provides a reliable foundation for fully connected multiprotocol SAN fabrics, FICON solutions, and Meta SANs capable of supporting thousands of servers and storage devices. This paper describes how IT organizations can leverage the benefits of this SAN director to maximize performance, flexibility, and data availability in mission-critical environments. In addition to summarizing the architectural advantages of the Brocade 48000, this paper explains how the various blades used in the platform can help optimize performance to address specific requirements. For more information about SAN design or additional Brocade solutions, such as the Brocade Multiprotocol Router, visit the Brocade Bookshelf at www.brocade.com/products/sanadmin_bookshelf.
OVERVIEW
In May 2005, Brocade introduced the Brocade 48000 Director (see Figure 1), a third-generation SAN director and the first in the industry to provide 4 Gbit/sec Fibre Channel capabilities. Since that time, the Brocade 48000 has become a key component in thousands of data centers around the world. Figure 1. The Brocade 48000 Director in a 384-port configuration.
Compared to competitive offerings introduced in 2006, the Brocade 48000 is the industrys fastest and most advanced SAN director, providing numerous advantages: The platform scales from as few as 16 to as many as 384 4 Gbit/sec ports in a single domain. The central memory architecture used in Brocade Application Specific Integrated Circuits (ASICs) is never subject to Head of Line Blocking (HoLB). The product design enables simultaneous uncongested operation on all ports as long as simple best practices are followed. The platform can provide over 1.5 Tbit/sec (3 Tbit/sec full duplex) of usable switching capacity in a chassis designed to support even higher port speeds in the future. In addition to providing the highest levels of performance, the Brocade 48000 features a modular high-availability architecture that supports five-nines environments. Moreover, the platforms industry-leading power and cooling efficiency help reduce ownership costs while maximizing rack density. The Brocade 48000 uses just 2.9 watts per port in its largest configuration (.75 watts per gigabit). This is twice as efficient as its predecessor, and up to six times more efficient than competitive products. This efficiency not only reduces data center electric billsit reduces cooling requirements and minimizes or eliminates the need for data center infrastructure upgrades, such as new PDUs, power circuits, and larger HVAC units. In addition, the highly integrated architecture uses fewer components per board, which improves key reliability metrics such as Mean Time Between Failure (MTBF).
The Brocade 48000 is also highly flexible, supporting Fibre Channel, FICON, FCIP with IPSEC, and iSCSI today, and additional protocols in the future. IT organizations can easily mix various Fibre Channel blade options to build an architecture that has the optimal price/performance ratio to meet the requirements of specific SAN environments. As of late 2006, the Brocade 48000 supports the following blades: Control processor CPU plus 256 Gbit/sec (512 Gbit/sec full duplex) backplane switching module 16-port 4 Gbit/sec Fibre Channel blade (FC4-16 or 16-port blade) 32-port 4 Gbit/sec Fibre Channel blade (FC4-32 or 32-port blade) 48-port 4 Gbit/sec Fibre Channel blade (FC4-48 or 48-port blade) 16-port 4 Gbit/sec Fibre Channel routing blade with two FCIP ports (FR4-18i or router blade) with FICON support 8-port 4 Gbit/sec Fibre Channel blade with eight iSCSI ports (FC4-16IP or iSCSI blade) Even though it provides all of these enterprise-class capabilities, the Brocade 48000 has plug-and-play setup characteristics, and IT organizations can follow a few simple guidelines to maximize its performance and availability. This paper describes the directors internal architecture and how to utilize the director and its blades to address particular business requirements.
Because port-blade Condor ASICs can act as independent switching engines, the Brocade 48000 can leverage localized switching within a port group in addition to switching over the backplane. On the 16- and 32-port blades, local switching is performed within 16-port groups and, on the 48-port blade, local switching is performed within 24-port groups. Unlike Unlike competitive competitive offerings, frames being switched offerings, frames being within port groups do not need to traverse the backplane. This enables every port on switched within port high-density blades to communicate at full groups do not need to 4 Gbit/sec speed with port-to-port latency traverse the backplane. of just 800 ns, 25 times better than the next-fastest SAN director on the market. The Brocade 48000 also has 1024 bufferto-buffer credits within each Condor ASIC to support longer-distance configurations. Similarly, hardware-enforced zoning resources provide more flexible hardware-enforced zone sets as well as increased security between the connected devices in a shared network. The Condor ASIC also enhances Brocade Inter-Switch Link (ISL) Trunking features with 32 Gbit/sec frame-level trunks (up to eight 4 Gbit/sec links in a trunk) and Dynamic Path Selection (DPS) for exchange-level and device-level balancing between trunk groups. Up to eight trunks can be balanced for 256 Gbit/sec (512 Gbit/sec full duplex). (A Fibre Channel exchange is generally equivalent to a SCSI operation.) Furthermore, Brocade has significantly improved frame-level trunking: trunks are now masterless. If any trunk member drops, the trunk will not have to re-build. The trunk bandwidth will drop proportionally but it will remain active.
32 Gbit/sec pipe
32 Gbit/sec pipe
Figure 3 shows how the blade positions in the Brocade 48000 are connected to each other using 16-port blades in a 128-port configuration.
On the left is an abstract cable-side view of the director, showing the ten slots populated with 16-port blades. On the right is a high-level diagram of how the slots interact with each other over the backplane. Each thick line represents 32 Gbit/sec (64 Gbit/sec full duplex) of internal links connecting the port blades with the control processor blades. The control processor blades contain the ASICs that switch between the port blades. Every port blade is connected to both control processors, and the aggregate bandwidth of these internal links is equal to the aggregate bandwidth available on external ports. Each port blade has sixteen 4 Gbit/sec ports (64 Gbit/sec, or 128 Gbit/sec full duplex) available externally, and sixteen 4 Gbit/sec ports (64 Gbit/sec, or 128 Gbit/sec full duplex) provided by internal links to the backplane. The every port blade to every control processor blade mesh design makes it analogous to a core/edge network, and the 1:1 internal/external bandwidth ratio makes it a fat-tree or non-over-subscribed layout. It is possible to locally switch between ports on the blade, but with a 1:1 backplane subscription ratio, it is not necessary to do so. The primary utility of local switching on this blade is to reduce port-to-port latency in order to provide the highest possible performance: crossing the backplane causes a 2.4 s latency, whereas locally switched frames cross the switch in only 800 ns. However, even 2.4 s is still 50 times faster than a read-cache hit on a disk array, and is much faster than any competing director product.
16 4 Gbit/sec Local Switching Group 16:8 Over-subscription 32 Gbit/sec Pipe ASIC 64 Gbit/sec to Control Processor ASIC 32 Gbit/sec Pipe
ASIC
Figure 5 shows how the blade positions in the director are connected to each other using 32-port blades in a 256-port configuration. Figure 5. Overview of a 256-port configuration.
When connecting a large number of devices that need sustained 4 Gbit/sec line rates, IT organizations can use locality to avoid congestion. The blade is divided into two 16-port groups for local switching. The physically lower 16 ports (ports 0 to 7 and ports 16 to 23) form one group and the upper ports (ports 8 to 15 and ports 24 to 31) form the other group. Figure 6 illustrates the internal connectivity between 32-port blades and the control processors.
Port Blade 1 16 4 Gbit/sec Condor ASIC Condor ASIC 32 Gbit/sec pipe (64 Gbit/sec full-duplex)
16 4 Gbit/sec
32 Gbit/sec pipe (64 Gbit/sec full-duplex) Each line is a 16 Gbit/sec frame-balanced pipe (32 Gbit/sec full-duplex)
Core CP 0 (slot 5)
Core CP 1 (slot 6)
There are two ASICs on each port blade, and each ASIC has a group of 16 outward-facing ports. For each group, there are two internal 8 Gbit/sec connections to each of the two control processors, for a total of 32 Gbit/sec (64 Gbit/sec full duplex) in backplane switching capacity. Traffic is balanced across the paths, such that the four 8 Gbit/sec connections form this virtual 32 Gbit/sec backplane pipe. Any combination of the 16 outward-facing ports in a group can use up to the full backplane bandwidth without congestion. This workload balancing and the resulting optimized performance represent the automatic behavior of the architecture and require no administration. If more than 32 Gbit/sec of total throughput is needed for each 16-port group, high-priority connections can be localized within the groupensuring that up to 16 devices or ISLs have ample bandwidth to connect to devices on other blades. Such connections do not use the backplane bandwidth. Likewise, localized traffic does not count against the subscription ratio and cannot be impacted by traffic from other devices. Regardless of the number of devices communicating over the backplane, locally switched devices are guaranteed 4 Gbit/sec bandwidth. This Brocade-unique technology for local switching helps preserve bandwidth to reduce the possibility of congestion in higher-density configurations.
32 Gbit/sec Pipe ASIC 64 Gbit/sec to Control Processor ASIC 24 4 Gbit/sec Local Switching Group 24:8 Over-subscription 32 Gbit/sec Pipe
10
ASIC
Encryption
11
Power and Control Path Fibre Channel Switching 8 Gbit Ethernet ports
32 Gbit/sec pipe
ASIC
32 Gbit/sec pipe
12
The shared memory architecture enables the entire director to be a single domain and a single hop in a Fibre Channel network.
It is important to understand that the internal ASIC connections within a Brocade 48000 are not E_Ports connecting a network of switches. The shared memory architecture enables the entire director to be a single domain and a single hop in a Fibre Channel network. Likewise, ASICs within a Brocade 48000 do not connect via E_Ports. When a port blade is removed, a fabric reconfiguration is not sent across the network, thereby simplifying operations.
However, unlike an actual fat-tree network, the Brocade 48000: Is easier to deploy and manage than the analogous network of switches Simplifies the cable plant by eliminating the ISLs and SFP media Is far more scalable, because it does not consist of a large number of independent domains Is less expensive in terms of both initial and ongoing costs Has far fewer active components and therefore much higher reliability Does not run switch-to-switch protocols (E_Port) between blades Provides multiprotocol support within a single chassis Is capable of achieving greater performance due to internal routing optimizations
13
ASIC
14
15
16
SUMMaRY
With an aggregate chassis bandwidth (backplane plus local switching) nearly three times that of competitive products, the Brocade 48000 is congestion-free in real-world cases that reflect the vast majority of SAN-based applications. Although congestion might occur in unique situations, it would be infrequent, low-level, and unlikely to impact application performance. Only in the worst contrived cases would congestion be noticeable at the application level. Even in those cases, congestion could be eliminated with very little effort by using local switching. The Brocade 48000 is designed to meet the most demanding performance requirements of a director-class SAN solution. As demonstrated by Brocade testing, the Brocade 48000: Delivers 4 Gbit/sec line-rate performance in full-duplex operation on all ports simultaneously Does not suffer from Head of Line Blocking (HoLB) Supports local switching for the highest-performance applications Is designed for maximum performance with real-world SAN traffic patterns Supports multiprotocol blades and applications Is designed to support future speeds and protocols For more information about the Brocade 48000, visit www.brocade.com. For more information about SAN design or other Brocade solutions, such as the Brocade Multiprotocol Router, visit the Brocade Bookshelf at www.brocade.com/products/sanadmin_bookshelf.
17
2007 Brocade Communications Systems, Inc. All Rights Reserved. 01/07 GA-WP-879-01 Brocade, the Brocade B-weave logo, Fabric OS, File Lifecycle Manager, MyView, Secure Fabric OS, SilkWorm, and StorageX are registered trademarks and the Brocade B-wing symbol and Tapestry are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.