Connectris B-Series Student Resource Guide PDF

Self Study modules consist of focused, in-depth training content that can be consumed in about 1-2 hours
EMC Global Education Self Study

Home
Welcome to Connectrix B-Series Architecture and Management
Complete Course Directions on how to
update your online transcript to reflect a
complete status for this course.
Course Description
Start Training Run/Download the PowerPoint presentation
Student Resource Guide Training slides with notes
Assessment Must be completed online
(Note: Completed assessments will be reflected online within 24-48 hrs.)
For questions or support please contact Global Education
2004 EMC Corporation. All rights reserved.
Link to Knowledgelink to update your transcript and indicate that you have completed the course.
EMC Global Education Self Study Course Completion
Connectrix B-Series Architecture and Management
Course Completion Steps:
1. Logon to Knowledgelink (EMC Learning management system).
2. Click on 'My Development'.
3. Locate the entry for this learning event you wish to complete.
4. Click on the complete icon [ ].
Note: The Mark Complete button does not apply to items with the Type: Class, Downloadable (AICC Compliant)
or Assessment Test. Any item you cancel from your Enrollments will automatically be deleted from your
Development Plan.
Click here to link to Knowledgelink
For questions or support please contact Global Education
Back to Home

If you have any questions, please contact us by email at GlobalEd@emc.com. Page 1 of 1
Course Description
Connectrix B-Series Architecture and Management- Self Study

EMC Global Education
e-Learning

This foundation level IMPACT module provides the
learner with an overview of the hardware and software
configurations of B-series switches. It introduces the
range of products available through EMC and discusses
alternative ways to manage them. SAN security issues are presented from the perspective of the features
available through the switch.
Course
Number:
MR-9WP-NSBROC
Method: Self Study Duration: 2 hours

Audience
This course is intended for any person who presently or plans to be engaged in the
Planning or Architecting of a SAN using B-series switches
Be involved in the implementation of a SAN using B-series switch technology
Management of an B-series switch
Configuration of an B-series switch
Prerequisites
There are no prerequisites for this training. It is recommended that the learner have an understanding of
Fibre Channel and Zoning concepts.
Course Objectives
Upon successful completion of this course, participants should be able to:
List the various B-series switch models
Describe the design of the B-series switches
Contrast ED switch architecture versus DS switch architecture
Identify the tools available for managing/configuring the B-series switches
Discuss the available security features and how to implement them
Modules Covered
These modules are designed to support the course objectives. The following modules are included
in this course:
Common Switch Architecture
Model Hierarchy and Functionality
Native Switch Tools
Optional Switch Tools
Switch Security
Labs
There are no labs included in this course.
Assessments
This course includes a post assessment quiz, to be conducted on-line via KnowledgeLink, EMCs
Learning Management System.
Copyright 2004 EMC Corporation. All Rights Reserved.
Connectrix B-Series Architecture and Management, 1
Connectrix B-Series Architecture
and Management
Welcome to Connectrix B-Series Architecture and Management.
2
Connectrix B-Series Architecture and
Management
After completing this course, you will be able to:
List the various B-series switch models
Describe the design of the B-series switches
Contrast ED switch architecture versus DS switch
architecture
Identify the tools available for managing/configuring
the B-series switches
Discuss the available security features and how to
implement them
The objectives for this course are shown here. Please take a moment to read them.
3
Connectrix B-Series Switches
Connectrix DS-8B Connectrix DS-16B2
Connectrix DS-16B Connectrix DS-32B2
Model Naming Convention:
DS = Departmental Switch
8, 16, 32 = maximum number of Fibre Channel ports on the switch
B = 1 Gb/sec. capable (B-Series OEM)
B2 = 1 or 2 Gb/sec. capable (B-Series OEM)
Current Models, at the time of this writing, include the Connectrix DS-16B2 and Connectrix DS-32B2.
4
Connectrix B-Series Directors
Connectrix
ED-12000B
The ED-12000B is a 128-port director-class switch that provides dynamic switched connections between Fibre
Channel servers and devices in a Storage Area Network (SAN) environment.
Multiple directors and the Service Processor (laptop attached to the front door of the Connectrix cabinet)
communicate on a LAN through one or more 10/100Base-T Ethernet hubs. Up to four hubs can be connected
together as more directors are installed on a customer network.
The directors Fibre Channel technology provides high-performance scalable bandwidth (2 Gb/s), highly
available operation, redundant switched data paths, long transmission distances (up to 20 km), and high device
population.
B-Series Management Tools
6
Management Ports
Connectrix DS-16B2
Maintenance Port
Ethernet Port
There are two ports used to configure and manage B-Series Switches.
The Maintenance Port is used to configure the IP address for the Ethernet port.
The Ethernet Port, once configured, is used to manage all aspects of the switch.
7
Setting the IP Address Using HyperTerminal
Maintenance Port
(9 pin RS232)
RS232
Serial
Maintenance
Cable
The process of setting the IP address through HyperTerminal will be discussed at length later in the module.
8
Telnet Features
IP LAN
The CLI can be used only through a Telnet Client session in an out-of-band management environment, using
the Ethernet port in the switch. Although the primary use of the CLI is in host-based scripting environments,
the CLI commands can also be entered directly at a command line. Any hardware platform that supports the
Telnet Client software can be used.
We will discuss this in more detail later.
9
WebTools
IP LAN
Web Tools provides a graphical interface that allows you to monitor and manage entire fabrics and individual
switches and ports from a standard workstation. All switches in the fabric are displayed in the main window of
Web Tools, including switches that do not have a Web Tools license. However, only switches that have a Web
Tools license installed can be managed through Web Tools. Other switches must be managed through Telnet.
We will discuss this in more detail later.
10
Connectrix B Fabric Manager
Connectrix B Fabric Manager Manager (hereafter called Fabric Manager) is a powerful application that
manages multiple EMC Connectrix B-Series and B-Series SilkWorm switches and fabrics in real time. In
particular, Fabric Manager provides the essential functions for efficiently configuring, monitoring,
dynamically provisioning, and managing SAN fabrics comprised of EMC Connectrix B-Series switches as
well as switches from the B-Series Silkworm family.
Through its single-point SAN management platform, Fabric Manager facilitates the global integration and
execution of management tasks across multiple fabrics-thereby lowering the overall cost of SAN ownership.
As a result, it provides a flexible and powerful tool optimized to provide organizations with rapid access to
critical SAN information.
In addition, Fabric Manager is tightly integrated with other EMC Connectrix B-Series and B-Series SAN
management products, such as Web Tools and Fabric Watch. Organizations can also use Fabric Manager in
conjunction with SAN and storage resource management applications as the drill-down element manager for a
single or multiple fabrics.
We will explore Fabric Manager in detail later.
B-Series Features and Functions
12
Departmental Switch Features and Functions
Buffering
High Bandwidth
Low Latency
Low Communication
Overhead
Multiple Topology Support
Multiple Classes of Service
ASIC
The B-Series departmental switches offer several key features:
Buffering is based on non-blocking algorithms to avoid head of queue blocking and destination port buffering
bandwidth issues
High bandwidth - Each port provides full-duplex serial data transfer at a rate of up to 2.125 Gbps, and a
sustained frame processing rate of 53 million per second
Low latency - The latency is less than two microseconds between transmission of a frame at a source port to
receipt of a frame at the corresponding destination port (with no port contention).
Low communication overhead - Fibre Channel protocol provides efficient use of transmission bandwidth,
reduces interlocked handshakes across the communication interface, and efficiently implements low-level error
recovery mechanisms. This results in little communication overhead in the protocol and a switch bit error rate
(BER) better than one bit error per trillion (10
-12
) bits.
Multiple topology support - The switches support F_Port, FL_Port and E_Port.
Multiple Class of Service Support - The switches support Class 2, Class 3, and Class F.
Several methods of switch management are offered.
ASIC we will discuss this on the next slide
13
ASIC
Application Specific
Integrated Circuit
Memory local on ASIC
Communicates with message
bus for frame transmission
Non-Blocking
The term ASIC means Application-Specific Integrated Circuit. Within Fibre Channel switch concepts, the ASIC
is generally referred to the chip that manages a certain number of external FC ports. The BLOOM ASIC
manages two Quads of four independent ports each.
B-Series switches manage port to port traffic using an internal messaging BUS. There is at least enough memory
for 108 buffers per Quad, approximately 216k ram per ASIC.
14
SFP
Small Form Pluggable
LC connector
Short-wave
Long-wave
The LC technology was first introduced in 1996 by Lucent technology (hence the name LC Lucent connector).
It fits into a standard RJ-45 telephone-style jack and is about half the size of older SC connectors. The small
size allows construction of transceivers half the size of their predecessors, effectively doubling the number of
fiber-optic ports that can be connected to a system. The interface also improves optical performance by keeping
signal strength up. LC connectors support both single and multimode fiber, used primarily in LAN and WAN
environments. All vendors are producing LC-based transceivers for the Fibre Channel protocol commonly used
in SANs. Gigabit Ethernet allows for only one-fifth the power loss of Fast Ethernet. Cables using the LC
connector averages only 0.1dB power loss.
EMC offers adapter cables with an SC connector on one end and an LC connector on the other. Barrel
connectors for the SC end allow you to extend the existing fiber cable.
15
Hardware Components
Non-Replaceable Components:
Control Processor (CP) embedded in motherboard
Field Replaceable Components (FRUs)
SFP Optics
Fans
Power Supplies
The B-Series DS switches provide a modular design that enables fast replacement of field-replaceable units
(FRUs). FRUs accessed from the front include small form-factor pluggable (SFP) LC transceivers.
FRUs accessed from the rear include power supplies and cooling fan modules.
16
DS-8/16B
8 or 16 optical ports (1gb)
LEDs to indicate port status
Switch power & status LEDs on
the front panel
Front Display Panel (DS-16B
only)
One serial port with an RS-232
connector (DS-8B only)
One 10/100 Mb per second
Ethernet port with an RJ-45
connector
Two power supplies, with LEDs
Two fan units (single module)
The Connectrix DS-8/16B is an 8 or 16-port Fibre Channel switch that provides high-performance connections
between computers, storage devices, and other peripherals in an Open Systems Fibre Channel switched network.
The DS-8/16B can transfer data at up to 1.0625 Gb/s (gigabits per second) through each port at distances up to
500 meters using 50/125-micron multimode fiber-optic cable with shortwave laser transceivers and up to 20
kilometers using 9/125 micron single-mode fiber-optic cable with longwave laser transceivers.
The DS-8/16B is managed and installed in an EMC CLARiiON cabinet or customer system with an Internet
connection to the WebTools interface installed on the switch.
17
DS-8/16B2
8 or 16 optical ports (2gb)
Two LEDs to indicate port status
and link speed
Switch power LED on the front
panel
Switch status LED on the back
panel
connector
connector
Two power supplies, with built in
fans and LEDs
Two fan trays, individually hot
swappable
The Connectrix DS-8/16B2 is an 8 or 16-port Fibre Channel switch that provides high-performance connections
The DS-8/16B2 can transfer data at up to 2.125 Gb/s (gigabits per second) through each port at distances up to
300 meters (500 meters if the port is set to 1 Gb/s) using 50/125-micron multimode fiber-optic cable with
shortwave laser transceivers and up to 20 kilometers using 9/125 micron single-mode fiber-optic cable with
longwave laser transceivers.
The DS-8/16B2 is managed and installed in an EMC CLARiiON cabinet or customer system with an Internet
18
DS-32B2
32 optical ports (2gb)
Two LEDs to indicate port status
and link speed
Switch power LED on the front
panel
Switch status LED on the back
panel
connector
connector
Two power supplies, with built in
fans and LEDs
Two fan trays, individually hot
swappable
The Connectrix DS-32B2 is a 32-port Fibre Channel switch that provides high-performance connections
The DS-32B2 can transfer data at up to 2.125 Gb/s (gigabits per second) through each port at distances up to 300
meters (500 meters if the port is set to 1 Gb/s) using 50/125-micron multimode fiber-optic cable with shortwave
laser transceivers and up to 20 kilometers using 9/125 micron single-mode fiber-optic cable with longwave laser
transceivers.
The DS-32B2 is managed and installed in an EMC CLARiiON cabinet or customer system with an Internet
19
Director Switch Features and Functions
Buffering
High Bandwidth
Low Latency
Low Communication
Overhead
Multiple Topology Support
The B-Series Director switches offer several key features:
Buffering is based on non-blocking algorithms to avoid head of queue blocking and destination port buffering
bandwidth issues
High bandwidth - Each port provides full-duplex serial data transfer at a rate of up to 2.125 Gbps. Sustained
frame processing rate of 53 million per second
Low latency - The latency is less than two microseconds between transmission of a frame at a source port to
receipt of a frame at the corresponding destination port (with no port contention).
Low communication overhead - Fibre Channel protocol provides efficient use of transmission bandwidth,
reduces interlocked handshakes across the communication interface, and efficiently implements low-level error
recovery mechanisms. This results in little communication overhead in the protocol and a switch bit error rate
(BER) better than one bit error per trillion (10
-12
) bits.
Multiple topology support - The switches support point-to-point and multiswitch connectivity.
20
More Features and Functions
N+1 component redundancy
Multiple Classes of Service
BLOOM ASIC
Non-blocking
Reset/IML
More B-Series features and functions include:
N+1 component redundancy vital components are provided with at least one redundant part to reduce the
likelihood of downtime due to failure
Multiple Class of Service Support - The switches support Class 2, Class 3, and Class F.
The BLOOM ASIC design enhances internal frame transmission and reduces head of line blocking.
Reset/IML a single button reset of IML is provided for ease of reconfiguring the switch
21
Hardware Components
Field Replaceable Units (FRUs)
Control Processor (CP)
Port Cards
SFP Optics
Fans
Power Supplies
The B-Series Director switches provide a modular design that enables fast replacement of field-replaceable units
(FRUs). Although all components are hot-replaceable, it is recommended that you perform a manual failover
and always replace the inactive card.
22
Control Processor Cards
Dual CP
Active/Standby mode
Hot Plug Replacement
Each CP has an IP address
CP0 in slot 5; CP1 in slot6
Hot Standby
High availability is provided through redundant CP cards: a standby CP card and an active CP card. The
active CP card is the one currently controlling the Director. The standby CP card allows for rapid failover.
Failover occurs automatically as soon as the active CP card fails or is removed from the chassis. Slots 5 and 6
house the two Control Processor Boards.
Information about both CP cards, including which card is active, is available through the haShow command. The
majority of commands must be entered through the active CP card, except for the haShow command, which can
be entered through either CP card, and the firmwareDownload command, which must be entered through the
standby CP card. Each CP card provides the following ports:
Modem serial port The modem serial port has a DB-9 connector wired as a DTE device, and can be used
to connect to a DCE device. Not used by EMC.
Terminal serial port (also known as a console port) The terminal serial port has a DB-9 connector wired
as a DCE device and can be used to connect to a DTE device, such as a PC serial port or dumb terminal.
Ethernet port The ethernet port has an RJ-45 connector, and is capable of speeds of 10/100 Mbps. The
switch can continue to operate while a CP card is replaced.
The active CP card controls the following services:
System initialization; Extended Fabrics; High availability and switch drivers; Fabric Watch; Name server;
Remote Switch; SNMP; SES; Fabric OS; Web Tools; Fabric Access; Zoning
23
16-Port Cards
Ports
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
Ejector
Quads
3
2
1
0
Ejector
Each 16-port card provides 16 auto-sensing fibre channel ports, capable of speeds of 1 and 2 Gbps.
The ports on each of the 16-port cards are color-coded to indicate which ports can be used in the same ISL
Trunking group: four ports marked with black solid ovals alternate with four ports marked with oval outlines.
The switch can continue to operate while a port card is replaced, but any devices connected to the port card must
be disconnected. To ensure correct air circulation inside the switch and protection from dust, filler panels can be
ordered for any empty slots. The 16-port card slots should not be left empty for an extended period of time.
Note
ISL Trunking is a feature that enables distribution of traffic over the combined bandwidth of up to four
ISLs between two directly connected switches, while preserving in-order delivery.
24
LEDs on the Power Supplies
Power Supply
Up to (4) power supplies
Connected to (2) circuits
Designed to fully function
with just two power supplies
Locking Tab
Handle
Power LED
Predictive
Failure LED
Fail LED
The ED-12000B uses up to four power supplies; however, only two power supplies are required to power a
completely loaded chassis. Two power supplies receive power from one of the AC power inputs, and the other
two receive power from the other AC power input; each AC power input and its associated power supplies are
marked. The switch will continue to operate, provided at least one power supply remains in either one of the
two upper power supply slots (numbered 3 and 4), and one in either of the two lower power supply slots
(numbered 1 and 2), and both AC switches are on and powered.
The power status is also displayed on the WWN bezel on the non-cable side of the switch. A fully loaded chassis
can function with just two power supplies. The left power input connector provides power to the power supplies
in slots 1 and 3, and the right power input connector provides power to the power supplies in slots 2 and 4.
Removal and Installation
Power supply removal
Press on the release tab located on upper left corner;
Pull down the release handle;
Grab the release handle and pull
Power supply installation
Orient the power supply so that the tab is towards the front of the switch;
Insert the power supply all the way into the slot and push the handle until it clicks;
Verify status. If the circuit is powered, a green status appears indicating it is online.
25
WWN Card and Bezel
16-Port Cards in Slots 1-4 and 7-10
Power LED (Upper LED)
Status LED (Lower LED)
CP Cards in Slots 5-6
Power LED (Upper LED)
Status LED (Lower LED)
Power LED for
Power Supplies 1-4
The WWN card and bezel are located at the top of the back of the chassis. The WWN card is an integral part of the chassis,
and the bezel covers the card. The WWN card provides LEDs for monitoring the switch from the back side of the chassis. It
also stores the following information:
The chassis serial number (used when installing software licenses)
The two native IP addresses assigned to the CP card slots
Information for the two logical switches in the ED-12000B:
The logical switch names
The logical IP addresses
The logical WWNs
The bezel protects the card and provides visual orientation for the LEDs on the card. Together, the bezel and card provide a
consolidated view of the LEDs for the CP cards, 16-port cards, and power supplies.
The bezel and card provides a consolidated LED view of:
The incoming power to and status of the 16-port cards in slots 1-4 and 7-10;
The incoming power to and status of the CP cards in slots 5 and 6;
The incoming power to the four power supplies.
If a slot contains a filler panel, the corresponding LEDs on the WWN card are dark. Both the WWN card and bezel are hot-
swappable, since the information stored in the card is also stored in the flash memory of the CP cards. However, the switch
should not be rebooted while the WWN card is uninstalled, as this can cause the switch to boot incorrectly.
Note
The LEDs patterns may temporarily change during POST and other diagnostic tests.
26
Blower Assemblies
Supports (3) blowers
Blower Indicators
Blower Power
Blower Status
Blower Status
sw0_155:admin> fanshow
Fan #1 is absent
Fan #2 is OK, speed is 3013 RPM
Fan #3 is OK, speed is 3183 RPM
Blower Assembly
Power LED
Blower Assembly
Fault LED
There are three hot swappable blower assemblies and only two are required for cooling. Air enters from the non-
cable side of the chassis and exits at the top front of cable side. The ED-12000B hardware is designed to
withstand the loss of a single blower assembly and continue to run. All slots must either have a 16-port card
installed or a filler panel installed for efficient cooling.
If there is a blower assembly that is not functioning then it is marked as faulty. There are status LEDs on the
blower assembly. A green LED indicates that power is good. An amber LED indicates that the blower assembly
needs attention. When there are less than three blower assemblies operating, the fan RPM increases to
compensate for the reduced airflow. The nominal RPM range is 2001-3400. Speeds above or below this range
result in a warning being issued. The vents require a minimum of 2" airspace. The blower assemblies are hot-
swappable, if two blowers remain operating at all times. If more than one blower must be removed at the same
time, the switch should be turned off first.
Removal
Loosen the thumbscrews on the top and bottom of the assembly.
Press on the top of the blower assembly handle until the handle pops open.
Grab the handle and pull. The blower assembly should easily slide out
Install
Line up the power plug on the blower assembly to the power socket in backplane.
Slide it in until it connects, and a power LED light should come on.
Push the top of the handle into the recess.
Tighten the thumbscrews to lock it in place
EC-1230B Cabinet
28
EC-1230B Enterprise Cabinet System
Mounting for up to two ED-12000B
Redundant power distribution
Cable Management system
10BaseT Ethernet hub for
management connectivity (in back)
Power is provided by the power distribution system within the EC-1230B Connectrix equipment cabinet.
When the EC-1230B cabinet is positioned such that it is not immediately adjacent (within 2 inches) to another
equipment cabinet, the stabilizer/outrigger brackets must be positioned in their outermost position.
29
Facility Requirements
To ensure correct operation of the ED-12000B, the facility where the switch is used must meet the following
requirements:
Power is provided by the power distribution system within the EC-1230B Connectrix equipment cabinet.
An air flow of at least 350 cubic feet per minute, available in the immediate vicinity of the ED-
12000B/EC-1230B.
There must be two 30-Amp dedicated circuits, 208 to 264 VAC, 47 to 63 Hz. Each PDU within the EC-
1230B should be connected to its own power circuit independent of the other PDUs power circuit.
30
LAN Configuration
SNMP
Management
Station
Connectrix B
Fabric Manager
Customer
Data Network
(Public or Private)
WebTools
The number of cabinets managed is not specified. Each management software product has a maximum number
of switches that it can manage and this is the determining factor. LAN configuration and IP addressing must be
established from the customer prior to installation. EMC recommends that the SAN management LAN be
segmented from the general customer LAN for security purposes.
31
EC-1230B Cabinet Configurations
256 ports
128 128
128 128
128 ports
128 128
160 ports
128 128
8 8
8 8
16 16
The EC-1230B contains 36 units (36u) of available rack-mount height. With this in mind, here are some
guidelines for installation:
Install the heaviest units from the bottom up.
DS-xxBs should be installed to allow 3u of rack height per switch. 3u per switch is necessary to facilitate
organizing and managing the fiber-optic cables in the cabinet and to maintain proper airflow and
ventilation.
DS-xxBs can also be mounted in the CLARiiON product cabinet or a standard EIA cabinet.
Using HyperTerminal
33
Setting the IP Address Using HyperTerminal
Maintenance Port
(9 pin RS232)
RS232
Serial
Maintenance
Cable
Connect the serial cable to the serial port on the switch and to an RS-232 serial port on the workstation. If the
serial port on the workstation is RJ-45 instead of RS-232, you can remove the adapter on the end of the serial
cable and insert the exposed RJ-45 connector into the RJ-45 serial port on the workstation.
The serial cable must be a pass through serial Cable. Null Modem cables will establish a connection but will not
transfer data or commands. It is recommended that you use your laptop to configure these switches through
HyperTerminal.
34
Creating a Connection
Open the HyperTerminal application, located in
Start Programs Accessories Communications
You will be prompted to name this connection.
Ensure that the Connect using field displays COM1 or COM2 (depending on the serial communication port
connection to the switch). Be aware of other devices or applications using the COM port for serial
communications.
35
Port Properties
Create a connection establishing Telnet session options using the HyperTerminal interface. Set the connection
properties to the following:
9600 bits per second
8 Data bits
None for Parity
1 Stop Bit
None for Flow control
36
Set the IP Address
Fabric OS (cp0)
cp0 console login: admin
Password:
Enter Switch Number to Login <0 or 1>: 0
swd77:admin> ipaddrset
Switch number [0 for switch0, 1 for switch1, 2 for
CP0, 3 for CP1]: 0
Ethernet IP Address [10.127.37.4]:
Ethernet Subnetmask [255.0.0.0]:
Fibre Channel IP Address [0.0.0.0]:
Fibre Channel Subnetmask [0.0.0.0]:
Committing configuration...Done.
swd77:admin>
Change the IP address, subnet mask, and gateway address as directed by the customers network administrator.
Verify the address was correctly set by entering the ipAddrShow command at the prompt. Record the IP address
on the label provided for this purpose on the port side of the switch. Once the IP address is verified as correct,
log off of the serial console, remove the serial cable, and replace the shipping plug in the serial port.
Telnet
38
Telnet Features
IP LAN
The B2 series can support 2 simultaneous Telnet sessions at a time. The older B series only supports a single
connection.
If the Ethernet cable is disconnected from the switch during a Telnet session:
Replace the Ethernet cable before the client connection times out, and the Telnet session will continue.
Wait 15 minutes for the client connection times out; then replace the Ethernet cable and restart the
connection.
39
Telnet Functions
The functions of the Telnet
CLI are:
Automating management of
large numbers of switches
Writing and using scripts
Administrators should be
knowledgeable in:
Networking, SAN, and zoning
concepts
Connectrix products in the
users network
Scripts
The Command Line Interface (CLI) provides an alternative to the WebTools for switch management
capabilities. The primary purpose of the CLI is for scripts written for use in a host-based scripting environment.
The CLI can be used only through a Telnet client session in an out-of-band management environment, using the
Ethernet port in the switch. Although the primary use of the CLI is in host-based scripting environments, the
CLI commands can also be entered directly at a command line. Any hardware platform that supports Telnet
client software can be used.
The primary purpose of the CLI is to automate management of a large number of switches/directors with the use
of scripts. Because the CLI is not an interactive interface, no prompts are displayed to guide the user through a
task. If an interactive interface is needed, use WebTools.
CAUTION
If you choose to configure the switchs network IP address to be on a public network, this could allow someone
to attempt to manage the switch via the CLI. If you intend to use the CLI, be sure to change the default
passwords of the CLI.
40
Fabric OS (NY_DS32B_2)
NY_DS32B_2 login: admin
Password:
NY_DS32B_2:admin> passwd
Changing password for admin
Enter old password:
Enter new password:
You cannot reuse the old password.
Enter new password:
Password must be between 8 and 40 characters long.
Enter new password:
Password unchanged.
passwd: Authentication token manipulation error
NY_DS32B_2:admin>
Passwords
There are four accounts for each switch instance.
At each account level, you can change passwords for that account and all accounts that have lesser privileges.
There is one exception to the password structure; an admin level user can change the root password by entering
the passwd root. They must also know the old root password.
There are four levels of account access:
root - not recommended
factory - not recommended
admin- recommended for administrative operations
user - recommended for non-administrative operations
Therefore, if you are logged in as admin, you can change the passwords for both admin and user.
41
Telnet Commands
help - shows all commands
w/info
licenseHelp license commands
diagHelp diagnostic commands
switchDisable - OFFLINE
switchEnable - ONLINE
configure change switch
parameters
diagShow POST results - last
bootup
routehelp routing commands
switchShow - display switch info
supportShow full detailed
switch info
portShow # - displays port info
nsShow NameServer contents
nsAllShow NS for full fabric
fabricShow fabric information
version firmware code revision
reboot full reboot w/POST
fastboot reboot w/o POST
The listed commands are among the most commonly used commands.
After logging into a telnet session, type help to get a full list of all commands available to be run. At the
bottom of the list of available commands are submenus of other commands that can be run as well (ex.
Quickloop commands under qlhelp and Zoning commands under zonehelp).
The telnet is a fully functional component and is the preferred method of management on B-Series switches.
42
Command Tree
NY_DS32B2:admin> help
agtcfgDefault Reset SNMP agent to factory default
agtcfgSet Set SNMP agent configuration
agtcfgShow Print SNMP agent configuration
aliasDelete Deletes a port from all local groups
aliasJoin Creates a group of N-Ports
aliasPurge Removes an alias group
aliasShow Print Alias Server information
bcastShow display broadcast routing information
chassisName display or set the chassis's Name
chassisShow display Field Replaceable Units(FRU)
configDefault Reset config to factory default
configDownload Load switch config from a server
configShow Print switch config parameters
configUpload Save switch config to a server
configure Set switch config parameters
date Print/set the system date and time
diagHelp Print diagnostic help info
dlsReset turn off dynamic load sharing option
dlsSet turn on dynamic load sharing option
Type <CR> or <SPACE BAR> to continue, <q> to stop
CLI commands can be entered directly at the command line of a terminal or coded in a script. Unlike the
standard B-Series switches, EMC Connectrix B-Series CLI commands are not case sensitive.
43
NY_DS32B_2 :admin> switchshow
switchName: DS32B2_1
switchType: 10.1
switchState: Online
switchRole: Subordinate
switchDomain: 4
switchId: fffc04
switchWwn: 10:00:00:60:69:00:54:e9
switchBeacon: OFF
blade7 Beacon: OFF
blade9 Beacon: OFF
Area Slot Port Gbic Speed State
=====================================
0 7 0 id N2 No_Light
1 7 1 id N2 Online L-Port 8 public
2 7 2 id N2 Online F-Port 10:00:00:00:c9:27:2e:9b
3 7 3 id N2 No_Light
4 7 4 id N2 Online E-Port (Trunk port, master is Slot 7 Port 5)
5 7 5 id N2 Online E-Port 10:00:00:60:69:00:54:e8 "DS32B2_2" (upstream)
(Trunk master)
6 7 6 id N2 Online E-Port (Trunk port, master is Slot 7 Port
7 7 7 id N2 Online E-Port (Trunk port, master is Slot 7 Port
switchshow
Use this command to display switch and port status information. Information may vary by switch model. Below
is the information provided. The first section provides switch summary information; it is followed by a section
covering summary information by port.
Switch summary information:
switchName Displays the switch symbolic name.
switchType Displays the switch model number:
9 = DS-16B2
10.1 = ED-12000B
12.1 = DS-32B2
16.2 = DS-8B2
switchState Displays the switch state: online, offline, testing, faulty.
switchRole Displays the switch role: principal, subordinate, disabled.
switchDomain Displays the switch domain ID: 0-31 or 1-239.
switchId Displays the switch embedded port D_ID.
switchWwn Displays the switch Worldwide Name.
switchBeacon Displays the switch's beaconing state (either ON or OFF).
blade_n:Beacon Displays the blades beaconing state (either ON or OFF).
44
NY_DS32B_2 :admin>cfgshow
Defined configuration:
cfg: mike_config
nt8_hba1; nt8_hba2
zone: nt8_hba1
10:00:00:00:c9:29:20:bb; 50:06:01:60:11:02:81:ab;
50:06:04:82:c3:3e:c8:8f; 11:11:11:11:11:11:11:11
zone: nt8_hba2
10:00:00:00:c9:2a:b6:f9; 50:06:04:82:c3:3e:c8:ae;
50:06:04:82:c3:3e:c8:af; 11:11:11:11:11:11:11:11
alias: trbl_nt8_hba1
20:00:00:00:c9:29:20:bb
alias: trbl_nt8_hba2
10:00:00:00:c9:2a:b6:f9
Effective configuration:
cfg: mike_config
zone: nt8_hba1
10:00:00:00:c9:29:20:bb
50:06:01:60:11:02:81:ab
50:06:04:82:c3:3e:c8:8f
11:11:11:11:11:11:11:11
zone: nt8_hba2
cfgshow
Use this command to display zone configuration information. If no operand is specified, all zone configuration
information (both Defined and Effective) is displayed.
If an operand is specified, it is used as a pattern to match zone configuration names in the defined configuration;
those that match the pattern are displayed.
The Defined configuration is the complete set of all zone objects that have been defined in the fabric. There can
be multiple zone configurations defined, but only one can be enabled at a time. There may be inconsistencies in
the definitions, zones or aliases that are referenced but not defined, or there may be duplicate members. The
Defined configuration is the current state of the administrator input.
The Effective configuration is the single zone configuration that is currently enabled. The devices that an
initiator sees in the fabric are based on this configuration. The Effective configuration is built when a specific
zone configuration is enabled and all error checking has been completed successfully.
45
NY_DS32B_2 :admin> switchdisable
NY_DS32B_2 :admin> configure
Configure...
Fabric parameters (yes, y, no, n): [no] y
Domain: (1..239) [97]
R_A_TOV: (4000..120000) [10000]
E_D_TOV: (1000..5000) [2000]
Data field size: (256..2112) [2112]
Sequence Level Switching: (0..1) [0]
Disable Device Probing: (0..1) [0]
Suppress Class F Traffic: (0..1) [0]
VC Encoded Address Mode: (0..1) [0]
Per-frame Route Priority: (0..1) [0]
Long Distance Fabric: (0..1) [0]
BB credit: (1..27) [16]
Virtual Channel parameters (yes, y, no, n): [no] n
Zoning Operation parameters (yes, y, no, n): [no] n
RSCN Transmission Mode (yes, y, no, n): [no] n
Arbitrated Loop parameters (yes, y, no, n): [no] n
configure
Use this command to change the following system configuration parameters:
Fabric Parameters
Virtual Channel Settings
Zoning Operation Parameters
Rscn Transmission Mode
NS Pre-zoning Mode
Arbitrated Loop Parameters
System Services
Portlog Events Enable
This command must be run in offline mode. You must run the switchdisable command first.
46
NY_DS32B_2:admin> configupload
Server Name or IP Address [host]: 10.127.37.55
User Name [user]: administrator
File Name [config.txt]:
Password:
Upload complete
configupload
Use this command to upload the switch configuration to a host file. The upload process uses FTP. If the
command is entered without operands, it becomes interactive and prompts the user for input.
The configuration file is written as three sections.
The first section contains the switch boot parameters. It has variables such as the switch's name and IP
address. This section corresponds to the first few lines of output of the configshow command.
The second section contains general switch configuration variables, such as diagnostic settings, fabric
configuration settings, and SNMP settings. This section corresponds to the output of the configshow
command (after the first few lines), although there are more lines uploaded than shown by the command.
The third sections contains the zoning configuration parameters.
47
NY_DS32B_2:admin> configdownload
configDownload:This command may not be executed on an operational switch.
You must first disable the switch using the "switchDisable" command.
NY_DS32B_2:admin> switchdisable
NY_DS32B_2:admin> configdownload
Server Name or IP Address [host]: 10.127.37.55
User Name [user]: administrator
File Name [config.txt]:
Password:
*** CAUTION ***
This command is used to download a backed-up configuration
for a specific switch. If using a file from a different
switch, this file's configuration settings will override
any current switch settings. Downloading a configuration
file, which was uploaded from a different type of switch,
may cause this switch to fail.
Do you want to continue [y/n]: y
fwConfigReload: Fabric Watch configuration reloaded.
download complete
NY_DS32B_2:admin> switchenable
configdownload
The download process is additive; that is, the lines read from the file are added to the current switch
configuration. This enables you to change a single configuration variable by downloading a file with a single
line. All other variables remain unchanged. This is particularly important when downloading a zoning
configuration. Since the new zoning information is added to the current configuration, there may not be any
conflicts. Typically this command is used to add a consistent change to the current zoning configuration, or to
replace the current zoning configuration, in which case cfgclear must be invoked before configdownload.
WebTools
49
WebTools
IP LAN
Advantages of Web Tools
Web Tools is an excellent partner to the traditional Telnet commands, and in many ways can provide faster and
more effective results than can be achieved strictly through a command line interface.
Below are some of the features that make Web Tools an important part of the switch management and
administration process:
You can use Web Tools from a standard workstation which provides you the advantage of being virtually
in front of any fabric, switch, or port.
Web Tools makes zoning a simple click-and-drag process, rather than forcing you to type IP addresses or
port numbers into a configuration.
Web Tools provides the Advanced Performance Monitoring feature. This feature allows you to view the
status and traffic of a switch or port in seconds by easily creating a variety of effective graphs.
Web Tools is easy and intuitive to use.
50
Switch Explorer
Fabric Events
Fabric
Topology
Name Server
Zone Admin
Summary View
= expands /
contracts info
under switch
box
Status Legend
indicating
switch status
The Switch Explorer View (shown in Figure 1-1) is the first window that appears when you start Web Tools
from a web browser. In the browser, enter the switch name or IP address in the Location/Address field and press
ENTER. The Switch Explorer View includes:
Fabric Display Selector Allows you to change the way switches are displayed in the fabric tree.
Select from Name, IP, or WWN.
Fabric Tree Display a navigation menu (down the left side of the window) of icons for all switches in
the fabric. The refresh rate for the fabric tree is approximately 30 seconds.
Fabric Toolbar Provide easy access to fabric-level administration tasks.
Switch Graphic Displays a physical representation of the switch and real-time status information.
Switch Management Toolbar Enables you to access various configuration and monitoring functions
for the switch.
Switch Information Displays a synopsis of useful information about the selected switch.
Zone Configuration Displays the name of the currently enabled Zone configuration. If no Zone
configuration is enabled, this field displays none.
Status Legend Explains the meaning of colors that appear in the background of various icons:
Green = Healthy
Yellow = Marginal (mix of good and faulty readings)
Red = Critical (more than two faulty readings)
Gray = Unknown or unmonitored
51
Fabric Topology
List of all
switches in the
fabric
Indicates which
is the principal
switch and
which ones are
the subordinates
Includes WWNs
of switches
In multiple-
switch fabrics,
ISL ports & hop
count will be
shown for the
switches in the
fabric
The Fabric Topology View summarizes the physical configuration of the fabric from the perspective of the local
domain (the domain of the switch entered as a URL in the web browser). This includes information about the
destination domains (all other domains in the fabric) and the paths between each destination domain and the
local domain.
To access the Fabric Topology View, click the Fabric Topology icon at the lower left of the Switch Explorer
View.
52
Main Window Switch Icons
Each switch in
the fabric will
have its own
box & icons
Faceplate Icon
= Faceplate
Window
Bell Icon =
Switch Events
(log)
Key Folder
Icon = Admin
Window
Computer Icon
= Telnet
session
The switch graphic is an interactive real-time display of the selected switch. The displays in Web Tools
correspond to the front of the actual switch and its various switch and port LEDs. Clicking a port displays a Port
Information window, which contains functional and statistical information.
The Switch Explorer View is the first window that appears when you start Web Tools from a web browser. In
the browser, enter the switch name or IP address in the Location/Address field and press ENTER.
The buttons below the switch graphic enable you to access various configuration and monitoring functions for
the switch. Some of these buttons may require a license key to activate. If the appropriate license key is not
installed for a particular feature that button might not appear.
53
Port View
The Port Information View displays
statistics for the selected port. This
information is automatically updated
when the view is opened, and is also
refreshed periodically while the view
remains open.
54
Performance View
The Performance Monitor graphically displays throughput (megabytes per second) for each port and for the
entire switch. Port throughput is the number of bytes received at a port plus the number of bytes transmitted.
Switch throughput is the total throughput for all the ports.
Performance Monitoring also allows the graphing of traffic based on the Source ID and the Destination ID
hardware filtering mechanism. Some features available in the Performance Monitor include:
An existing report can be selected from a list of reports that are predefined. In some cases, you can supply
the object to be monitored and graphed (such as port number, SID/DID pair, AL_PA, or switch domain
number).
Graphs are displayed on a canvas, which can hold a maximum of eight graphs simultaneously. An
individual graph can be maximized to occupy the entire canvas. The size of the graphs on the canvas is
determined by the number of graphs being displayed. The window does not need to be scrolled to view all
the selected graphs.
The collection of graphs in the canvas can be stored for later retrieval on the switch. Up to 20 individual
canvases can be saved. Each canvas is saved with its name, a brief description, and up to 8 graphs which
comprise the canvas.
Any graph can Zoom In, or be magnified and detached from the main canvas or Removed from the
main canvas using a pop-up menu. The pop-up menu is displayed by pointing the mouse at any graph on
the main canvas and clicking the right mouse button. To re-attach the (Zoomed Out) detached graph back
to the main canvas, you can point the mouse to the detached graph, click the right button and select
Zoom Out.
Each graph can be printed to a printer of choice.
55
Switch Admin
Switch
configuration
Licensed
features
interface
E/OS updates
The Administrative Interface provides access to the administrative functions through the following tabs:
Switch Information
Network Config
Upload/Download
SNMP Admin
License Admin
Port Settings
Configure
Routing
Extended Fabric
Trunk Information
The Administrative Interface requires administrative privileges. Once an administrative login is entered,
administrative privileges remain available until the Web browser is exited.
56
Switch Information
Use the Switch Information tab to manage basic switch setup for items such as switch name, switch domain ID
and enabling and disabling the switch.
57
Network Config
Use the Network Config tab of
the Administrative interface to
manage the IP networking
functionality of the switch.
The Syslog IP represents the IP
address of the server that is
running the syslog process. The
Syslog daemon reads and
forwards system messages to the
appropriate log files and/or users,
depending on the system
configuration. When one or more
IP addresses are configured, the
switch forwards all error log
entries to the syslog on the
specified server(s). Up to six
servers are supported.
58
Upload/Download
Use the Upload/Download tab of the Switch Admin window to complete tasks such as:
Download firmware
Upload a Configuration file to the host
Download the configuration from a host to the switch
Return the switch to the original configuration
Reboot a switch
You must provide host information for the entire upload and download tasks listed above.
59
Configupload report
Here is an example of the configuration information logged to a local file.
60
SNMP
Use the SNMP tab of the Administrative Interface to perform administration of the SNMP Subsystem. Use the
SNMP tab to specify the switch community string, location, trap level and trap recipients.
The SNMP tab is affected by the use of Secure Fabric OS; the ACL list will not be visible if security is enabled.
In order for the switches to send SNMP traps, you must first enter the CLI command snmpmibcapset. This
enables the MIBs on all switches to be monitored.
61
License Admin
Use the License Admin tab to install and
remove license keys that are provided to you.
Even if you have a license key for one of the
newer features on a switch (like Extended
Fabric or Fabric Watch) and the switch itself is
old, it may not have the capability to run the
newer feature.
Contact the Support Group for assistance on
upgrading the capabilities of your switch to
allow you to enter in the license key for the
newer functionality.
62
Port Setting
Use the Port Setting tab of the
Administrative interface to perform
functions such as: disable/enable
ports on the switch, set port speed to
1GB/2GB/Negotiate, enable/disable
Trunking, or name a port.
The port speed is displayed as
follows:
1G = 1 Gb/s
2G = 2 Gb/s
N1 = Negotiated 1 Gb/s
N2 = Negotiated 2 Gb/s
63
Configure
Use the Configure tab of the
Administrative Interface to
configure Fabric Parameters, Virtual
Channel parameters, Arbitrated
Loop parameters, and System
Services parameters.
Many parameter fields and
checkboxes within the Configure tab
can only be modified if the switch is
disabled. If a parameter value is
shaded gray, that value is currently
read-only. To modify the value, you
must first disable the switch.
Connectrix B Fabric Manager
65
Features
Grouping
ISL Monitoring
Sequenced Reboot
Firmware Download
License Management
Fabric Merge Checking
Grouping allows you to organize switches by function, switch type, firmware version, or any other criteria you
choose.
ISL Monitoring alerts you when an ISL is added, removed or moved in a fabric.
Sequenced Reboot allows you to define groups of switches to reboot simultaneously, then configure groups to
reboot sequentially. Reboot groups let you simultaneously reboot switches that run the same firmware, serve the
same function, reside in the same physical location, or share any other attribute by which you want to group
them.
Perform a Firmware Download with Fabric Manager to concurrently download firmware to multiple switches
and reboot the switches simultaneously or in sequence.
License Management can display, store, load, and reload your license keys so that you do not lose them if your
switch fails.
Perform a Fabric Merge Check to determine if you can merge two fabrics successfully.
66
Installing Fabric Manager
Installation of Fabric Manager consists of two portions: a server and a client. During installation, you may
choose to install either, or both, on a single host. Both a server and a client must be installed for Fabric Manager
to operate properly.
A client installation must have IP access to the switches, it is not a relay client.
67
Launching Fabric Manager
When launching Fabric Manager you must enter the following fields: Username, Password, Server and Port.
User names and passwords correspond to Windows domain user names and passwords. Server is either the IP
address of the server or the word localhost if the server component is on the same host as the client.
68
Main Window
The basic Fabric Manager GUI includes standard menus, tabs, and pull down menus, as well as custom icons
and windows.
The Address field keeps a drop-down list of the fabrics that you already discovered and lets you enter the IP
address of new switches and fabrics that you want to monitor.
The ID pull down menu lets you set the naming of your switches and fabrics.
The SAN Elements tab shows the various elements that you can monitor. You can control+click to select
multiple items. When an elements status changes, the element changes color in the SAN Elements tab. The
parent items in the SAN Elements tab tree change color to match.
The filter tab lets you view entities that include a particular alphanumeric string.
69
Licensing
Fabric Manager can display, store, load, and reload your license keys so that you do not lose them if your switch
fails.
70
Exporting and Importing Licenses
Also under the Tools > Licensing menu, the import and export license functions allow you to capture the
licensing from a switch and restore those licenses. You can export the license keys of multiple switches to a
single file. You can even export keys from different switches in different fabrics to one file. The file matches the
license keys to the WWN of the appropriate switch so you can quickly and easily import the keys at any time.
If you need to restore license keys to a switch, import the keys that you saved to an exported file.
71
Discover a Fabric
You must discover a fabric to add it to the SAN Elements tab and administer it with Fabric Manager.
Fabric Manager can scan a subnet to discover fabrics. You may enter a wildcard * in the last section of the IP
Address Range field to represent the range of scan. With this discovery mechanism, you do not need to know the
exact address of a switch to discover a fabric. When you specify a subnet, Fabric Manager lists the switches and
fabrics that it finds so you can add them to the SAN Elements tab.
The check-boxes in the Scan Result section adds these switches to the management station.
72
Managing Fabrics
Once you have discovered the switches and fabrics available, you can now manage those entities. You can
manage switches by deleting, renaming or adding switches. You can also perform these same operations on
fabrics.
73
View SAN Information
To view information about a particular element of your SAN, click the element immediately above it in the
hierarchy. When you click the parent element, information about the child element that you want to view
appears in the right-hand view window.
When you click an element in the SAN Elements tab, Fabric Manager displays information about all
immediately-subordinate elements in the hierarchy. For instance, if you click My SAN in Summary view, Fabric
Manager displays summaries for Fabrics, SwitchGroups, and PortGroups. If you click a particular fabric in the
hierarchy, Fabric Manager displays information on each switch in the fabric. If you click a particular group,
Fabric Manager displays information on each member of the group.
Each view in the View menu displays different informational content, and you can customize which content
appears.
74
Config Download
With Fabric Manager, you have the opportunity to download a configuration from a baseline file or from a
switch. This file allows you to compare the configuration settings of a baseline switch against the other switches
in the fabric. During the download process, you can selectively choose the settings that you want to download
and the settings that you want to omit.
75
Zoning
Fabric Manager uses Web Tools to configure and administer zoning. A new feature is the Zoning Security.
When you enable security, you can only access zoning with the primary Fibre Channel switch. The zoning icon
does not appear on any other switch in the display. If you do not enable security, you can configure zoning from
any switch.
76
Security Management
Fabric Manager provides a GUI interface to manage security once you enable security with command line
commands. For Fabric Manager to access secure switches, you must configure your security policies (at the
command line) as follows:
You must add the IP address of any host that runs the Fabric Manager server to the IP policy of your
fabric. The server cannot communicate with the fabric if you do not include this IP address. (This holds
true whether or not the machine also runs the Fabric Manager client.)
You must add every client that you want to run API to the API policy of your fabric.
Fabric Manager lets you configure SAN security to restrict sensitive operations to a few trusted switches. It
allows administrators to designate a small number of switches, known as Fabric Configuration Servers (FCS), to
perform fabric-wide management operations. Security acts on a policy basis, which means you can selectively
choose what functionality a switch can access.
EMC does not support this functionality.
77
Firmware Download
Perform a firmware download with Fabric Manager to concurrently download firmware to multiple switches and
(optionally) reboot the switches simultaneously. Before you download firmware, verify that you have assigned
the correct version of firmware to the switches.
78
ISL Checking
Enable Inter-Switch Link (ISL) Checking to monitor any changes to your ISL topology. When you enable ISL
Checking, Fabric Manager takes a snapshot, or stamp, of your ISLs. Whenever you add, remove, or change an
ISL, Fabric Manager compares the change to the stamp and triggers an event when mismatches occur. Events
change the color and status of related icons in the Fabric Manager display and create log entries.
79
Fabric Checking
Fabric Checking monitors the fabric and registers events when you add switches to or remove switches from the
fabric. When you add or remove a fabric, Fabric Checking adds an entry to the switch event log and changes the
status color of the fabric. You can configure Fabric Checking to ignore additions to the fabric, but the software
always registers an event when you remove a switch.
Fabric Manager polls the fabric every fifteen seconds to determine if the fabric has changed. For larger fabrics,
the poll time changes to up to 45 seconds.
80
Fabric Change
When you remove a switch from a checked fabric, the switch appears ghosted in Topology view and the links
to the switch no longer appear. When you return the switch, the standard node replaces the ghost node.
81
Course Summary
Key Points covered in this course:
The various B-series switch models
The design of the B-series switches
ED switch architecture versus DS switch architecture
The tools available for managing/configuring the B-
series switches
The available security features and how to implement
them
Key points covered in this course are shown here. Please take a moment to review them.
82
Thank you for your attention. This ends our training on Connectrix B-Series Architecture and Management.

Connectris B-Series Student Resource Guide PDF

Diunggah oleh

Informasi Dokumen

Deskripsi Asli:

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Connectris B-Series Student Resource Guide PDF

Diunggah oleh

Hak Cipta:

Format Tersedia

Self Study modules consist of focused, in-depth training content that can be consumed in about 1-2 hours

EMC Global Education Self Study

Anda mungkin juga menyukai