A Primer of FCoE: Fib re Ch a n n el ov er Et h ern et

Dr Walter Dey, Technical Consulting, Advanced Technologies Cisco S ystem s, w dey@ cisco. com

This paper is an intro d uc tio n to Fib re Channel o v er E thernet (FCo E ), a d irec t m apping o f Fib re Channel f ram es into E thernet f ram es. In April 2 0 0 7, a pro j ec t f o r a new IN CITS S tand ard , Fib re Channel Bac k b o ne 5 (FC-BB-5), within AN S I T1 1 was started [1 ]. The g o al o f FCo E is to enab le I/O c o nso lid atio n as well as f ab ric c o nso lid atio n. The pro j ec t pro po sal rec o m m end s the d ev elo pm ent o f a set o f ad d itio nal and enhanc ed m ec hanism s, serv ic es and pro to c o ls to c o nnec t Fib re Channel entities o v er selec ted no nFib re Channel inf rastruc tures. The em phasis is o n FCo E , no t Classic al E thernet and all o f the sec urity relev ant issues (see e.g ., [2 ]). Fo r m o re tec hnic al d etails, please ref er to ref erenc es [3], [4 ].

Abstract

S CS I o v er TCP /IP (iS CS I) is a ano ther so lutio n that has had lim ited suc c ess f o r I/O c o nso lid atio n, and is m ainly used in lo wer-end serv ers. It uses TCP o v er IP f o r I/O

The b ig g est lim itatio n o f using Inf inib and f o r I/O c o nso lid atio n is its appro ac h to ac c essing sto rag e traf f ic in a m ix ed Inf inib and /Fib re Channel env iro nm ent. The ho sts speak S CS I R D MA P ro to c o l (S R P ) and req uire an Inf inib and to Fib re Channel g ateway to c o m m unic ate with sto rag e array s. The IB/FC g ateway is statef ul, c o m plex and o f lim ited perf o rm anc e, d ue to the nature o f the req uired g ateway pro c essing . N ativ e Inf inib and sto rag e array s are g rad ually b ec o m ing av ailab le, b ut are no t c o m m o n. In ad d itio n, the c lassic al FC sec urity m o d el b ased o n H o st W o rld W id e N am es (W W N s) c anno t b e used , sinc e the W W N s are tho se o f the g ateway , and no t o f the ho sts.

In the c ase o f Inf inib and , c usto m ers need to install a third netwo rk , in ad d itio n to the ex isting E thernet and Fib re Channel netwo rk s. This initially ad d s c o sts in to o ls, training , and o peratio ns ex penses. The d istanc e ac hiev ab le with Inf inib and is also lim ited (appro x im ately 8m f o r passiv e c o pper c ab les, D D R ).

I/O c o nso lid atio n c an b e d o ne with Inf inib and o r iS CS I. W hy has I/O seen a hig her ad o ptio n rate with these o ptio ns?

I/O c o nso lid atio n is the m ultiplex ing o f d if f erent U pper L ay er P ro to c o ls (U L P ) o v er a c o m m o n transpo rt netwo rk . The m o st c o m m o n pro to c o ls are IP and S CS I, and in so m e c ases Messag e P assing Interf ac e (MP I) f o r the hig hest perf o rm anc e/lo west latenc y applic atio ns. The b enef its o f I/O c o nso lid atio n are: 1 ) a red uc tio n in the num b er o f I/O slo ts req uired 2 ) a red uc ed num b er o f f ab ric s 3) red uc ed po wer and c o o ling req uirem ents 4 ) and red uc ed c ab ling . These b enef its are b ec o m ing inc reasing ly im po rtant f o r 1 R U rac k m o unt serv ers and b lad e sy stem s.

1 I n tro d u cti o n

c o nso lid atio n no t

c o nso lid atio n and is ty pic ally d eplo y ed o v er E thernet. TCP is a well k no wn pro to c o l that pro v id es a hig h lev el o f reliab ility ; it c an b e used in lo c al and wid e area netwo rk s, and pro v id es c o ng estio n m anag em ent. H o wev er, TCP is c o m plex , and m o stly im plem ented in so f tware; iS CS I g ateway s and ho sts hav e perf o rm anc e lim itatio ns. TCP O f f lo ad E ng ines (TO E s) hav e b een pro po sed to so lv e this issue, with lim ited m ark et ad o ptio n. N ativ e iS CS I d isk sub sy stem s are av ailab le to d ay , while iS CS I targ ets with 1 0 G speed interf ac es are still rare.

FCo E is a way to c o nso lid ate Fib re Channel o v er E thernet, b y transpo rting (m apping ) Fib re Channel f ram es in the pay lo ad o f E thernet f ram es. In the Aug ust 2 0 0 7 T1 1 m eeting , an ag reem ent o n the FCo E f ram e lay o ut was reac hed (Fig . 1 ) [5].

FCo E is a tec hno lo g y d esig ned f o r the d ata c enter. W hile E thernet has b een suc c essf ul as a transpo rt f o r IP traf f ic , it has no t b een suc c essf ul as an I/O c o nso lid atio n tec hno lo g y (i.e., as a unif ied netwo rk c apab le o f c arry ing sto rag e (S CS I) and c luster traf f ic (IP C) in ad d itio n to IP ). Fib re Channel is the d o m inant tec hno lo g y f o r sto rag e interc o nnec ts.

2 W h at i s F i bre C h an n e l o v e r E th e rn e t ( F C o E ) ?

Fig . 1 . FCo E Fram e L ay o ut FCo E will b e po ssib le o v er 1 G E thernet, b ut the m aj o rity o f im plem entatio ns m o st lik ely will use 1 0 G E thernet (c o m pared to to d ay s suppo rt f o r 2 x 4 G Fib rec hannel and 2 x 1 G E thernet d ata stream s). In the f uture 4 0 G and 1 0 0 G will also b e ab le to transpo rt FCo E .

W hile c lassic al E thernet is a lo ssy netwo rk (i.e., f ram es m ay b e d isc ard ed in the presenc e o f c o ng estio n), FCo E req uires the und erly ing E thernet lay er to b e lo ssless. This m eans that the und erly ing E thernet suppo rt has to im plem ent so m e ex tensio ns in o rd er to pro v id e FCo E with the serv ic e it need s (see 3).

W hen FCo E is c arried o v er E thernet the perf o rm anc e is im pro v ed b ec ause TCP is no t req uired , unlik e iS CS I. S ee Fig ure 2 b elo w.
Ap p li c a ti ons IP S CS I

T CP

U D P

F i b re Cha nnel F CoE

Lossless Ethernet MAC

Tro ub lesho o ting is sim plif ied , sinc e FCo E o f f ers all o f the Fib re Channel to o ls plus all o f the E thernet to o ls. 1 ) the ev o lutio n o f a Classic al E thernet switc h with ad d itio nal Fib re Channel switc hing f unc tio ns and lo ssless E thernet transpo rt c apab ility , o r A FCo E switc h m ay b e b uilt as:

FCo E appears c o nc epts lik e W the sam e traf f ic and the sam e m

The m ax im um siz e o f a Fib re Channel f ram e is 2 1 1 2 b y tes, f o r a FCo E f ram e it is 2 1 65 b y tes, larg er than the m ax im um Classic al E thernet f ram e siz e. Ad d itio nal head ers (e.g ., MACsec E nc ry ptio n H ead ers) m ay req uire the f ram e leng th to g ro w ev en f urther. To av o id any f rag m entatio n and reassem b ly , FCo E im plem entatio ns will suppo rt b ab y j um b o f ram es o f at least 2 1 65 b y tes. A m inim um f ram e siz e o f 2 .5 K B is rec o m m end ed . as Fib re Channel to a ho st, a sto rag e array o r a switc h. W ell k no wn W N s, FCID s, hard /so f t z o ning , d N S , R S CN s are retained . FCo E c an use m anag em ent tec hniq ues (e.g ., in o rd er d eliv ery o r FS P F lo ad b alanc ing ), anag em ent and sec urity m o d els.

Fig . 2 . S tand ard IP applic atio n and FCo E S tac k

A FCo E switc h has two c o m po nents: the d ata plane that f o rward s the f ram es and enf o rc es hard z o ning and the c o ntro l plane that runs the Fib re Channel c o ntro l pro to c o ls, lik e d N S and FS P F.

The f o llo wing sec tio ns c o v er o ptio n 1 , with lo ssless E thernet switc hing ev o lv ing to hand le FCo E traf f ic .

2 ) the ev o lutio n o f a stand ard Fib re Channel switc h with the ad d itio n o f E thernet L inec ard s and Brid g ing f unc tio nality , b o th suppo rting lo ssless E thernet.

Fig ure 3 sho ws a lo g ic al d iag ram o f a FCo E switc h whic h has o nly E thernet po rts (m o stly 1 0 G , altho ug h 1 G is po ssib le as well), suppo rting Classic al E thernet as well as FCo E o v er lo ssless E thernet.

Fig .3. FCo E switc h with E thernet P o rts The f o llo wing f ig ure 4 sho ws a FCo E switc h, whic h in ad d itio n to E thernet po rts also has Fib re Channel po rts, that im plem ent the m apping b etween FCo E and Fib re Channel. This stateless Fib er Channel f ram e-m apper (FCM) is im plem ented in hard ware, pro v id ing hig h perf o rm anc e. FCo E to Fib re Channel m appers m ay b e ab le to ac hiev e thro ug hput o ne o r two o rd ers o f m ag nitud e hig her than c urrent iS CS I to Fib re Channel g ateway s.

Fig .4 . FCo E S witc h with E thernet and Fib rec hannel P o rts

The m apping o f Fib re Channel lev els into the E thernet lay ers is sho wn in Fig 5.

The f irst applic atio n o f FCo E will b e serv er I/O c o nso lid atio n, as sho wn in f ig ure 6. Instead o f hav ing m ultiple Fib re Channel H BA c o nnec tio ns and G ig ab it E thernet c o nnec tio ns, the serv ers hav e o nly two FCo E c apab le interf ac es (o r o ne f o r no n-hig h av ailab ility env iro nm ents).

Fig .5. P ro to c o l L ay ering o f FCo E

LAN
Management

S AN A

S AN B

E th e rn e t

F C

Ethernet w i th F C o E
SER V ER SER V ER

Fig .6. First P hase FCo E Applic atio n: I/O c o nso lid atio n.

FCo E c an b e seen as an alternativ e to iS CS I o r c lassic al Fib re Channel o n the ho sts, no t replac ing Fib re Channel, b ut pro v id ing an ad d itio nal o ptio n that is f o rward c o m patib le with c urrent S AN s. Classic al E thernet to d ay d o esnt g uarantee f ram e d eliv ery . An E thernet netwo rk m ay lo se f ram es d ue to two reaso ns: S witc h c o ng estio n Transm issio n erro rs

In a f uture phase, the 2 separate f ab ric s f o r L AN and S AN sho wn at the to p o f Fig .6 c o uld b e c o nso lid ated as well, ex tend ing the FCo E f ab ric f ro m the ed g e to the c o re o f the netwo rk . N ativ e FCo E sto rag e array s will also b e d ev elo ped .

The FCo E switc h c o nnec ting the serv ers has the f unc tio nal units sho wn in Fig .4 ab o v e. It d em ultiplex es the I/O c o nso lid ated traf f ic (Fib re Channel o v er E thernet and IP o v er E thernet) running the FCo E pro to c o l, c o nnec ting to the c lassic al E thernet and to the Fib re Channel f ab ric s.

Fro m the pro to c o l po int o f v iew, the Fib re Channel link is replac ed b y a lo ssless E thernet link , whic h is represented b y a pair o f MAC ad d resses and f unc tio ns as an N _ P o rt to F_ P o rt c o nnec tio n. In this E thernet env iro nm ent, the FC b uf f er c red it param eters are no t used . L o ssless transpo rt o v er E thernet is attained b y using a traf f ic prio rity P ause m ec hanism (see b elo w). A d etailed d esc riptio n o f FCo E ad d ressing and ad d ress reso lutio n c an b e f o und in [6]. The S to rag e Ad m inistrato rs v iew is no d if f erent f ro m a Fib re Channel H BA; the FCo E H BAs d isplay FC initiato rs.

3 L o ssl e ss E th e rn e t

Transm issio n erro rs are rare in the d ata c enter, where the c ab les are sho rt and o f g o o d q uality . All pro to c o ls are po tentially af f ec ted b y transm issio n erro rs. Ty pic ally , b it erro r rates (BE R ) o f G ig ab it E thernet, 1 0 G E thernet, Inf inib and and all Fib re Channel speed s are d ef ined as < 1 0 -1 2 , altho ug h m any v end o rs hav e im pro v ed this v alue b y 3 o rd ers o f m ag nitud e. In the presenc e o f transm issio n erro rs, FCo E b ehav es as Fib re Channel (i.e., it d ro ps c o rrupted f ram es and lets S CS I rec o v er the erro r).

W ith the ad v ent o f f ull-d uplex link s, IE E E 80 2 .3 ad d ed the c o nc ept o f P ause with IE E E 80 2 .3X (IE E E 80 2 .3 Annex 31 B). Altho ug h P ause and c red its are d if f erent m ec hanism s, they m ay ac hiev e the sam e results o v er the sho rt d istanc es present in the d ata c enter. S witc hes pro perly im plem enting P ause will pro v id e an ad eq uate suppo rt o f FCo E , c reating a no -d ro p serv ic e eq uiv alent to the o ne o f Fib re Channel if no c o ng estio n

Fib re Channel has a ho p-b y -ho p b uf f er-to -b uf f er c red it m ec hanism that g uarantees that a f ram e m ay b e sent o n a link o nly if there is a b uf f er av ailab le at the rec eiv ing sid e.

Current E thernet im plem entatio ns are c halleng ed to pro v id e a so lutio n to this pro b lem .

Co ng estio n insid e switc hes is unav o id ab le if traf f ic to ward a d estinatio n ex c eed s the c apab ility o f the link o r o f the d estinatio n. W hen c o ng estio n o c c urs, b uf f ers b eg in to f ill up and ev entually they o v erf lo w resulting in d ro pped f ram es.

o c c urrs. There are ex tensio ns to E thernet b eing pro po sed at IE E E that c an rem ed y the lo ssy nature o f E thernet f ab ric s so that FCo E traf f ic c an b e c o nso lid ated with o ther E thernet traf f ic . An o v erv iew o f the k ey E thernet ex tensio ns f o r FCo E are o utlined b elo w. O v er the last 2 5 y ears, E thernet has g o ne thro ug h m any ev o lutio nary phases, f o r enhanc ed speed and f unc tio nality . O rig inally d esig ned f o r the L AN env iro nm ent, E thernet has rec eiv ed spec ial f eatures f o r Carrier E thernet . IE TF and IE E E are c urrently ex tend ing the c lassic al E thernet f eatures to suppo rt sto rag e traf f ic , nam ely lo ssless E thernet. The f o llo wing stand ard s ac tiv ities are k ey E thernet f ab ric . f o r ac hiev ing a f ully

4 E th e rn e t S tan d ard s Acti v i ti e s

f unc tio nal lo ssless

IE E E 80 2 .1 Qau Co ng estio n N o tif ic atio n [8]

IE E E 80 2 .1 P er-P rio rity P ause o r Class-Based Flo w Co ntro l [7]

In the f o llo wing sec tio ns, P er P rio rity Manag em ent are c o v ered in m o re d etail.

The c o ng estio n m anag em ent wo rk o f 80 2 .3ar is at the link lev el, the o ne o f 80 2 .1 au is at the netwo rk lev el. There is v ery little ac tiv ity c urrently g o ing o n in 80 2 .3ar. P ause (P P P , ak a CBFC) and

S pec if y a m ec hanism

IE E E 80 2 .3ar E nhanc em ents f o r rate lim iting [9]

S ig nal c o ng estio n inf o rm atio n to end statio ns to av o id f ram e lo ss to lim it the rate o f transm itted d ata o n an E thernet link

Co ng estio n

IP traf f ic m ay b e asso c iated with a P rio rity that is no t paused at all, while Fib re Channel traf f ic m ay b e c lassif ied as a P rio rity that is paused to g uarantee a no -d ro p serv ic e lik e nativ e Fib re Channel.

The IE E E 80 2 .1 Q U ser P rio rity c o nc ept is ex tend ed to im plem ent a P ause Mec hanism f o r eac h U ser P rio rity . In this way eac h P rio rity m ay b e paused ind epend ently and so m e m ay no t b e paused at all.

Cisc o has pro po sed to IE E E 80 2 .1 the c o nc ept o f P er-P rio rity P ause o r Class-Based P ause. IE E E 80 2 .1 has no t f inally d ec id ed ab o ut the stand ard iz atio n o f this ex tensio n. The new P AU S E f unc tio n is also ref erred to as Class Based Flo w Co ntro l (CBFC) [7].

Per Priority Pause

Cred its and P auses are k no wn to hav e sc alab ility lim itatio ns d ue to c o ng estio n spread ing and H ead O f L ine Blo c k ing . IE E E 80 2 .1 has started a pro j ec t nam ed IE E E 80 2 .1 au to c o ntro l the c o ng estio n in the netwo rk and ad d ress it at the ed g e. IE E E 80 2 .1 au is stud y ing the c o nc ept o f Bac k ward Co ng estio n N o tif ic atio n (BCN ), in whic h the c o ng ested po int sig nals the presenc e o f the c o ng estio n b ac k ward to the so urc e.

C on g estion M an ag em en t

P er P rio rity P ause as well as Co ng estio n Manag em ent are no t m and ato ry f o r FCo E , b ut will o f f er the ab ility to pro v id e a lo ssless E thernet netwo rk f o r the transpo rt o f nativ e FC traf f ic .

The ev o lutio n o f to d ay s c lassic al E thernet to lo ssless E thernet will pro v id e suppo rt f o r FCo E , the transpo rt o f Fib rec hannel f ram es o v er E thernet. This tec hno lo g y allo ws in a f irst step the im plem entatio n o f serv er I/O c o nso lid atio n, and in the f uture f ab ric c o nso lid atio n, m inim iz ing Capex and O pex (c ab ling , po wer, c o o ling ) in the d atac enter. The f inal FCo E stand ard iz atio n in AN S I T1 1 is ex pec ted in the f irst half o f 2 0 0 8. FCo E is c o m plem entary to iS CS I and FCIP , the latter b eing used f o r S AN ex tensio n o v er wid e area IP netwo rk s.

5 C o n cl u si o n

6 R e f e re n ce s

[1 ] http://www.t1 1 .o rg /f tp/t1 1 /ad m in/pro j ec t_ pro po sals/0 7-1 96v 1 .pd f [2 ] E . V y nc k e, C. P ag g en " L AN S eptem b er 2 0 0 7 [4 ] http://www.f c o e.c o m

S witc h S ec urity " , Mac m illan Tec hnic al P ub lishing ,

[3] http://www.t1 1 .o rg /f c o e

[6] http://www.t1 1 .o rg /f tp/t1 1 /pub /f c /b b -4 /0 7-30 3v 0 .pd f [8] http://www.ieee80 2 .o rg /1 /pag es/80 2 .1 au.htm l

[5] http://www.t1 1 .o rg /f tp/t1 1 /pub /f c /b b -5/0 7-4 79v 0 .pd f

[7] http://www.ieee80 2 .o rg /1 /f iles/pub lic /d o c s2 0 0 7/new-c m -b arrass-pause-pro po sal.pd f

[9] http://g ro uper.ieee.o rg /g ro ups/80 2 /3/ar/ind ex .htm l