Lesson 2

Disk-Side and Host-Side Cabling

Disk-Side Cabling Best

Practices
Cable redundantly from both
controllers to both ESMs in each disk
shelf
Be consistent with ESM in ports
In Expansion Ports
Out Expansion Port
Use top-down-bottom-up cabling
method

E5500 Disk-Side Cabling

Example
E5560 Controller
Shelf

DE6600 Disk Shelf 1

DE6600 Disk Shelf 2

DE6600 Disk Shelf 3

E5500 Disk-Side Cabling

Example
To start the topdown loop, cable
from the left EXP
port on
Controller A to
the left in port
on the top ESM
in Disk Shelf 1.

E5500 Disk-Side Cabling

Example
Next, cable
from the left
EXP port on
Controller B to
the right in
port on the top
ESM in Disk
Shelf 1.

E5500 Disk-Side Cabling

Example
Then cable from
the out port
on the top ESM
in Disk Shelf 1
to the left in
port on the top
ESM in Disk
Shelf 2.

E5500 Disk-Side Cabling

Example
Then cable from
the out port
on the top ESM
in Disk Shelf 2
to the left in
port on the top
ESM in Disk
Shelf 3 to
complete the
top-down
cabling loop.

E5500 Disk-Side Cabling

Example
To start the
bottom-up loop,
cable from the
right EXP port
on Controller A
to the left in
port on the
bottom ESM in
Disk Shelf 3.

E5500 Disk-Side Cabling

Example
Next, cable
from the right
EXP port on
Controller B to
the right in
port on the
bottom ESM in
Disk Shelf 3.

E5500 Disk-Side Cabling

Example
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 3 to the
left in port on
the bottom ESM
of Disk Shelf 2.

10

E5500 Disk-Side Cabling

Example
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 2 to the
left in port on
the bottom ESM
of Disk Shelf 1
to complete the
bottom-up loop.

11

E5400 Disk-Side Cabling

Example
E5460 Controller
Shelf

DE6600 Disk Shelf 1

DE6600 Disk Shelf 2

DE6600 Disk Shelf 3

12

E5400 Disk-Side Cabling

Example
To start the topdown loop,
cable from the
EXP port on
Controller A to
the left in port
on the top ESM
in Disk Shelf 1.

13

E5400 Disk-Side Cabling

Example
Next, cable
from the out
port on the top
ESM in Disk
Shelf 1 to the
left in port on
the top ESM in
Drive Shelf 2.

14

E5400 Disk-Side Cabling

Example
Then cable from
the out port
on the top ESM
in Disk Shelf 2
to the left in
port on the top
ESM in Drive
Shelf 3 to
complete the
top-down loop.

15

E5400 Disk-Side Cabling

Example
To start the
bottom-up loop,
cable from the
EXP port on
Controller B to
the left in port
on the bottom
ESM in Disk
Shelf 3.

16

E5400 Disk-Side Cabling

Example
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 3 to the
left in port on
the bottom ESM
of Disk Shelf 2.

17

E5400 Disk-Side Cabling

Example
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 2 to the
left in port on
the bottom ESM
of Disk Shelf 1
to complete the
bottom-up loop.

18

E2700 Disk Expansion

Architecture
E2700 Controllers:
Share two disk-side SAS-3 domains
Use dual-ported expansion, so
external bandwidth capability
matches internal bandwidth
Use new mini-SAS HD cable from
Mini-SAS
controller to ESM
connectors (disk
SAS Mini-HD Connectors
(E2700 controllers)

shelves)

19

Maximizing Bandwidth Performance

Single-stack cabling
Connect both controller EXP ports to same ESM
Daisy-chain from first disk shelf to remaining
shelves

Dual-stack cabling maximizes throughput

performance
Use two daisy-chain cascades
Cable one EXP port from each controller to one
in port in ESM in one stack
Cable second EXP port from each controller to
second in port in ESM in other stack
20

E2700 Disk-Side Cabling Example

Single Stack
E2760 Controller
Shelf

DE6600 Disk Shelf 1

DE6600 Disk Shelf 2

DE6600 Disk Shelf 3

21

E2700 Disk-Side Cabling Example

Single Stack
To start the topdown loop, cable
from the left EXP
port on
Controller A to
the left in port
on the top ESM
in Disk Shelf 1.

22

E2700 Disk-Side Cabling Example

Single Stack
Next, cable
from the right
EXP port on
Controller A to
the right in
port on the top
ESM in Disk
Shelf 1.

23

E2700 Disk-Side Cabling Example

Single Stack
Next, cable
from the out
port on the top
ESM in Disk
Shelf 1 to the
left in port on
the top ESM in
Drive Shelf 2.

24

E2700 Disk-Side Cabling Example

Single Stack
Then cable from
the out port
on the top ESM
in Disk Shelf 2
to the left in
port on the top
ESM in Drive
Shelf 3 to
complete the
top-down loop.

25

E2700 Disk-Side Cabling Example

Single Stack
To start the
bottom-up loop,
cable from the
left EXP port on
Controller B to
the left in port
on the bottom
ESM in Disk
Shelf 3.

26

E2700 Disk-Side Cabling Example

Single Stack
Next, cable
from the right
EXP port on
Controller B to
the right in
port on the
bottom ESM in
Disk Shelf 3.

27

E2700 Disk-Side Cabling Example

Single Stack
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 3 to the
left in port on
the bottom ESM
of Disk Shelf 2.

28

E2700 Disk-Side Cabling Example

Single Stack
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 2 to the
left in port on
the bottom ESM
of Disk Shelf 1
to complete the
bottom-up loop.

29

E2700 Disk-Side Cabling Example

Dual Stack
E2760 Controller
Shelf

DE6600
Disk
Shelf 1

DE6600
Disk
Shelf 3

DE6600
Disk
Shelf 2

DE6600
Disk
Shelf 4

30

E2700 Disk-Side Cabling Example

Dual Stack

31

E2700 Disk-Side Cabling Example

Dual Stack

32

E2700 Disk-Side Cabling Example

Dual Stack

33

E2700 Disk-Side Cabling Example

Dual Stack

34

E2700 Disk-Side Cabling Example

Dual Stack

35

E2700 Disk-Side Cabling Example

Dual Stack

36

E2700 Disk-Side Cabling Example

Dual Stack

37

E2700 Disk-Side Cabling Example

Dual Stack

38

E2600 Disk-Side Cabling

Example
E2660 Controller
Shelf

DE6600 Disk Shelf 1

DE6600 Disk Shelf 2

DE6600 Disk Shelf 3

39

E2600 Disk-Side Cabling

Example
To start the topdown loop,
cable from the
EXP port on
Controller A to
the left in port
on the top ESM
in Disk Shelf 1.

40

E2600 Disk-Side Cabling

Example
Next, cable
from the out
port on the top
ESM in Disk
Shelf 1 to the
left in port on
the top ESM in
Drive Shelf 2.

41

E2600 Disk-Side Cabling

Example
Then cable from
the out port
on the top ESM
in Disk Shelf 2
to the left in
port on the top
ESM in Drive
Shelf 3 to
complete the
top-down loop.

42

E2600 Disk-Side Cabling

Example
To start the
bottom-up loop,
cable from the
EXP port on
Controller B to
the left in port
on the bottom
ESM in Disk
Shelf 3.

43

E2600 Disk-Side Cabling

Example
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 3 to the
left in port on
the bottom ESM
of Disk Shelf 2.

44

E2600 Disk-Side Cabling

Example
Then cable from
the out port
on the bottom
ESM in Disk
Shelf 2 to the
left in port on
the bottom ESM
of Disk Shelf 1
to complete the
bottom-up loop.

45

Snapshot Copy-on-Write Technology

Base
Volume

Logical (No
Actual Disk
Snapshot Group Capacity)
Repository

Physical Disk
Capacity

Physical disk
capacity

46

Copy-on-Write Example
A

CJ

K
E

F
L

H
M

Base
Volume

Snapshot

Repository

Changed Data Blocks in Base

Volume

47

Snapshot Considerations
Performance impact considerations
Copy-on-write technology
Number of changes made to Snapshot
volume
Repository full situations

Capacity considerations
Snapshot takes less actual capacity than
actual clone copy
Repository takes some disk capacity
48

Disk Pools
To start creating disk pools, you right-click
Unconfigured Capacity.
Disk pools have fewer options but they have
dynamic functionality:

In GUI, no choice
of individual disks at
time of creation
No RAID-level
choice
11-disk minimum
No drawer loss
protection

49

Disk Pool Data Usage

4-GB d-stripe made of 10 d-pieces residing on 10 disks
within pool
Intelligent algorithm defines which disks used:
Different set of 10 disks used for each d-stripe
Pseudorandom d-stripe distribution maintains balance
24-Drive Disk Pool

50

Default Preservation
Capacity
Functions like hot spares for disk pools
Number Default Capacity
Set at pool creation
of Disks in
Pool
(Number of
Disks Worth)
Can be altered later to
11
1
more or less capacity:
Minimum amount
required: 0
Maximum amount
supported: 10 disks
worth

1231

3263

64127

128191

192255

256384

51

Dynamic Disk Pools: Disk Failure

For each d-stripe with data on the failed disk:
Segments on other disks read to recreate data
Data written to set of 10 disks in pool

Rebuild operations run in parallel across all

disks 24-Drive Disk Pool Becomes 23-Drive Disk Pool

52

Controller Cache
Is dedicated to these I/O operations:
Between controllers and hosts
Between controllers and disks

Increases controller performance:

Acts as buffer for I/O
Provides faster reads/writes than disk
access
Speeds up writes when write-back
caching used
Uses prefetch to speed up sequential
53

Cache Data Flow

Write Caching
Host App
?

Disk I/O
Queue
WW R R W

Response: okay

54

Cache Data Flow

Request for Data Location in Cache
Host App
?

0x03 0x01
0x03
0x0f

Disk I/O
queue is full.

Response: okay
I/O is written
to cache.
55

Cache Data Flow

Read Caching
Host App
0x03 0x01

Read
0x3

0x0f

Disk I/O
Queue Full

Response: okay
I/O is handled
by cache.
56

Cache Flushing
Cache automatically flushed to
disk:
Age based: When data times out (10second default)
Demand based: When cache space
reaches fill point

Start demand cache flushing: When

start percentage reached,
controller starts
flushing data in cache
57

Cache Blocks
Free cache: Empty cache blocks
available for data
Dirty cache: Data in cache block not
yet written to disk
Clean cache: Data in cache block
consistent with data on disk
Free
Cache

Dirty
Cache

Clean
Cache

58

Flushing Example
Free Cache Area

Free
Cache

Dirty
Cache

Clean
Cache

Flush threshold start = 50%

Write cache is enabled.
Read cache is enabled.

59

Flushing Example
Initial I/O
Read 0x0
Read 0x1 0x0
Read 0x3
Read 0x4
Write 0x3 0x1
(cache
hit)
Read 0x6
Free
Cache

0x3

0x6

0x4

Dirty
Cache

Read
Read
Read
Read
Read

0x0
0x1
0x3
0x4
0x6

Clean
Cache

Flush threshold start = 50%

Write cache is enabled.
Read cache is enabled.

60

Flushing Example
Dirty Cache
Read 0x3
Read 0xa 0x0
Read 0x1
Write 0x7
Write 0x3 0x1

Free
Cache

0x3

0x6

0x7
Read 0xa

0x4

Dirty
Cache

0xa

Clean
Cache

Flush threshold start = 50%

Write cache is enabled.
Read cache is enabled.

61

Flushing Example
Start demand cache flushing Threshold
Reached
Write 0x3
Write 0xa 0x0
Write 0xb
Write 0x2
Read 0x3 0x1

Free
Cache

0x3

0x6

0x7

0x4

0xa

0xb

Dirty
Cache

Write
0x0
Write
0x7
Write
0xb

Clean
Cache

Flush threshold start = 50%

Write cache is enabled.
Read cache is enabled.

62

Flushing Example
After the Cache Flush
0x0

0x3

0x6

0x7

0x1

0x4

0xa

0xb

Free
Cache

Dirty
Cache

Clean
Cache

Flush threshold start = 50%

Write cache is enabled.
Read cache is enabled.

63

Flushing Example
Least Recently Used Queue
Write 0xc
(uses
0x0
Read
LRU) 0x5
(uses
LRU)
Read 0x0 0x1
0xd
(cache
hit)
Read 0xd
(uses
LRU)
Write 0x3 Free
(cache
Cache
hit)

0x3

0x5
0x6

0x7
Read 0x5

0x4

Dirty
Cache

0xa
0xc

0xb

Read 0xd

Clean
Cache

Flush threshold start = 50%

Write cache is enabled.
Read cache is enabled.

64