Anda di halaman 1dari 16
HOW to CONFIGURE a LOGICAL DOMAIN > OpenSolaris™ How To Guides
HOW to CONFIGURE
a LOGICAL DOMAIN
> OpenSolaris™ How To Guides
a LOGICAL DOMAIN > OpenSolaris™ How To Guides Brian Leonard, OpenSolaris Technical Marketing Engineer

Brian Leonard, OpenSolaris Technical Marketing Engineer

Version 2.0 | Last updated: 06/18/09

About This OpenSolaris How To Guide

This Logical Domain How To guide is intended to help OpenSolaris system adminstrators configure their SPARC CMT machines with logical domains (LDoms). Users are guided step-by-step through the process of installing the logical domain manager and configuring a guest domain.

For more information about LDoms, see http://opensolaris.org/os/community/ldoms/

Contents

Overview

Page 1

Configuring a Solaris 10 Guest Domain on OpenSolaris

Page 1

Step 1: Check the Firmware

Page 1

Step 2: Install the Logical Domains Manager

Page 1

Step 3: Free Up Resources for the Guest Domain

Page 2

Step 4: Create Services Needed to Support the Guest Domains

Page 7

Step 5: Configure the Guest Domain

Page 8

Step 6: Create a Virtual Boot Disk for the Guest Domain

Page 8

Step 7: Create a Virtual Disk for the Guest Operating System Installation DVD

Page 9

Step 8: Define a Virtual Network Interface

Page 9

Step 9: Allocate Resources to the Guest Domain

Page 9

Step 10: Log Into the Guest Domain

Page 11

Step 11: Install Solaris 10 in the Guest Domain

Page 11

Step 12: Operate the Guest Domain

Page 12

For More Information

Page 13

1

OpenSolaris Operating System

opensolaris.com

Logical Domain How To Guide

Overview

Logical Domains (LDoms) allow you to virtualize OpenSolaris by creating isolated environments in which you can run other

operating systems that are LDom aware, namely Solaris and OpenSolaris. The virtualization features provided by LDoms

are designed into the SPARC CMT microprocessors and are therefore only available on machines containing either the

UltraSPARC T1 (32 threads), UltraSPARC T2 (64 threads), and UltraSparc T2+ (64 threads) chips.

To begin taking advantage of LDoms, the Logical Domains Manager software needs to first be installed which then runs

OpenSolaris in a “paravirtualized” state, meaning OpenSolaris is now aware of the virtualization layer beneath it. At this

point, there's a single domain, the “primary” domain, which is ready to provide virtualization services, such as networking

and disk i/o, to guest domains.

Before guest domains can be configured, resources such as CPUs and memory need to be removed from the primary

domain so that they can later be assigned to the guest domain. As part of the guest domain's configuration, a virtual disk

(or disks) needs to be set up. These virtual disks can map either to an actual disk or to a single file, which is the approach

that is used in this How To guide.

Once created, the guest domain still needs an operating system. As stated above, this can be either Solaris or OpenSolaris.

Solaris 10 is used as the guest operating system for this guide.

Configuring a Solaris 10 Guest Domain on OpenSolaris

Step 1: Check the Firmware

As mentioned above, the SPARC CMT microprocessors have virtualization features built into their firmware. Before

installing the Logical Domains Manager, make sure you have the latest firmware installed. You can check the latest firware

revs at http://docs.sun.com/source/820-4914-10/chapter1.html#d0e419. See the "Required Software to Enable LDoms 1.1

Features" table and the "Required System Firmware Patches" table.

The exact steps for checking and updating your firmware vary depending on your actual hardware. Please reference

http://docs.sun.com/app/docs/prod/coolthreads.srvr#hic for the specific steps that match your hardware.

Step 2: Install the Logical Domains Manager

Before beginning, run the uname -m to verify your processor supports LDoms. The proper result will be sun4v.

jack@dcsw-t5240:~$ uname -m

sun4v

The Logical Domans Manger is available in the IPS repository:

jack@dcsw-t5240:~$ pkg info -r ldomsmanager

Name: system/ldoms/ldomsmanager

Summary: Logical Domains Manager

Category: System/Virtualization

State: Not installed

Publisher: opensolaris.org

Version: 1.1

Build Release: 5.11

Branch: 0.111

Packaging Date: Sat Apr 18 20:05:08 2009

Size: 8.16 MB

FMRI: pkg:/system/ldoms/ldomsmanager@1.1,5.11-0.111:20090418T200508Z

2

OpenSolaris Operating System

opensolaris.com

Install it as follows:

jack@dcsw-t5240:~$ pfexec pkg install ldomsmanager

DOWNLOAD

PKGS

FILES

XFER (MB)

Completed

1/1

40/40

3.78/3.78

PHASE

ACTIONS

Install Phase

74/74

PHASE

ITEMS

Indexing Packages

1/1

Step 3: Free Up Resources for the Guest Domain

Once the Domain Manager is installed, OpenSolaris is running in a domain - the control domain - which is named primary. You can see it by running ldm list:

jack@dcsw-t5240:/usr/bin$ ldm list

NAME

STATE

FLAGS

CONS

VCPU MEMORY

UTIL UPTIME

primary

active

-n-c--

SP

128

130848M

0.2%

3h 41m

The control domain is responsible for managing all of the other domains on the system and initially has all of the system resources allocated to it. Before we can create a guest domain, we need to steal resources from the primary domain. In the output above you can see that we have 128 virtual CPUs and 130GB of memory. The T5240 has two T2+ processors, each with 8 cores and each core having 8 threads which gives us the total of 128 virtual CPUs. Theorectically, we could create 127 guest domains, however allocating an entire core to a single domain is generally considered best for peformance. That's because the T2+ processor in the T5240 I'm using has 4 cores, each represented as a virtual CPU.

You can view all of the free devices on the system by running:

ldm list-devices

At this point, the command does not return a list because all the system resources are assigned to the control domain. Use the -a option to see all of the devices including the domain to which they're assigned:

jack@dcsw-t5240:~$ ldm list-devices VCPU

PID

%FREE

PM

0

0

no

1

0

no

2

0

no

3

0

no

4

0

no

5

0

no

6

0

no

7

0

no

8

0

no

9

0

no

10

0

no

11

0

no

3

OpenSolaris Operating System

opensolaris.com

12 0

no

13 0

no

14 0

no

15 0

no

16 0

no

17 0

no

18 0

no

19 0

no

20 0

no

21 0

no

22 0

no

23 0

no

24 0

no

25 0

no

26 0

no

27 0

no

28 0

no

29 0

no

30 0

no

31 0

no

32 0

no

33 0

no

34 0

no

35 0

no

36 0

no

37 0

no

38 0

no

39 0

no

40 0

no

41 0

no

42 0

no

43 0

no

44 0

no

45 0

no

46 0

no

47 0

no

48 0

no

49 0

no

50 0

no

51 0

no

52 0

no

53 0

no

54 0

no

55 0

no

56 0

no

57 0

no

58 0

no

59 0

no

4

OpenSolaris Operating System

opensolaris.com

60 0

no

61 0

no

62 0

no

63 0

no

64 0

no

65 0

no

66 0

no

67 0

no

68 0

no

69 0

no

70 0

no

71 0

no

72 0

no

73 0

no

74 0

no

75 0

no

76 0

no

77 0

no

78 0

no

79 0

no

80 0

no

81 0

no

82 0

no

83 0

no

84 0

no

85 0

no

86 0

no

87 0

no

88 0

no

89 0

no

90 0

no

91 0

no

92 0

no

93 0

no

94 0

no

95 0

no

96 0

no

97 0

no

98 0

no

99 0

no

100 0

no

101 0

no

102 0

no

103 0

no

104 0

no

105 0

no

106 0

no

107 0

no

5

OpenSolaris Operating System

opensolaris.com

108 0

no

109 0

no

110 0

no

111 0

no

112 0

no

113 0

no

114 0

no

115 0

no

116 0

no

117 0

no

118

0

no

119 0

no

120 0

no

121

0

no

122 0

no

123 0

no

124 0

no

125 0

no

126 0

no

127

0

no

MAU

ID

CPUSET

BOUND

0

(0, 1, 2, 3, 4, 5, 6, 7)

 

primary

1

(8, 9, 10, 11, 12, 13, 14, 15)

primary

2

(16, 17, 18, 19, 20, 21, 22, 23)

primary

3

(24, 25, 26, 27, 28, 29, 30, 31)

primary

4

(32, 33, 34, 35, 36, 37, 38, 39)

primary

5

(40, 41, 42, 43, 44, 45, 46, 47)

primary

6

(48, 49, 50, 51, 52, 53, 54, 55)

primary

7

(56, 57, 58, 59, 60, 61, 62, 63)

primary

8

(64, 65, 66, 67, 68, 69, 70, 71)

primary

9

(72, 73, 74, 75, 76, 77, 78, 79)

primary

10

(80, 81, 82, 83, 84, 85, 86, 87)

primary

11

(88, 89, 90, 91, 92, 93, 94, 95)

primary

12

(96, 97, 98, 99, 100, 101, 102, 103)

primary

13

(104, 105, 106, 107, 108, 109, 110, 111)

primary

14

(112, 113, 114, 115, 116, 117, 118, 119)

primary

15

(120, 121, 122, 123, 124, 125, 126, 127)

primary

MEMORY

PA

SIZE

BOUND

0x0

512K

_sys_

0x80000

1536K

_sys_

0x200000

94M

_sys_

0x6000000

32M

_sys_

0x8000000

96M

_sys_

0xe000000

130848M

primary

6

OpenSolaris Operating System

opensolaris.com

IO

 

DEVICE

PSEUDONYM

BOUND

OPTIONS

pci@400

pci_0

yes

pci@500

pci_1

yes

Notice in the output above that the system also has 16 MAUs, which are cryptographic Mathmatical Arithmetic Units that

are used to accelarate cryptographic processing. There's a MAU associated with each core and we'll need to deal with

those as well as we free up resources.

Free up 1 MAU for the guest domain by setting the control domain to 15 MAUs:

ldm set-mau 15 primary

Free up 8 CPUs (1 core) for the guest domain by setting the control domain to 120 CPUs:

ldm set-vcpu 120 primary

Free up 16,356 MBs of memory for the guest domain by setting the control domain to 114,492 MB:

jack@dcsw-t5240:~$ ldm set-memory 114492M primary

Initiating delayed reconfigure operation on LDom primary. All configuration

changes for other LDoms are disabled until the LDom reboots, at which time

the new configuration for LDom primary will also take effect.

Now list-devices shows the resources that have been made available for the guest domain:

jack@dcsw-t5240:~$ ldm list-devices VCPU PID %FREE 120 100 121 100 122 100 123 100 124
jack@dcsw-t5240:~$ ldm list-devices
VCPU
PID
%FREE
120 100
121 100
122 100
123 100
124 100
125 100
126 100
127 100
MAU
ID
CPUSET
15
(120, 121, 122, 123, 124, 125, 126, 127)
MEMORY
PA
SIZE
0x1c01c00000
16356M

7

OpenSolaris Operating System

opensolaris.com

Step 4 : Create Services Needed to Support the Guest Domains

Virtual disk and network services need to be established in the control domain to support the guest domains. Create the

virtual disk service named primary-vds:

ldm add-vds primary-vds0 primary

Create the virtual network service. In this example it is named primary-vsw0 and maps to the physical NIC nxge0:

ldm add-vsw net-dev=nxge0 primary-vsw0 primary

Create the Virtual Console Concentrator service. We will use this service to log into the domain. In this example the service

is named primary-vcc0:

ldm add-vcc port-range=5000-5100 primary-vcc0 primary

View all the configured services using the list-services command:

jack@opensolaris:~$ ldm list-services VCC

 

NAME

LDOM

PORT-RANGE

primary-vcc0

primary

5000-5100

VSW

 

NAME

LDOM

MAC

NET-DEV DEVICE DEFAULT-VLAN-ID PVID

VID

MODE

primary-vsw0 primary 00:14:4f:f9:21:7c nxge0

VDS

switch@0 1

1

NAME

LDOM

VOLUME

OPTIONS

MPGROUP

DEVICE

primary-vds0

primary

Reboot the primary domain so the changes can take effect. Note, if you know you're going to be creating multiple guest

domains, allocate the minimum number of resources you need for the guest domain to minimize the need for future

reboots:

pfexec reboot

After reboot, you can view the new configuration for the primary domain, now using only 120 CPUs and 114492M of memory:

jack@dcsw-t5240:~$ ldm list

NAME

STATE

FLAGS

CONS

VCPU MEMORY

UTIL UPTIME

primary

active

-n-c-- SP

120

114492M 0.1%

3m

Finally, enable the virtual network terminal server daemon, which provides console services to the guest domain:

8

OpenSolaris Operating System

oepnsolaris.com

Step 5: Configure the Guest Domain

Add a new guest domain called myldom:

ldm add-domain myldom

Allocate the MAU:

ldm set-mau 1 myldom

Allocate the 8 free CPUs to the guest domain:

ldm set-vcpu 8 myldom

Allocate the 16356 megabytes of free memory to the guest domain:

ldm set-memory 16356M myldom

Step 6: Create a Virtual Boot Disk for the Guest Domain

The guest domain needs a virtual boot disk. The device can either be an actual disk or a file in a ZFS file system, which was the option chosen for this How To guide:

Create a new file system:

pfexec zfs create rpool/myldom

Create a file to use as the boot disk for the domain:

pfexec mkfile 10G /rpool/myldom/zdisk

Define a virtual disk service device named vol1 for the zdisk file and associate it with the primary domain:

ldm add-vdsdev /rpool/myldom/zdisk vol1@primary-vds0

Define a virtual disk for the guest domain named vdisk1 that uses the primary domain’s virtual disk service:

ldm add-vdisk vdisk1 vol1@primary-vds0 myldom

Set the auto-boot property to false so when we first start the domain it doesn't try to boot from this empty disk:

ldm set-var auto-boot\?=false myldom

9

OpenSolaris Operating System

opensolaris.com

Step 7: Create a Virtual Disk for the Guest Operating System Installation DVD

Either Solaris or OpenSolaris can be installed in the guest domain. If you prefer to install OpenSolaris, you'll need to use the Automated Installer (AI). For more information on the AI, see http://www.opensolaris.com/use/Auto_Installer.pdf.

For this How To guide Solaris 10 will be installed as the guest operating system.

Download the Solaris 10 on SPARC ISO from http://www.sun.com/software/solaris/get.jsp.

Then verify it's MD5 checksum against the one provided:

jack@dcsw-t5240:~$ md5sum sol-10-u7-ga-sparc-dvd.iso c72ad13efb20904251ed334f99f95bd6 sol-10-u7-ga-sparc-dvd.iso

Define a virtual disk service device for the ISO file and associate it with the primary domain. The disk service device in this example is named solarisdvd:

ldm add-vdsdev ~/sol-10-u7-ga-sparc-dvd.iso solarisdvd@primary-vds0

Define a virtual disk for the guest domain that uses the primary domain’s virtual disk service. The virtual disk in this example is named dvd:

ldm add-vdisk dvd solarisdvd@primary-vds0 myldom

Step 8: Define a Virtual Network Interface

Define a virtual network interface for the guest domain that uses the primary domain's virtual network service. The virtual network interface in this example is named vnet0:

ldm add-vnet vnet0 primary-vsw0 myldom

Step 9: Allocate Resources to the Guest Domain

The guest domain is currently inactive:

jack@dcsw-t5240:/rpool/myldom$ ldm list

NAME

STATE

FLAGS

CONS

VCPU MEMORY

UTIL UPTIME

primary

active

-n-cv-

SP

120

114492M

0.5%

41m

myldom

inactive

------

8

16356M

Run the bind command to activate it:

ldm bind myldom Not enough free memory present to meet this request Could not bind requested memory for LDom myldom

Reduce the memory on the guest domain by 256M and try the bind again:

10

OpenSolaris Operating System

opensolaris.com

Now its state shows as bound:

jack@dcsw-t5240:/rpool/myldom$ ldm list

NAME

STATE

FLAGS

CONS

VCPU MEMORY

UTIL UPTIME

primary

active

-n-cv- SP

120

114492M

0.2%

1h 8m

myldom

bound

------ 5000

8

16100M

You can run the list command to see the full configuration of the new domain:

jack@dcsw-t5240:~$ ldm list -l myldom

NAME

STATE

FLAGS

CONS

VCPU MEMORY

UTIL UPTIME

 

myldom

bound

-----

5000

8

16100M

MAC

00:14:4f:fb:b7:bf

HOSTID

0x84fbb7bf

 

VCPU

VID

PID

UTIL STRAND

 

0

120

100%

1

121

100%

2

122

100%

3

123

100%

4

124

100%

5

125

100%

6

126

100%

7

127

100%

MAU

ID

CPUSET

15

(120, 121, 122, 123, 124, 125, 126, 127)

 

MEMORY

RA

PA

SIZE

0x2000000

0x1c02000000

16100M

NETWORK

NAME

SERVICE

DEVICE

MAC

MODE

PVID VID

vnet

primary-vsw0@primary network@0 00:14:4f:fb:2e:06

1

DISK

NAME

VOLUME

TOUT DEVICE SERVER

 

MPGROUP

vdisk1

vol1@primary-vds0

 

disk@0

primary

dvd

solarisdvd@primary-vds0

disk@1

primary

VCONS

NAME

SERVICE

PORT

myldom

primary-vcc0@primary

5000

11

OpenSolars Operating System

opensolaris.com

Step 10: Log Into the Guest Domain

Notice from the output above that the virtual console is running at port 5000. From the control domain:

jack@dcsw-t5240:$ telnet localhost 5000 Trying 127.0.0.1 Connected to localhost. Escape character is '^]'. Connecting to console "myldom" in group "myldom" Press ~? for control options

In another terminal window, start the guest domain:

jack@dcsw-t5240:~$ ldm start myldom LDom myldom started

Back in the guest domains console, you should find yourself at the OK prompt:

T5240, No Keyboard Copyright 2008 Sun Microsystems, Inc. All rights reserved. OpenBoot 4.29.0.a, 512 MB memory available, Serial #83528876. Ethernet address 0:14:4f:fa:8c:ac, Host ID: 84fa8cac. {0} ok

Step 11: Install Solaris 10 in the Guest Domain

Our next step is to install Solaris. First we need to find the device alias for the dvd virtual disk we configured earlier:

{0} ok devalias dvd

vidsk1 /virtual-devices@100/channel-devices@200/disk@0 vnet0 /virtual-devices@100/channel-devices@200/network@0

net /virtual-devices@100/channel-devices@200/network@0 disk /virtual-devices@100/channel-devices@200/disk@0 virtual-console /virtual-devices/console@1

name

/virtual-devices@100/channel-devices@200/disk@1

aliases

Use the dvd device alias to boot and install the domain:

{0} ok boot /virtual-devices@100/channel-devices@200/disk@1 Boot device: /virtual-devices@100/channel-devices@200/disk@1 File and args:

SunOS Release 5.10 Version Generic_139555-08 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Configuring devices. Using RPC Bootparams for network configuration information. Attempting to configure interface vnet0 Skipped interface vnet0 Reading ZFS config: done. Setting up Java. Please wait

12

OpenSolaris Operating System

opensolaris.com

At this point you run through a standard installation of Solaris 10. Some items to note, the installer will recognize the virtual network interface, vnet0:

Host Name for vnet0 ─────────────────────────────────

Enter the host name which identifies this system on the network. The name must be unique within your domain; creating a duplicate host name will cause

problems on the network after you install Solaris. A host name must have at least one character; it can contain letters, digits, and minus signs (-). Host name for vnet0 myldom ─────────────────────────────────────────────── Esc-2_Continue Esc-6_Help

As well as the 10GB virtual disk that was created:

Select Disks ─────────────────────────────────────

On this screen you must select the disks for installing Solaris software. Start by looking at the Suggested Minimum field; this value is the approximate space needed to install the software you've selected. For ZFS,

multiple disks will be configured as mirrors, so the disk you choose, or the slice within the disk must exceed the Suggested Minimum value. NOTE: ** denotes current boot disk

Disk Device

=============================================================================

(F4 to edit)

Available Space

10152 MB

[X] c0d0

Maximum Root Size:

10152 MB

Suggested Minimum:

6424 MB

────────────────────────────────────────────────

Esc-2_Continue

F3_Go Back

F4_Edit

F5_Exit

F6_Help

Step 12: Operate the Guest Domain

When installation completes, the system will boot and can be operated as any other standalone Solaris 10 system.

Note, the system can also be halted from the primary domain. First, verify that the guest domain is active:

jack@dcsw-t5240:~$ ldm list

NAME

STATE

FLAGS

CONS

VCPU MEMORY

UTIL UPTIME

primary

active

-n-cv-

SP

120

114492M 0.9%

20m

myldom

active

-n---- 5000

8

16100M

2.1%

9m

Then stop the domain:

13

OpenSolaris Operating System

opensolaris.com

For More Information

For more information about Logical Domains and OpenSolaris, check out the following URLs:

Description

Logical Domains OpenSolaris Community OpenSolaris LDoms Discussion Forum OpenSolaris Automated Installion Project OpenSolaris Home Page

http://opensolaris.org/os/project/caiman/auto_install/ http://www.opensolaris.com/
http://opensolaris.org/os/project/caiman/auto_install/ http://www.opensolaris.com/
http://opensolaris.org/os/project/caiman/auto_install/ http://www.opensolaris.com/
http://opensolaris.org/os/project/caiman/auto_install/ http://www.opensolaris.com/
http://opensolaris.org/os/project/caiman/auto_install/ http://www.opensolaris.com/

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN Web sun.com

95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN Web sun.com ©2009 Sun Microsystems, Inc. All rights reserved. Sun,

©2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, Solaris and OpenSolaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. SunWIN # 487538 Lit # SWWP11968-0 06/09