Anda di halaman 1dari 83

®

IBM Power Systems

HMC V7.3 POWER6 and Power Systems


Allyn Walsh
awalsh@us.ibm.com

IBM System p Virtual User Group


November 20, 2008

i want stress free IT.


i want control.
i want IBM.
© IBM Corporation 2008
IBM Power Systems - System p Virtual User Group

Agenda
ƒ Hardware Management Console background
– What is an HMC and when is it required
– HMC Hardware models
– POWER firmware architecture

ƒ HMC upgrade and update management


ƒ HMC Backup and recovery strategies
ƒ Connection options
– Single server, multiple server, open network access

ƒ Remote management - Features and functions


– GUI server management
– WebSM or browser based access with HMC V7
– SSH (secure shell)

ƒ Added features with V7.3.1 - V7.3.3


– New Call home wizard
– User data Replication
– IPv6, Authentication options (local, Kerberos or LDAP)
– Multiple Shared Processor pools
– Shared Dedicated Processor
– Hot Node add / Repair

2 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

When is it required on POWER Systems


ƒ A user interface for POWER5 and POWER6 • HMC is Required:
hardware management • POWER6 570 (9406-MMA / 9117-MMA)
– Virtualization management • HMC V7.330 for unified 9117-MMA
– Intel based platform - GUI console application
• POWER6 595 (9119-FHA)
– Third party applications locked out
• HMC V7.330 minimum level to support FHA
– Regular refreshes on the hardware technology
• Required on POWER5 or POWER6 systems when:
ƒ POWER6 minimum HMC firmware level V7.310 • Systems with LPARs - Logical Partitions
– V7.3 is compatible with POWER5 servers • Except 8203-E4A and 8204-E8A using PowerVM
– Support statement says level SF240_299 express edition
or later on POWER5 servers • Systems requiring full CuOD (Capacity upgrade On
Demand)
ƒ Remote management of HMC functions • ASMI has subset of CuOD functions
– WebSM client – remote GUI to HMC for V6.1.3 and • Systems with redundant flexible service processors
all earlier versions (FSP)
– Standard browser for remote GUI access to HMC • POWER5 570 with optional redundant FSP
V7.310 and above
• POWER5 590 & 595
– Secure shell (SSH) access to CLI command line
interface • Systems with Bulk Power Assemblies (595 only)

ƒ HMC Supports: • CFM - Concurrent Firmware Maintenance enabled only


– Max of 48 Servers, up to 254 partitions. for HMC managed systems
– Max of 2 HMC's per server.

NOTE:
If none of the above conditions are true (single OS image), HMC is optional

3 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Hardware Management Console - Model refreshes


Rack mounted display and keyboard
Ordered separately (desktop)
(rack)
7310-CR2
7310-C03
7310-CR3
7310-C04
7310-CR4
7310-C05
7042-CR4 7042-C06

For POWER6 servers, new HMC orders will default to 7310-C06


models 7042-Cxx & ship preloaded with V7.3xx

General Availability
• New orders for POWER5 will continue to default to 7310-CR4/C06 models
• HMC function is based on HMC Version/Release installed (not the HMC model)
• 7310-Cxx models can be upgraded to V7.3 (minimum HMC level required for POWER 6

4 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

The 3 Pieces of i5 Maintenance


Partition Maintenance
• i5/OS & LIC, AIX, Linux

Server Firmware
• Power Hypervisor
i5 Managed System

• Flexible Service
Processor
• Power Code (BPC)

HMC
• Updates and Upgrades

5 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Check HMC code level and eFixes installed


Notice that HMC code uses a
Version/Release format and maintenance
is identified by MHxxxxx ptf IDs. MHxxxxx
PTF’s are also used for i5/OS server
firmware maintenance.

Starting with V4R5, this


interface was enhanced to
automatically show eFix
information along with Version
Release, even from a WebSM
session

To view Version/Release, Expand License Internal Code


Maintenance and select HMC Code Update. Prior to
V4R5, the only way to see eFixes was to select ‘Help’ on
the HMC Code Update panel on the HMC. eFix
information was not available from a WebSM session

6 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Multiple ways to check HMC code level on V7.310


Just hover the cursor over the HMC Version on
the Welcome screen or select Updates.

Select the Updates tab

7 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Upgrading and Updating the HMC


ƒ Terms
ƒ Obtain the HMC code maintenance
– Upgrading the HMC
– Changing the HMC to a new
– Traditional
Version or a new release level – Order the fix as a PTF and is received on
– Recovery Media = Upgrade media

– Updating the HMC – Internet


– Updating the HMC within the – Fix or Update file can be downloaded
current release directly to the HMC
– Install Corrective Service – Zip file can be downloaded
– i.e. Applying a V5R2.0 – Zip file can be used with the updhmc
required efix command or copied from a FTP server
– Bring the HMC to a new point that’s behind the firewall
release by installing an – ISO file can be downloaded and burned
update PTF – Image of the media, requires an
authoring program
ƒ Upgrading or Updating the HMC
– FTP
is a Concurrent Process
– Files can be downloaded to any FTP
(performed independent of the
server that HMC can access
servers that it manages)

8 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Upgrading the HMC to a New Release


ƒ Ver/Rel/point_level of HMC i.e. (V6.1.3 = version 6, release 1, point level 3)
ƒ HMC version upgrade required Recovery media.
– Starting with version 5, release upgrades require recovery media
– V4.x to V4.x (Could be done using Recovery media or Update media)
– V4.5 to V5.1 – Recovery media required
– V4.5 (with special steps) to V6.1 – Recovery media required Starting with V5R1
– V5.2 to V6.1.x – Recovery media required can upgrade
– V6.1.2 to V7.310 – Recovery media required and install the HMC
– V7.3.3 must use recovery media AND come from V7.3.1 or V7.3.2 from a server on
the network
ƒ Recovery Media is an Upgrade - (boot the HMC from media)
– One step short of a scratch install
– Upgrade option preserves saved upgrade data on hard drive and/or media
– Overwrites all other HMC code and configuration files
ƒ Update the HMC release using Corrective Service
– Uses an update PTF instead of Recovery Media
– Overwrites existing files, adds those needed, does not change the recovery point
– Can increase time need for Backup Critical Console Data

9 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Home page has a new look to it


https://www14.software.ibm.com/webapp/set2/sas/f/hmc/home.html

10 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Documentation website


https://www14.software.ibm.com/webapp/set2/sas/f/hmc/resources.html

Why do you think we still


give you the option to get
back-level HMC code?

11 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

POWER code matrix


https://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html

12 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Sign up for notification on critical topics


https://www.ibm.com/systems/support/myview/subscription/css.wss/folders?methodName=listMyFolders

Example Notification

13 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Firmware – Upgrade & Install Instructions


https://www14.software.ibm.com/webapp/set2/sas/f/hmc/home.html
BOOK MARK this page

14 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Select downloads for details

15 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7 R3.3.0 (Released on 5/7/08) upgrade notes


System EM310_060 and higher Minimum levels required for POWER6 systems
Firmware EM320_040 and higher managed by HMC 7.3.3.0. See Power code
EL320_040 and higher matrix for supported and recommended F/W
and HMC code combinations for different
POWER6 systems.
SF240_299 and higher Minimum levels required for POWER5 systems
managed by HMC 7.3.3.0
Upgrade notes Recovery Media Required for upgrade from lower level of HMC
V7.

Upgrading from HMC V6.1.2 or higher HMC V6


is a two-step process. You must first upgrade
via Recovery media to HMC V7 R3.1.0 or
HMC V7 R3.2.0 before you can upgrade to
HMC V7 R3.3.0
Save Upgrade Data Prerequisite task required to preserve existing
HMC configuration
• HMC machine type 7310 with V6 R1.2 or higher may upgrade to HMC V7 R3.1.0 or V7R3.2.0.
• Order MES (Miscellaneous Equipment Specification) 0962 through IBM Sales Representative
or Business Partner.
• HMC code V7.3 Recovery media may also be downloaded, but MES 0962 must still be ordered to
complete the conversion from V6 to V7.

16 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Select Recovery media for upgrades to reset recovery point


Select Update for routine
maintenance

17 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Optional Network install or Upgrade


https://www14.software.ibm.com/webapp/set2/sas/f/hmc/power6/install/v7.Recover_Network.html#PXE

This is an example using


Network images

18 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Upgrade procedure
using DVD media

19 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Back Up Critical Console Data

• What gets Backed Up


• Where can the Back Up be Saved
• Performing a Back Up
• Restoring Critical Data

20 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

What gets Backed Up


ƒ Those files with a timestamp later ƒ In general, Backup Critical Console Data
than the trigger file are candidates for archives:
backup. – Code Changes to the HMC Application(s)
since the last install or recovery upgrade
ƒ The data that is backed up can be
– HMC unique configuration changes since
divided into 2 categories the last install or recovery upgrade
– HMC specific – User Profiles
– User Profiles – Network Configuration
– Roles and Tasks – Service Data
– Network Configuration – Etc.
– Applied Corrective Service Updates
– Platform specific information since the last
– HMC operating system install
– Service packs, eFixes – Managed Systems
– OS Security update – Partition Profiles
– Infocenter – Etc.
– HMC operating system updates applied
since the last install

21 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Save Critical Console Data (Save CCD) vs. Save Upgrade


data
ƒ CCD is not a ‘bootable’ save of the HMC
– In a recovery scenario you need:
– The HMC Recovery CDs (or DVDs) that were used with the install or the last upgrade
– PLUS the Save CCD data.
– Options to Save to:
– DVD-RAM
– FTP server
– NFS server

ƒ Save Upgrade data is different from CCD


– Use this option only before HMC Upgrade
– Saves only user modified System configuration, User Profile & Security info,
Service information

22 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Backup options To back up the HMC (Back-up Critical Console Data), you must be
a member of one of the following roles:
• super administrator
• operator
• service representative
To back up the HMC, do the following:
1. In the Navigation area, click the Licensed Internal Code
Maintenance icon.
2. In the Contents area, click the HMC Code Update icon.
3. Select Back up Critical Console Data.
4. Select an archive option. You can back up to DVD on the
HMC, back up to a remote system mounted to the HMC file
system (such as NFS), or a remote site through FTP.
5. Follow the instructions on the panel to back up the data.

Saving Critical Console data should


be part of your normal system
management process. A backup
should be performed following any
significant change. i.e. upgrades,
updates, security changes – other
user customizations, etc. It will save
all changes to the HMC firmware
and all users modifications.

23 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

V7.3 All HMC management functions from a single menu

24 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Recovery media
Required for version upgrades / Suggested for release upgrades.

ƒ Benefit to install from Recovery CD's


ƒ Becomes the new the recovery point
ƒ Reset the Save CCD data point
ƒ Reduces the volume of data to be save by Save CCD
ƒ Upgrade release level using update.zip files.
ƒ Increased the volume of data to be saved by 'Save Critical Consol Data‘ (save CCD).
ƒ Makes a copy of every object that has changed since the base install or last recovery media
ƒ Example: system ships with V4R1.0 preloaded.
ƒ Update using Update.Zip files to move to V4R2.0 then V4R3.0 then V4R4.0
ƒ Recovery point is still V4R1 Recovery CD’s plus the Critical Console Data after V4R4.0
ƒ Recommended: Upgrade procedure with Recovery media:
ƒ Example: Upgrade from V4R1 to V4R4 the procedure would be:
ƒ1) optional – Save CCD
ƒ2) Save Upgrade Data
ƒ3) Boot from the V4R4 Recovery CD's MH00221
ƒ4) Restore the saved Upgrade Data (will occur automatically as part of the upgrade)
ƒ5) Apply any V4 R4.0 Required Maintenance efix (MH00219 in this example)
ƒ6) Save Critical Console Data
ƒ7) Recovery point is now V4R4 CD’s, plus CCD taken after maintenance

25 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Why CCD backup can take a while…


ƒ Two methods of HMC software
Release Upgrades
– Update via corrective service –
‘Update.zip’ files
– Upgrade with new release Recovery CD’s
(V5R1 and later uses DVD media) For
example upgrading from V5R1 to V5R2 or Critical Console Data Backup
directly to V5R2.1

ƒ Size matters – could be +1GB A Data saved in backup

ƒ Save performance is improved


HMC Recovery DVD set V5R1 HMC Correct Service Update to V5R2.1 User data

HMC Recovery DVD set 5.2.1 User data

Data saved in backup


B

26 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Release Update Matrix


HMC Date Available Upgrade PTF Recovery Media Firmware
V4R1.0 - NA NA NA SF210
V4R1.3 8/3/2004 MH00097 NA
V4R2.0 8/30/2004 MH00107 MH00106 SF220
V4R2.1 9/24/2004 MH00138 NA
V4R3.1 - 11/19/2004 MH00153 MH00191 SF222
V4R4.0 4/8/2005 MH00223 MH00221 SF225
V4R5.0 6/6/2005 MH00268 MH00267 SF230
V5R1.0 - 12/7/2005 NA MH00407 SF235
V5R1.2 4/14/2006 MH00593 NA
V5R2.0 2/24/2006 MH00564 MH00521 SF240

V5R2.1 5/19/2006 MH00594 MH00563


V6R1 08/2006 NA MH00752
V6R1.1 11/2006 MH00839 NA
V6R1.2 02/2007 MH00915 MH00919
V6R1.3 09/2007 MH01012 NA

V7.310 06/2007 NA MH00998 (POWER6)


V7.310 SP1 09/2007 MH01042 NA EM310
V7.320 12/2007 MH01060 MH01059
V7.320 SP1 03/2008 MH01084 NA EM320 or EL320

EM320 or EL320 @
V7.330 04/2008 NA MH01097 sp 076
V7.330 SP1 06/2008 MH01105 NA And EH330

27 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Home page for Power Systems Firmware


http://www14.software.ibm.com/webapp/set2/firmware/gjsn

28 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

POWER6 Server Firmware: Update Policy Set to HMC


http://www-912.ibm.com/s_dir/slkbase.nsf/ibmscdirect/E58D7BBF0EAC9A2786256EAD005F54D8

29 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Network Connection options


Single HMC to one Power System – valid on POWER5 5+ and POWER6
systems with a single FSP

ƒ Direct Ethernet cable from HMC Eth0 to HMC0 or HMC1 port on the FSP (Flexible Service processor)
ƒCable can be a standard Ethernet cable or crossover cable
ƒ Configure DHCP services on the HMC or assign ‘static’ IP addresses:
ƒ HMC and Service processor
ƒ FSP default configuration is DHCP client, however, each port has a predefined default IP addresses
POWER 5 default FSP addresses
192.168.2.147 for HMC0 port 192.168.3.147 for HMC1 port
POWER6 default FSP addresses
FSP-A:
168.254.2.147 for HMC0 port 168.254.3.147 for HMC1 port
FSP-B
168.254.2.146 for HMC0 port 168.254.3.146 for HMC1 port
HMC-0

FSP
HMC-1

HMC to FSP is always SL-Encrypted


HMC to a single server, no external switch is required

30 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

POWER5 570 with “optional” redundant FSP and single HMC


NOTE: POWER6 570 – 5-way or larger, automatically has redundant FSP
The option for dual service processors on the POWER5 570 was introduced in October, 2005. Dual
service processor support is available only on the 5/8 way or larger 570 systems and requires an
HMC to manage this. The HMC and server firmware must be on minimum HMC code level and
server firmware level, and HMC must be correctly networked to support dynamic FSP failover.

HMC0 Service
HMC1 Processor P5 570-A
Switch
Ethernet cable

HMC1 Service
HMC0 Processor

The HMC must attach to both FSPs on the 570 using an Ethernet switch or
hub. Typically this would be HMC0 ports. The HMC can provide DHCP
services for IP address assignment or static IP addressing may be used
Attach a standard Ethernet cable (or cross-over cable) from HMC1 port on the
first service processor directly to HMC1 port on the second service processor.
This is required for failure detection and dynamic failover.
POWER6 570 two enclosure (5/8 way or larger) comes standard with dual
FSPs and the HMC1 to HMC1 connecting cable is no longer needed.

31 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Redundant HMC configuration – using a central control center


In this example, HMC1 will manage 570-A, attaching to HMC0-Port on each of the two Service Processors
HMC2 will manage 570-B, attaching to HMC0-Port on each of the two Service Processors
HMC3 will manage 570-A and i570-B attaching to HMC1-Port
HMC1
on each of the four Service Processors
DHCP-SUBNET-1.1.1.1

570-A
Switch
HMC0
FSP
HMC1

HMC0
FSP
HMC3 HMC1
Intranet
DHCP-SUBNET-
Network
Switch
HMC2
3.3.3.3
For a WAN connection
use Static IP DHCP-SUBNET-2.2.2.2
addressing

570-B
Switch
HMC0
FSP
HMC1

HMC0
FSP
HMC1

32 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC1 will manage the local 570-A in LA, attaching to HMC1-Port on each of the two Service Processors using DHCP. It will
provide redundant management of 570B in NYC using static/assigned IP addresses within the the corporate intranet
HMC2 will manage the local 570-B in NYC attaching to HMC1-Port on each of the two Service Processors using DHCP. It will
provide redundant management of 570A in NYC using static/assigned IP addresses within the corporate intranet

WebSM V6 or earlier HMC Open Network - Static IP Address – i.e - 12.34.56.xx


LA Center
remote management
Private – HMC Managed DHCP-SUBNET-192.168.x.x

HMC1

Switch 570-A
Seattle HMC1
Static IP Address – i.e - 12.34.56.xx FSP
HMC2

Corporate Static IP Address – i.e - 12.34.56.xx


HMC1
FSP
HMC2
network

Open Network - Static IP Address – i.e - 12.34.55.xx NYC Center

Private – HMC Managed DHCP-SUBNET-172.17.x.x


Miami
HMC2

Switch 570-A
HMC1
Static IP Address – i.e - 12.34.55.xx FSP
HMC2
Browser V7 HMC
remote HMC1
management Static IP Address – i.e - 12.34.55.xx FSP
HMC2

33 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC (V6.1.3 or earlier) Setup Wizard

34 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC (V7.3xx) Welcome page – Browser based interface

HMC V7.xxx provides a


Browser based user interface.
Very user friendly, multiple
paths to various features and
functions, i.e.:
Status indicators

Easy (float over)


access to HMC code
level details

35 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Start with the POWER Systems Hardware Info center


http://publib.boulder.ibm.com/infocenter/systems/scope/hw/index.jsp?topic=/iphad/f7310cr4hmc.htm

Configuration Checklist

http://http://publib.boulder.ibm.com/infocenter/eserver/v1r3s/index.jsp?topic=/iphai/configchklst.htm
36 © 2008 IBM Corporation
IBM Power Systems - System p Virtual User Group

HMC Electronic Service Agent: Connections options

Prior to V7.3.3.0,
the Call-Home
setup was included
in the Guided setup
wizard. Now a
separate wizard.

37 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Electronic Service Agent: Connections options cont.

38 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

LPAR Service and Support Configuration Options


http://publib.boulder.ibm.com/infocenter/systems/scope/aix/index.jsp?topic=/rzaji/service_agent.htm

Basic set up

39 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC provides multiple User Interfaces and Access


ƒ Local GUI (on HMC)
– Starts automatically whenever the HMC is started
– Requires user login prior to access
– HMC V7.310 introduced Browser based user interface
– (eliminates need for WebSM for remote access & management)
ƒ Remote GUI (on network PC)
– Uses an installable standalone remote client application (WebSM Client)
– Windows (NT,XP) or Linux
– Downloadable as an installable application directly from the HMC, via web browser
– Also supported from one HMC to another, also to/from AIX 5L
– Can be SSL secured through public/private key files
– Generate on the HMC, and transfer to client systems

ƒ Local Command Line (on HMC)


– Launched from the HMC desktop
– Linux based commands - Restricted to a set of supported HMC commands

ƒ Remote Command Line (on network PC)


– Accessed through encryption-protected Secure Shell (SSH)
– Key files can be set up and exchanged to avoid password prompts
– Very useful for automation and scripting without human intervention

40 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V6 and earlier - Remote GUI management requires WebSM:


(Web-based System Manager)
ƒ Two WebSM clients available:
– Web-based System Manager Remote Client
– Web-based System Manager Remote Client for Java Web Start
– Either client works the same after installation

ƒ Remotely access your HMC by installing WebSM client on your PC workstation


ƒ WebSM client allows you to manage your system from virtually anywhere
– Up to 5 remote clients can be logged in simultaneously

ƒ The HMC and WebSM can negotiate SSL encryption for additional security
ƒ Some tasks not available using the remote client
– These tasks include restarting the HMC interface and configuring System Manager Security for
certificate authority or viewing overview and status information.
– Some of these functions are available remotely using SSH command line interface
– V7.310 remote management resolved some of these limitations, such as rebooting the HMC

HMC V7 and after - Remote GUI access via Secure Browser


- WebSM is no longer needed or supported.

41 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

WebSM client - http://HMC Name/remote_client.html


Example to download WebSM client: http://myHMC.IBM.com/remote_client.html

42 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Starting the WebSM Remote Client

Desktop icon

Web-based System Manager Remote Client.lnk

43 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V6 and earlier GUI Interface – using WebSM

Manage HMC
configuration, users,
services, ...

Guide setup wizard


and online documents

Update your Licensed


Internal Code

Manage your servers


and partitions

Service tools to
analyze and repair

Set up security for


remote GUI access

44 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Remote access to V7 HMC using most any standard browser


• Secure HTTPS
• Logon window can be customized
• Managed systems status indicators

45 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7.3xx User interface changed to Browser based

Some new
features

More to come on Data replication

46 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

V7R3.x.x Rollout

ƒ Overview
– General HMC Function Set
– V7R3.1.0 Content
– V7R3.2.0 Content
– V7R3.3.0 Content

ƒ Details
– Selected 3.1 functions
– Selected 3.2 functions
– Selected 3.3 functions

47 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC R7V3.1.0 – Web UI, Quick Overview


ƒ Motivation / Strategy for new GUI:
– Real http / https GUI – no need for WebSM interface
– Leverage modern, familiar UI paradigms and technologies
– Unify HMC family of user interfaces
– z,p,i,Storage and Integrated Virtualization Manager (IVM)
– Consistent User Experience across HMC and IBM Web UI family

ƒ Navigation
– Simplification – some tasks rearranged
– Improving task discovery
– Reduce task depth
– Consistent task placement
– Consistent task categorization
– Display more information in main views
– Simplify complex functional areas
– Task list available multiple ways (Task list, Tasks button, right click)

ƒ New Functions
– Status Bar
– View Users logged on
– Disconnect / Reconnect
– Custom Groups
– Ability to customize which columns are displayed
– Table filtering and sorting
– Replaces tree structure

48 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC R7V3.1.0 – Web UI, Browser Support


ƒ Remote access disabled by default
– Task to enable/disable remote access
– User Profiles task to enable for specific users

ƒ Security
– Certificate based SSL security for encrypted
communications
– Certificate Management task

ƒ Browser Compatibility
Linux Windows
Mozilla Firefox 1.5.x + 1.5.x +
Microsoft Internet Explorer 6.x +

49 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.1.0 – Integrated Virtual Ethernet (aka HEA)

ƒ What’s in a Name
– Integrated Virtual Ethernet (IVE) – External name in marketing & other documentation
– Host Ethernet Adapter (HEA) – Name used on user interfaces

ƒ New Hardware capability with all POWER6 servers except 595


– Provides accelerated Ethernet connectivity
– Essentially, a system with HEA has several integrated Ethernet adapters, called logical ports.
– Can be assigned to one or more partitions.
– Integrated on most POWER6 system (all but the POWER6 595)
– Several variations of physical, external ports
– Dual 1 Gbit copper, supporting 10BASE-T, 100BASE-T, 1000BASE-T
– Quad 1 Gbit copper , supporting 10BASE-T, 100BASE-T, 1000BASE-T
– Dual 10 Gbit fiber, supporting 10GBASE-SR or 10GBASE-LR
– Logical Ports
– Up to 32 logical ports, but can also be configured as 1, 2, 4, 8, 16 logical ports
– Controlled by parameter called “Multi-Core Scaling Value”

50 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.1.0 – Integrated Virtual Ethernet (aka HEA)

51 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.1.0 – Shared Dedicated Capacity - before


200
175 – Consider a 2-way server with 1-way dedicated
partition with variable workload (between 0%
150 and 100%).
125 0.5 Uncapped 2 – Add two evenly weighted 1-way uncapped
0.5 Uncapped 1 partitions that are CPU bound.
100
Wasted Dedicated – Each uncapped partition will share the
75 1-way Dedicated remaining physical processor even though each
50 can consume an entire processor.

25 – The excess capacity on the


0 dedicated processor is wasted.

200
175
– With the new support, dedicated partitions with
150 unused cycles will donate those unused cycles to
0.5 Uncapped 2 the uncapped partitions.
125 – Donated cycles do not increased a (shared
0.5 Uncapped 1 processor) partition’s entitled capacity.
100
Wasted Dedicated – The uncapped partitions will share all the available
75 1-way Dedicated cycles:
50 – They consume an entire processor if available
(when dedicated at 0%) and
25 – Split a processor when the dedicated processor is
fully utilized.
0 – The total processor capacity in the system is better
utilized.
– Dedicated proc LPs still maintain their perf
characteristics and predictability

52 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC R7V3.1.0 – HMC Management Enhancements


ƒ New monhmc command
– Can gather disk space usage and system
memory usage
– Some specific processes on the HMC are
monitored

ƒ Clickable SRCs
– The new browser-based interface allows a user
to click on an SRC or Progress Code and pop-up
a description of the SRC.

ƒ Data replication
– This is an application from System z HMC that
allows customer call-home configuration to be
replicated between HMCs

ƒ 3rd party certificates


– A customer can now install their own 3rd party
certificates to use with remote connections to the
HMC.

ƒ Selectable VPN
– Allow choice of IP interface to use for the VPN
connection back to IBM.

53 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Selectable SRC for more detail information

54 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Power HMC – V7R3.2.0 Content Overview


ƒ Server Management
– Discovery / Inventory of I/O Devices (aka Hardware Discovery)
– Enhanced System Plan information
– Uses Hardware Discovery (item above) and
– OS round-tripping
– Concurrent GX+ adapter add & cold repair for L4 / ML
– Redundant Service Processor for ML – Enabled at EM320
– Energy Scale (Active Energy) Support
– Scheduled Operations for Utility Capacity on Demand

ƒ Virtualization Management
– Partition Mobility (between servers managed by same HMC)
– Multiple Shared Processor Pool Support

ƒ HMC Management
– User Authentication via Kerberos
– Enhanced HMC Monitoring
– Ongoing HMC Performance Improvements

55 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Hardware Discovery Overview


This function requires 320 level system firmware (or newer).
ƒ Functional improvements:
– Provides capability to collect I/O device information before partitions and OS’s
are configured or booted to make configuration easier.
– Provides HMC users more detailed view of slot contents especially child
devices.
– Can view storage device information (what’s on the SCSI bus).
– Can view Ethernet MAC addresses and FibreChannel WWN’s via HMC.
– Can even be done before partitions are configured.
– Provides more detailed inventory to System Plan Tool to verify system
configuration and validate customer orders.

ƒ Hardware Discovery can be performed three ways


1. Initial Power On (after system leaves IBM): Hardware Discovery performed
automatically.
• Only when system is in non-partitioned mode and IO data hasn’t already been collected.
2. New Power On “Hardware Discovery” option.
• I/O data collected for both assign and unassigned I/O.
• Existing LPs booted into Hardware Discovery; later returned to (bootable) state.
• Special LP used to collect I/O data for unassigned I/O.
• System left in Operating state (no auto-boot of partitions).
3. Is initiated automatically by System Plans tool. (see next function)

56 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – HW Discovery, New Power On


ƒ User initiated Hardware Discovery from the HMC.
ƒ System Powers On and collects IOR data utilizing service
partition.
ƒ System left in “Operating” state upon completion.
ƒ Partition auto-boot settings not enforced on this type of power
on (are persisted for future).
When to use this option:
ƒ After repair and verify.
ƒ After attaching new expansion unit.
ƒ After firmware upgrade.

Checkbox
appears with
managed
systems that
are Hardware
Discovery
Capable

57 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – HW Discovery, User Visibility


ƒ IOR Collection LPAR
– Automatically creates service
partition.
– Hardware Discovery state.
– Persist for 1-20 min.
– Deleted upon completion.

ƒ Existing LPAR

– Hardware Discovery state.


– Returned to previous
(bootable) mode upon
completion.

ƒ New tab under Slot Details


launched from
System/Partition/Partition
Profile Properties I/O table.
ƒ High level view intended to
be more of a “Device
Manager” for
administrators.

58 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Create System from HMC – (export / import SPT)

The System Planning Tool (SPT) is free, it runs on a stand-alone workstation


and enables the user to design a new system. Select the Machine Type and
Model, processors, memory and IO hardware, and design the partitions. This can
be sent to IBM order processing and manufacturing can build to the plan
specifications. The SPT output can also be imported by the HMC and the
DEPLOY option can automatically create the partitions.
http://www-304.ibm.com/systems/support/tools/systemplanningtool/
59 © 2008 IBM Corporation
IBM Power Systems - System p Virtual User Group

Create a System plan from the HMC

Summary information:
Partitions, processors, shared
pools, memory, memory page-
size, and attached I/O
towers… Now we can see the
devices details below the IOA

60 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC Generated System Plan at


V7.330, on POWER6 can see more
of the device level details

61 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Hardware Discovery CLI

ƒ lssyscfg –r sys displays new cec capability for IOR,


hardware_discovery_capable

ƒ chsysstate –r sys includes new option onhwdisc

ƒ lssyscfg –r lpar displays new lpar definition state for IOR in the lpar_env
and lpar_type attribute.

ƒ lssyscfg –r lpar displays new VSP definition state for IOR in the state
attribute.

ƒ lssyscfg –r prof displays new lpar definition state for IOR in the lpar_env
attribute.

ƒ lshwres –r io –rsubtype includes new type slotchildren to list IOR slot


information
62 © 2008 IBM Corporation
IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Energy Scale Support

ƒ HMC / ASM provides a small subset of the EnergyScale features.


– Enable / Disable of Power Saver Mode
– Schedule Operation for Power Saver Mode
– Pass through channel for PowerExecutive / Director

ƒ PowerExecutive / Director provides the complete set of


EnergyScale functions.
– PowerExecutive was originally a power management plugin for System x.
It is now the official management console for EngeryScale feature

63 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Energy Scale Capabilities


ƒ HMC displays 2 capabilities for
power saver mode:
– Firmware power saver capable –
Whether the firmware running on the
server is capable for power saver
mode
– Hardware power saver capable –
Whether the hardware supports the
power saver mode.

ƒ HMC User Interfaces:


– GUI (Managed system properties
panel) and
– Command line (lssyscfg -r sys)

64 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Energy Scale, Power Saver Mode


ƒ With power saver capable systems,
users can enter static power save
through the power saver mode
configuration task.
ƒ With power saver mode enable, the
processing utilization reporting might not
be accurate if the OS does not have the
proper support.
ƒ Commands for power saver mode:
chpwrmgmt and lspwrmgmt

ƒ Scheduled Operation for Power Saver


Mode

65 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Shared Processor Pools


Terminology

Active VSPP Pool 0 (default pool) VSPP Pool 1 VSPP Pool 2


Dedicated
Dedicated
Processor
Processor
LP2
LP1
Physical Shared Processor Pool

ƒ One Physical Shared Processor Pool (PSPP) – Spans all VSPPs


ƒ Comprised of all of licensed CPUs that are
– Not assigned to an active dedicated processor partition
– Assigned to active dedicated processor partition that is capable of donation
– Assigned to inactive dedicated processor LPs and system administrator has chosen to
make CPUs available for shared processor work when the LP is powered off
VSPP Capacity
Maximum Pool Capacity Maximum CPU cycles that can be consumed by the partitions in a VSPP (whole number of proc units).

Additional capacity assigned to a virtual pool for the purpose of reserving uncapped cycles for the
Reserved Pool Capacity VSPP’s use. Must be less than Maximum Pool Capacity. Cannot be stolen by the HMC during partition
activation. The units are processor units and for HMC users the granularity is 0.01 processor units.
A computed quantity – not user specified.
This is the number of CPU cycles guaranteed to be available to the set of partitions in the virtual pool. It
is equal to the sum of the entitled capacities of the partitions in the VSPP (pending processor units of the
Entitled Pool Capacity
active and inactive partitions) plus the Reserved Pool Capacity for the VSPP. Must always be Equal to
or smaller than Maximum Pool Capacity. The units and granularity are the same as Reserved Pool
Capacity.

66 © 2008 IBM Corporation


VSPP Management
IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Shared Processor Pools, VSPP Functions


ƒ Prior to this Multiple VSPP support, all shared processor partitions were in the one and only
Virtual Shared Processor Pool, pool 0, also known as the default pool
ƒ VSPP attributes:
ƒ Pool ID – ranging from 0 to 63
ƒ User Defined Name
¾“DefaultPool” for pool 0; can’t be changed
ƒ Reserve Pool Capacity
¾Exception: no Reserve Capacity for pool 0. Can’t be modified, not displayed
ƒ Maximum Pool Capacity
¾Exception: pool 0 varies to cover remainder of PSPP. Can’t be modified, not displayed
¾Maximum value (for non-default VSPPs) is maximum system processor capacity

ƒ VSPP Pool Function


ƒ VSPPs are configured and unconfigured, not created or deleted, by setting Maximum Pool Capacity
ƒ All partitions must be moved out of a VSPP to unconfigure it, i.e. set the Maximum Capacity to zero
ƒ VSPP modifications can be done when the system is in the Operating or Standby states, and take effect
right away
ƒ Initialize Profile Data will initialize all VSPPs, i.e. setting Maximum Pool Capacity and Reserved Pool
Capacity to zero and pool names to the default names, after all the partitions are deleted
ƒ DLPAR Add and Move of shared processing units will fail if target VSPPs Maximum Capacity is
exceeded

67 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Configure Shared Processor Pools

68 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Configure shared pools – edit partition profile or on create

69 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Kerberos Authentication, Overview

1. Remote User
establishes https
session to HMC.

Kerberos KDC
Remote Workstation (Key Distribution
(Web browser) Center) Server
2. User logs in to HMC
3. HMC sends principal name /
(either local or remote)
TGT Request to KDC.
Supplies User ID & PW
4. KDC computes
encrypted TGT
5. TGT returned to HMC using password.

6. Client (HMC) verifies


7. If PW passes, all HMC operations HMC TGT with supplied PW.
proceed as normal. No further
usage of Kerberos / TGT.

System i/p Server(s) . . .

Kerberos TGT (Ticket Granting Ticket)

70 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.2.0 – Kerberos Authentication, Configuration


ƒ HMC supports MIT Kerberos V5.
ƒ Must have hmcsuperadmin authority to configure Kerberos Client IP on HMC.
ƒ Must configure (Extended Network Time Protocol) XNTP correctly and make sure
both i/p HMC's date/clocks are in sync with all Kerberos KDCs.

ƒ Support both manual login as well as SSH (CLI)


ƒ SSH include Kerberos password-less login
– Requires HMC Kerberos service key to be set-up
– Requires client machine to support ssh GSSAPI (Generic Security Services Application
Programming Interface). This is usually found in the *nix ssh configuration file (hint: look for
“GSSAPI” and enable).
– Requires client machine to be configured for Kerberos (i.e., client).
– Requires client to obtain “forwardable” credentials prior is issuing ssh command (e.g., kinit
–f foobar@REALM).
– Ssh supported GSSAPI options:

ssh -o PreferredAuthentications=gssapi-with-mic foobar@<hmc hostname>


ssh -o PreferredAuthentications=gssapi foobar@<hmc hostname>

71 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Option for access authentication:Local, Kerberos, LDAP

72 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.3.0 – User Data Replication


ƒ Data Replication
– Automatically replicate changes made on one HMC
to other HMCs.

ƒ Replication Types
– Peer-to-Peer
– Master-to-Slave
– Multiple masters supported

ƒ Data which can be replicated


– Customer information data
– Administrator, System, Account information
– Group data
– All user-defined group definitions.
– Modem configuration data
– Configure modem for remote support.
– Outbound connectivity data
– Configure local modem to RSF
– Enable an internet connection
– Configure to an external time source
– User Profile data
– User account and access configuration
– Kerberos, LDAP configuration data
– Centralized user authentication server configuration

73 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.3.0 – User Data Replication


‰Data Replication only available via GUI (no CLI)
‰Only hmcsuperadmin users allow to configure
data replication
‰ Enable Data Replication
¾ HMC Management => Manage Data

Select Replication Source


¾ HMC Management =>
Manage Data Replication

Select Replication Data Types


Synchronize the replicated data
Save the configuration

74 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.3.0 – User Replication, Overview


ƒ Data Replication
ƒ Replication Configuration
ƒ User Profile Replication Discussion

75 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

HMC V7R3.3.0 – IPv6 Support, Overview


• IPv6 supported on “internet” connection only.
• Not supported on server connections in 3.3.0.

• UI changes
• Several CLIs updated
• Only one GUI panel changed: under Change Network Settings:
• HMC management->Change Network Settings-> IPv6 Settings tab
LAN Adapters->Details
• “Basic Settings” tab has changed
• New “IPv6 Settings” tab

76 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

For more information


http://publib.boulder.ibm.com/infocenter/systems/scope/hw/index.jsp

77 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Bookmark - the New HMC home page


https://www14.software.ibm.com/webapp/set2/sas/f/hmc/

78 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Support for HMC – on POWER Systems - References


https://www14.software.ibm.com/webapp/set2/sas/f/hmc/resources.html

HMC Version 7 specifically or to HMC for POWER6 and POWER 5 servers in general.
• HMC Version 7 Installation and Configuration Guide for the Hardware Management Console
• HMC Version 7 Operations Guide for the Hardware Management Console and Managed Systems
• HMC Version 7 Cumulative PTF History and Readme
Includes PTF-specific information, including verification information for package downloads and installation.
Also includes general information on HMC Version 7 fixes, enhancements, known issues and education.
• HMC V7 R3.2.0 Command line specification
Manual pages for HMC restricted shell command line.
• HMC V7 R3.1.0 Command line specification
• IBM service strategy and best practices
Various white papers on best practices, including HMC.
• HMC function and hardware prerequisites for HMC Version 7
Lists the supported functions, servers, and HMC hardware for each Code Level for HMC for IBM Systems with
POWER6 processors.
• IBM System Planning Tool
The IBM System Planning Tool (SPT) is the next generation of the IBM LPAR Validation Tool (LVT). It contains
all of the function from the LVT and is integrated with the IBM Systems Workload Estimator (WLE). System
plans generated by the SPT can be deployed on the system by the Hardware Management Console (HMC).
• MIB file for IBM Systems managed by an HMC
File for IBM Systems managed by an HMC. The MIB for HMC defines the Problem Event SNMP traps
generated by the HMC.

79 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Education from IBM Resource Link


https://www-01.ibm.com/servers/resourcelink/hom03010.nsf?OpenDatabase

80 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

Information Centers, Redbooks and more

ƒ The POWER Systems Hardware Info Center


http://publib.boulder.ibm.com/infocenter/eserver/v1r2s/en_US/index.htm
ƒ IBM iSeries™ Hardware Management Console Frequently Asked Questions
– http://www-
1.ibm.com/support/docview.wss?uid=nas148859a914db132a586256f42006003a7&rs=110

ƒ HMC - Support for UNIX servers and Midrange servers


– http://www14.software.ibm.com/webapp/set2/sas/f/hmc/home.html
ƒ HMC Best Practices –
https://www14.software.ibm.com/webapp/set2/sas/f/best/hmc_best_practices.pdf
ƒ iTC class room education –
http://www-03.ibm.com/servers/eserver/iseries/service/itc/educ.html
ƒ LPAR Redbook –
http://www.redbooks.ibm.com/redbooks.nsf/RedpieceAbstracts/sg248000.html?Open

81 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group

The End
Thank You !

82 © 2008 IBM Corporation


IBM Power Systems - System p Virtual User Group
© IBM Corporation 1994-2005. All rights reserved.
References in this document to IBM products or services do not imply that IBM intends to make them available in every country.
The following terms are trademarks or registered trademarks of International Business Machines Corporation in the United States, other
countries, or both:

Trademarks and Disclaimers


AIX DB2 Universal eServer Lotus Power Architecture RS/6000 z/OS
AIX/L DB2 OLAP Server Enterprise Storage Server MQSeries Power Everywhere S/390 zSeries

AIX 5L DataPropagator Hipersockets Notes POWER Hypervisor ThinkPad 400

AIX 5L (logo) Domino IBM OS/400 POWER6 Tivoli i5/OS


AS/400 e business(logo) IBM Virtualization Engine POWER pSeries TotalStorage

AS/400e e(logo)business IBM(logo) POWER4 Quickplace WebSphere

DB2 e(logo)server iSeries POWER5 Rational xSeries

Lotus, Freelance Graphics, and Word Pro are registered trademarks of Lotus Development Corporation and/or IBM Corporation.
Domino is a trademark of Lotus Development Corporation and/or IBM Corporation.
C-bus is a trademark of Corollary, Inc. in the United States, other countries, or both.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.
Other company, product and service names may be trademarks or service marks of others.
Information is provided "AS IS" without warranty of any kind.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and
performance characteristics may vary by customer.
Information in this presentation concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an
endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages.
IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed
to the supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full
text of the specific Statement of Direction.
Some information in this presentation addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery
schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development
activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon
considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual
user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

83 © 2008 IBM Corporation

Anda mungkin juga menyukai