Deep Dive To

OpenStack
Hands On Session
Haim Ateya
Group Manager, EMC

Session Goals
Provide the audience with an hands-on experience on how to operate
OpenStack Cloud Infrastructure based on RDO distribution
Learn how to operate the various OpenStack components, configuration
files, command line usage and log files
Review the different installation types and methods, how to begin and
deploy your First OpenStack Cloud
Provide you with tools and knowledge on how to troubleshoot problems
when they happen, and how to fix them

Before We Begin
Demonstration will be done over an RDO Liberty installation of OpenStack
A pre-installed version of OpenStack server based on PackStack will be used
OpenStack is purely based on Linux and python

To master OpenStack you need to master Linux

To master OpenStack you need to master Python

Troubleshooting is a by product of using OpenStack

Take notes, capture your screens, ask questions

About RDO
Stands for: Redhat Distribution OpenStack
RDO is a community of people using and deploying OpenStack on CentOS,
Fedora, and Red Hat Enterprise Linux
The Open Source version launched by Red hat
Provides stable, documented and secured releases of OpenStack with full
documentation and support tool
RDO To RHOS
Is Like
Fedora To RHEL

Seminar Lab Deployment

Our deployment consist of the following components:
o Distribution: Centos 7.2 installed with latest RDO
o Nodes: 2 physical servers (16G each)
o Compute (hypervisor engine): KVM
o Cinder (block storage): LVM \ iSCSI
o Glance (disk images): Images are stored locally on the disk

Installing OpenStack How ?

Several methods to Install OpenStack:

PackStack basic installer based on python & puppet that works with
configuration files to establish a working cloud instance

TripleO an advanced installer that is aimed at installing, upgrading and

operating OpenStack clouds using OpenStack's own cloud facilities as the
foundations

Manual installation of sources (tar balls) on a machine manually

About PackStack
Based on configuration files
Script that automates the installation process using puppet, RPM and YUM
Server must be pre-configured with all required RPM sources
Divided into sections for each one of the components {key: value}

Control if and where to install Nova, Swift, Glance, Cinder, etc.

Support AIO \ distributed \ Standalone installations

Control users, passwords and tenants

PackStack Installation Process

First install the basic package openstack-packstack
[root@rdo-openstack ~]# yum install openstack-packstack
Resolving Dependencies
--> Running transaction check
---> Package openstack-packstack.noarch 1:7.0.0-0.10.dev1682.g42b3426.el7
installed
--> Finished Dependency Resolution
Dependencies Resolved
Package
Repository

Arch
Size

will be

Version

Installing:
openstack-packstack
noarch
1:7.0.0-0.10.dev1682.g42b3426.el7
openstack-liberty
234 k
Installed:
openstack-packstack.noarch 1:7.0.0-0.10.dev1682.g42b3426.el7
Complete!

PackStack Installation Process

Second Generate a basic configuration file from template
[root@rdo-openstack
[root@rdo-openstack
[root@rdo-openstack
-rw-------. 1 root

~]# packstack --gen-answer-file=/tmp/rdo-openstack.conf

~]#
~]# ls -l /tmp/rdo-openstack.conf
root 46355 Mar 5 14:34 /tmp/rdo-openstack.conf

Then edit the generated conf file using your favorite text editor
[root@rdo-openstack ~]# vim /tmp/rdo-openstack.conf

PackStack Configuration File

Set desired services to install

PackStack Configuration File

Configure keystone User & Password for each \ all services

PackStack Start The Installation

Finally, after all changes are made to the configuration file, start the
installation (hint: better use screen)
[root@lg860 ~]# packstack --answer-file=/tmp/rdo-openstack.conf
Welcome to the Packstack setup utility
The installation log file is available at: /var/tmp/packstack/20160305-151722_Fn1J9/openstack-setup.log
Installing:
Clean Up
[ DONE ]
Discovering ip protocol version
[ DONE ]
Setting up ssh keys
[ DONE ]
^CAdditional information:
* Time synchronization installation was skipped. Please note that unsynchronized time on
server instances might be problem for some OpenStack components.

PackStack
What Could Go Wrong?
Everything.
In case of failure, inspect the following:
o Read log file and understand the error message
o Check network connectivity to all external servers
o Check security enhanced Linux (SELinux) configuration
o Make sure YUM repositories are configured correctly
o Make sure server has enough free disk space

OpenStack Storage Concepts

Ephemeral storage:
o
o
o
o

Persists until VM is terminated

Accessible from within VM as local file system
Used to run operating system and/or scratch space
Managed by Nova

Block storage:
o
o
o
o

Persists until specifically deleted by user

Accessible from within VM as a block device (e.g. /dev/vdc)
Used to add additional persistent storage to VM and/or run operating system
Managed by Cinder

Object storage:
o
o
o
o

Persists until specifically deleted by user

Accessible from anywhere
Used to add store files, including VM images
Managed by Swift

Cinder Block Storage

Cinder virtualizes pools of block storage devices and
provides end users with a self service API to
request and consume those resources without
requiring any knowledge of where their storage
is actually deployed or on what type of device.

Cinder Block Storage

Responsible for block device provisioning of virtual machines
Provides an API for various storage array vendors to manage their block
device and translate commands between Nova and other services
Best used for performance-sensitive scenarios, such as
database storage or expandable file systems

Cinder - Features
Volumes:
o

Persistent R/W Block Storage devices

Can be attached to VMs as secondary storage

Can be root store to boot VMs

Can be attached only to one instance at a time

Keep their state independent of instances

Snapshots:
o

Read-only point in time copy of a volume

Can then be used to create a new instance

Backups:
o

An archived copy of a volume

Cinder Architecture

Cinder Data & Control

Cinder Default Configuration

Default storage backend is LVM (Logical Volume Management)
o Each virtual machine disk is a logical volume (LV)
o Each virtual machine snapshot (volume) is a logical volume (LV)

Default SCSI transport mechanism is iSCSI using target-cli

[root@rdo-openstack ~(keystone_admin)]# lvs cinder-volumes 2>/dev/null
LV
VG
Attr
Data% Meta% Move Log Cpy%Sync Convert
_snapshot-985c2a51-1fd7-4380-8005-3141b1ff3c0c cinder-volumes swi-a-s--a6fd39ae-d4d3-4eb3-a561-2b887c3d9097 0.00
_snapshot-f5fa8bda-ba84-4cf8-8387-9db25680cf92 cinder-volumes swi-a-s--a6fd39ae-d4d3-4eb3-a561-2b887c3d9097 0.00
volume-a6fd39ae-d4d3-4eb3-a561-2b887c3d9097
cinder-volumes owi-a-s---

LSize

Pool Origin

25.00g

volume-

25.00g

volume-

25.00g

Cinder Log Files

All cinder log files are stored under /var/log/cinder/
o /var/log/api.log
o /var/log/backup.log
o /var/log/scheduler.log
o /var/log/volume.log

Start with scheduler.log and volume.log

Cinder Commands
create

Create a volume

delete

Delete a volume

rename

Rename a volume

manage

Manage an existing volume

list

List all volumes

show

Show volume details

snapshot-create

Creates a snapshot

snapshot-delete

Removes one or more snapshots

migrate

Migrates volume to a new host

Glance Image Repository

Provides image repository (store) services to virtual machines
in the OpenStack Cloud.
Provides discovering, registering, and retrieving virtual machine images.
Registered images can be stored in the Object Storage service or in other
locations, such as simple file systems or external Web servers.
Provides an easy to use API to manage the repository

Glance Image Repository

The Glance project provides services for
discovering, registering, and retrieving virtual
machine images."

Glance - Features
Boot from image
Create snapshots from image
Create image from volume

Glance Architecture

Glance Commands
Image-list

List images you can access

Image-create

Create a new image

Image-delete

Delete an image

Image-show

Show a specific image

Image-update

Update an existing image

Image-upload

Upload data for a specific image

member-list

Describe sharing permissions by image

location-add

Add a location (and related metadata) to

an image.

location-delete

Delete a location from an image

Glance Logs Files

All log files are stored under /var/log/glance/
o api.log
o registry.log

Swift Object Storage

Provides an HTTP-accessible storage system for large amounts
of data
Objects are stored as binaries along with stored metadata
Usually stores videos, images, email messages, files, or VM images
Each object is stored with 3 replicas to ensure availability and redundancy of
data
Runs on commodity hardware

Swift Architecture
Proxy Server
The Ring
Storage Policies
Object Server
Container Server
Account Server
Replication
Auditors

Swift Architecture

Swift Commands
delete

delete a container or objects within a container

download

Download objects from containers

list

Lists the containers for the account or the objects for a

container

post

Updates meta information for the account, container, or

object; creates containers if not present

stat

Displays information for the account, container, or

object

upload

Uploads files or directories to the given container

capabilities

List cluster capabilities

tempurl

Create a temporary URL

Nova - Compute
Responsible for instance (VM) life cycle management
Support multiple hypervisors (KVM, Xen, LXC, Hyper-V, ESX)
Exposes a rich API and a set of components that interact with other
OpenStack components such as Glance, Cinder and Swift.

Nova - Components
Restful API nova-api
interface exposed on TCP
port 8774
AMQP message queue
used for RPC
communication
Nova Scheduler handles
hypervisor selection for
instance placement

Nova - Components
Nova compute acts as the
compute agent, interacts
with all relevant hypervisor
APIs to launch image \
guests
Nova Conductor handles DB
access (no-db-compute)
Console auth and proxies
handles guest console services
for novnc, spice, etc.

Nova Flavors
Virtual hardware templates which defines sizes for RAM disk,
number of cores, and so on
Default install provides five flavors
Admin may want to customize depending of the workload and hardware

Nova Commands
list

List active servers

host-list

List all hosts by service

hypervisor-list

List hypervisors

network-list

Print a list of available networks

image-create

Create a new image by taking a snapshot of a running server.

interface-attach

Attach a network interface to a server

interface-dettach

Detach a network interface to a server

migrate

Migrate a server. The new host will be selected by the scheduler

stop

Start the server(s)

start

Start the server(s)

get-vnc-console

Get a vnc console to a server

Libvirt The Virtualization API

A toolkit to interact with the virtualization capabilities of recent
versions of Linux
Libvirt supports the following hypervisors:
o

The KVM/QEMU Linux hypervisor

The Xen hypervisor on Linux and Solaris hosts.

The LXC Linux container system

The VirtualBox hypervisor

The VMware ESX and GSX hypervisors

The Microsoft Hyper-V hypervisor

The IBM PowerVM hypervisor

Libvirt The Virtualization API

Provides a unified API called virsh to manage virtual machines
across different types of hypervisors
Common commands in the libvirt domain:
virsh list

Provides a list of running virtual machines

virsh dumpxml

Provides virtual machine configuration file

virsh capabilities

Returns capabilities of hypervisor / driver

virsh migrate

Performs migration between hypervisors

virsh destroy

Destroy an existing virtual machine

virsh create

Creates a new virtual machine

Neutron Network
Provides networking as a service
Provides Rich Topologies
Technology Agnostic
Extensible
Advanced Services Support
o LBaaS, VPNaaS, FWaaS

Neutron - Features
Support for Overlapping Ips
o Tenant A: 192.168.0.0/24
o Tenant B: 192.168.0.0/24
Configuration
DHCP/Metadata
Floating Ips
Vlan \ GRE VXLAN \ Tunneling

Neutron - Concepts
Ports
Networks
Subnet
VIF (Virtual Interface)
Floating IP (NAT)
External network
Provider Network
Tenenat network

Neutron Architecture

Neutron Topology

Neutron Log Files

All neutron log files are stored under /var/log/neutron/
o dhcp-agent.log
o l3-agent.log
o openvswitch-agent.log
o ovs-cleanup.log
o server.log

Neutron Commands
net-list

List networks that belong to a given tenant

port-list

List ports that belong to a given tenant

router-list

List routers that belong to a given tenant.

subnet-list

List subnets that belong to a given tenant

router-create

Create a router for a given tenant

subnet-create

Create a subnet for a given tenant

firewall-create

Create a firewall for a given tenant

net-create

Create a network for a given tenant

net-gateway-create

Create a network gateway.

Keystone Authentication
Keystone is an OpenStack project that
provides Identity, Token, Catalog and Policy
services for use specifically by projects in the
OpenStack family

Keystone Authentication
Provides authentication and authorization for all openstack components
Interacts with every service in the OpenStack domain and provide
access validation for each service using token validation

Keystone Internals
User: has account credentials, is associated with one or more tenants
Tenant: unit of ownership in openstack, contains one or more users
Role: a first-class piece of metadata associated with many user-tenant pairs.
Token: identifying credential associated with a user or user and tenant
Extras: bucket of key-value metadata associated with a user-tenant pair.
Rule: describes a set

Keystone Token & Access

Initial access User \ Password (get a token)
All access to OpenStack end-points are done using tokens
Token format:

Keystone Token & Access

Access to keystone API is done via the following command:

[root@rdo-cloud ~(keystone_admin)]# cat ~/keystonerc_admin
unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://10.82.55.64:5000/v2.0
export PS1='[\u@\h \W(keystone_admin)]\$ '
export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne

Or using the following command:

[root@rdo-cloud ~(keystone_admin)]# keystone --os-username=admin --os-password=admin
url=http://rdo-cloud:35357/v2.0 token-get
+-----------+---------------------------------------+
| Property |
Value
|
+-----------+---------------------------------------+
| expires
|
2016-03-12T20:42:53Z
|
|
id
| 2c4fe2b486234c7bb23a5b58734a3ca2
|
| tenant_id | 2817825172314573b4be9b5c36943b65
|
| user_id
| a7fe5bc8b48b49fc973fe592ff1effdd
|
+-----------+---------------------------------------+

--os-auth-

Keystone Auth Flow

Keystone Nova Auth Flow

Keystone Log Files

All log files are stored under /var/log/keystone
o keystone.log

Keystone Commands
role-list

List all roles

service-list

List all services in Service Catalog

user-get

Display user details

user-list

List users

tenant-list

List all tenants

endpoint-list

List configured service endpoints

tenant-create

Create new tenant

tenant-delete

Delete tenant

Horizon Dashboard
The OpenStack Dashboard (Horizon) provides a
baseline user interface for managing OpenStack
services."

Horizon - Dashboard
Is "Stateless"
Error handling is delegated to back-end
Doesn't support all API functions
Can use memcached or database to store
Sessions
Gets updated via nova-api polling

Horizon - Dashboard
Django based application that provides access to OpenStack services
Typically deployed as an Apache WSGI application
Leverage well known existing technologies

Bootstrap, jQuery, Underscore.js, Angular, D3.js

Horizon - Internals
2 subprojects
o horizon - generic Django libraries and components to work with RESTbased back-end
o Openstack-dashboard - web app itself, with styles, locale, etc.
Dashboard for each entity (like instances or images) - nested Django app

Horizon - Dashboard

Horizon - Logs
Log files are stored under Apache folder /var/log/httpd/
o access_log
o error_log
o horizon_access.log
o horizon_error.log

OpenStack - Configuration Files

Configuration files are usually stored under /etc/
Example Cinder configuration files are stored under /etc/cinder/
[root@rdo-cloud ~(keystone_admin)]# ll /etc/cinder
total 136
-rw-r----- 1 root
cinder
2018 Mar 9 21:32 api-paste.ini
-rw-r----- 1 root
cinder 121028 Mar 10 11:03 cinder.conf
-rw-r----- 1 root
cinder
4623 Dec 22 14:59 policy.json
-rw-r----- 1 root
cinder
973 Dec 22 14:59 rootwrap.conf
drwxr-xr-x 2 root
root
29 Mar 9 21:32 rootwrap.d
drwxr-xr-x 2 cinder root
6 Dec 22 17:54 volumes

System service must be reloaded for configuration change to take affect

OpenStack - Service Management

Services in RHEL \ Fedora \ Centos are managed by systemctl
2 dedicated tools:
o openstack-service control OpenStack service states
o openstack-status validate OpenStack service state
[root@rdo-cloud ~(keystone_admin)]# openstack-service status neutron
MainPID=10533 Id=neutron-dhcp-agent.service ActiveState=active
MainPID=10534 Id=neutron-l3-agent.service ActiveState=active
MainPID=10535 Id=neutron-metadata-agent.service ActiveState=active
MainPID=10536 Id=neutron-openvswitch-agent.service ActiveState=active
MainPID=10537 Id=neutron-server.service ActiveState=active

Troubleshooting OpenStack ?

Troubleshooting Guidelines
Start with the following questions:
o What triggered the failure ?
o Which of the components are involved ?
o Is the issue repeatable or its a one time occurrence ?
o Which logs should I start with ?
o Search the web \ forums \ chat rooms \ mailing lists

Useful Resources
RDO - https://www.rdoproject.org/
Miranits - https://www.mirantis.com/
OpenStack official site - http://www.openstack.org/
o Docs
o Training
o Learning
o Events
Ubuntu OpenStack - http://www.ubuntu.com/cloud/openstack

Q & A

Q & A

The End (part 4)