Tampa Performance3

Performance Tuning and
Analysis of Red Hat

Enterprise Linux 6 and 7
Jeremy Eder
Principal Software Engineer
March 13, 2014
RED HAT CONFIDENTIAL | JEREMY EDER
Agenda: Performance Analysis of RHEL6/7
Performance Engineering Overview
Performance Analysis Utilities
Tuned
NUMA Tuning
Power Management
Performance Engineering
Overview
Micro-Benchmarks
Applications/Benchmarks
Red Hat Performance Engineering
Benchmarks code path coverage
CPU linpack, lmbench
Memory lmbench, McCalpin Streams
Disk IO Iozone, aiostress scsi, FC, iSCSI
Filesystem IOzone, postmark ext3/4, xfs. gfs2,gluster
Network Netperf 10 Gbit, 40 Gbit IB, PCI3
Bare Metal, RHEL6/7 KVM
White box AMD/Intel, with our OEM partners
Red Hat Confidential
Performance Projects / Tooling
RHEL6.5 numad tuna, and tuned
Tuna used to bind IRQ's / real-time like isolation
Profiling challenges
Data address profiling (cache-2-cache detection),
providing:
the hottest contended cachelines
the process names, addresses, pids, tids causing that contention
the cpus they ran on,
and how the cacheline is being accessed (read or write)
Performance Optimization
Out-of-the-box
Performance Tuning Automation
Automatic Tuning
Manual Tuning
Tuned
N/A
Transparent Hugepages
Static Hugepages
numad
NUMA Pinning (numactl)
irqbalance
IRQ Pinning (tuna)
RHEL7
numa_balancing
But...what if we have a problem ?
Automatic not enough...
Need to eek out the last X percent
10
Need Determinism
11
Need Determinism
OK...here's where it get's fun!
12
Overview of Performance
Analysis Utilities
perf
13
perf
Userspace tool to read CPU

counters and kernel tracepoints
14
perf list
List counters/tracepoints available

on your system
15
perf list
grep for something interesting,

maybe to see what numabalance is
doing ?
...more on numabalance later...
16
perf top
System-wide 'top' view of busy

functions
17
perf record
18
Record system-wide (-a)
perf record

A single command
19
perf record

A single command
An existing process (-p)
20
perf record

A single command
Add call-chain recording (-g)
21
perf record

A single command
Only specific events (-e)
22
perf record

A single command
Only specific events (-e)
23
perf report
/dev/zero
24
perf report
/dev/zero
oflag=direct
25
perf diff
Compare 2 perf recordings
26
perf probe (dynamic tracepoints)
Insert a tracepoint on any function...

Try 'perf probe -F' to list possibilities
27
Cool, what else ya got ?
28
Overview of Performance
Analysis Utilities
Performance Co-Pilot (PCP)
29
Performance Co-Pilot (PCP)
(Multi) system-level performance

monitoring and management
30
pmchart graphical metric plotting tool
31
Can plot myriad performance statistics
Recording mode allows for replay
32
i.e. on a different system

Record in GUI, then
# pmafm $recording.folio
Recording mode allows for replay
33
i.e. on a different system

Record in GUI, then
# pmafm $recording.folio
Ships with many pre-cooked views...for example:
ApacheServers: CPU%/Net/Busy/Idle Apache Servers
Overview: CPU%/Load/IOPS/Net/Memory
What do these views look like ?
34
Performance Co-Pilot Demo Script
Tiny script to exercise 4 food groups...
# CPU
/root/pig -s 5
# DISK
dd if=/dev/zero of=/root/2GB count=2048 bs=1M oflag=direct
# NETWORK
netperf -H lab7 -l 5
# MEMORY
/root/pig -m 16384 -l sleep -s 5
35
CPU %
Load Avg
IOPS
Network
Memory
Allocated
36
collectl mode
CPU
37
collectl mode
CPU
IOPS
38
collectl mode
CPU
IOPS
NET
39
collectl mode
CPU
IOPS
NET
MEM
40
atop mode
41
Questions so far ?
42
NUMA Tuning
Discovery
43
Visualize NUMA Topology: lstopo

NUMA Node 0
How can I visualize my system's NUMA

topology in Red Hat Enterprise Linux?
https://access.redhat.com/site/solutions/62879
44

NUMA Node 0
NUMA Node 1

45

NUMA Node 0
PCI Devices
NUMA Node 1

46
Visualize NUMA Topology: lscpu

# lscpu
Architecture:
...
CPU(s):
On-line CPU(s) list:
Thread(s) per core:
Core(s) per socket:
Socket(s):
NUMA node(s):
...
NUMA node0 CPU(s):
NUMA node1 CPU(s):
47
x86_64
16
0-15
1
8
2
2
0-7
8-15
Logical Cores/HT
Visualize NUMA Topology: lscpu

# lscpu
Architecture:
...
CPU(s):
On-line CPU(s) list:
Thread(s) per core:
Core(s) per socket:
Socket(s):
NUMA node(s):
...
NUMA node0 CPU(s):
NUMA node1 CPU(s):
48
x86_64
16
0-15
1
8
2
2
0-7
8-15
Logical Cores/HT
NUMA Node/Core Map
NUMA Topology and PCI Bus
Install adapters close to the CPU that will run the

performance critical application.
When BIOS reports locality, irqbalance handles
NUMA/IRQ affinity automatically.
# lstopo-no-graphics |egrep 'NUMA|eth4'
NUMANode L#0 (P#0 144GB)
NUMANode L#1 (P#1 144GB)
Net L#10 "eth4"
49
RHEL NUMA Scheduler
RHEL6
numactl, numastat enhancements
numad usermode tool, dynamically monitor, auto-tune
RHEL7 numabalance
Enable / Disable
50
sysctl kernel.numabalancing={0,1}
How to manage NUMA manually - Checklist
Checklist
Tool
Research Topology
lstopo/lscpu
Make a resource plan
cgroups, numactl
Consider I/O
irqbalance/PCI Bus
Virtualization
numatune/numad
51
Checklist
Tool
Research Topology
lstopo/lscpu
cgroups, numactl
Consider I/O
irqbalance/PCI Bus
Virtualization
numatune/numad
52
Checklist
Tool
Research Topology
lstopo/lscpu
cgroups, numactl
Consider I/O
irqbalance/PCI Bus
Virtualization
numatune/numad
53
Checklist
Tool
Research Topology
lstopo/lscpu
cgroups, numactl
Consider I/O
irqbalance/PCI Bus
Virtualization
numatune/numad
54
Checklist
Tool
Research Topology
lstopo/lscpu
cgroups, numactl
Consider I/O
irqbalance/PCI Bus
Virtualization
numatune/numad
55
NUMA Tools: numastat
Completely rewritten for RHEL6.4

Per-node /proc/meminfo
Backwards compatible
56
NUMA Tools: numastat
Completely rewritten for RHEL6.4

Per-node /proc/meminfo
Backwards compatible
57
numastat: compatibility mode (old)

# numastat
numa_hit
numa_miss
numa_foreign
interleave_hit
local_node
other_node
58
node0
node1
77587739
0
0
30254
69302710
8285029
131990042
0
0
30099
129511360
2478682
numastat: per-node meminfo (new)

# numastat -mczs
MemTotal
MemFree
MemUsed
Active
FilePages
Active(file)
Active(anon)
AnonPages
Inactive
Inactive(file)
Slab
SReclaimable
SUnreclaim
AnonHugePages
59
Node 0
-----65491
60366
5124
2650
2021
1686
964
964
341
340
380
208
173
134
numastat: per-node meminfo (new)

# numastat -mczs
MemTotal
MemFree
MemUsed
Active
FilePages
Active(file)
Active(anon)
AnonPages
Inactive
Inactive(file)
Slab
SReclaimable
SUnreclaim
AnonHugePages
60
Node 0 Node 1 Total

------ ------ -----65491 65536 131027
60366 59733 120099
5124
5803 10927
2650
2827
5477
2021
3216
5238
1686
2277
3963
964
551
1515
964
550
1514
341
946
1287
340
946
1286
380
438
818
208
207
415
173
230
403
134
236
370
NUMA Tuning
numad
61
NUMA: Process Scheduler Behavior
Scheduler distributes load evenly across all cores
Maintains responsiveness
Optimizing for CPU utilization
Tries to use idle CPUs, regardless of where process

memory is located
BUT!
Using remote memory degrades
performance!
62
Red Hat has taken 2 approaches for customers:

Short Term (RHEL6.4)
Long Term (RHEL7)
Userspace solution
numad
63
Red Hat has taken 2 approaches for customers:

Short Term (RHEL6.4)
Userspace solution
numad
64
Long Term (RHEL7)

In-Kernel Solution
numabalance
What does it look like ?
65
Effect of Automatic NUMA Balancing Software

# numastat -c pig (default scheduler non-optimal)
Before numad
PID
2578
2579
2580
2581
66
(pig)
(pig)
(pig)
(pig)
Node 0 Node 1
2123 11878
1988 12013
14000
1
1981 12020
Effect of Automatic NUMA Balancing Software

# numastat -c pig (default scheduler non-optimal)
Before numad
PID
2578
2579
2580
2581
PID
2578
2579
2580
2581
67
(pig)
(pig)
(pig)
(pig)
(pig)
(pig)
(pig)
(pig)
Node 0 Node 1
2123 11878
1988 12013
14000
1
1981 12020
Node 0 Node 1
14000
0
0 14000
14000
0
0 14000
After numad
Effect of numad/numabalance
Automatic NUMA Balancing - NUMAD
14000
12000
M egabytes Per NUM A Node
10000
8000
6000
numad begins
numad done
4000
2000
0
1
10
11
Tim e (sec onds)
NODE-0-MB
68
NODE-1-MB
12
13
14
15
16
Questions on NUMA ?
69
tuned
70
What is tuned ?
Tuning profile delivery mechanism

Red Hat ships tuned profiles that
improve performance for many
workloads...hopefully yours!
71
Yes, but why do I care ?
72
Tuned: Storage Performance Boost
Larger is better
73
Tuned: Network Latency Performance Boost

C-state lock improves determinism, reduces jitter
Latency (Microseconds)
250
200
C6
C3
C1
150
100
50
0
Max
Time (1-sec intervals)

74
C0
What do the guts look like ?
75
tuned Profile Summary: RHEL6

Tunable
default
enterprisestorage
sched_min_
granularity_ns
4ms
10ms
10ms
10ms
10ms
sched_wakeup_granula
rity_ns
4ms
15ms
15ms
15ms
15ms
dirty_ratio
20% RAM
40%
10%
40%
40%
dirty_background_ratio
10% RAM
5%
swappiness
60
10
30
I/O Scheduler (Elevator)
CFQ
deadline
deadline
deadline
Filesystem Barriers
On
Off
Off
Off
CPU Governor
ondemand
performance
Disk Read-ahead
virtual-host virtualguest
latencyperformance
deadline
deadline
performance
performance
4x
Disable THP
Yes
CPU C-States
Locked @ 1
76
throughputperformance
What's happening to Tuned for RHEL7 ?
77
Tuned: Updates for RHEL7
78
Installed by default!
Installed by default!
Profiles automatically set based on install type:
79
Desktop/Workstation: balanced
Server/HPC: throughput-performance
80
Re-written for maintainability and extensibility.
81
Configuration consolidated to single

tuned.conf file
82
Configuration is now consolidated a single

tuned.conf file
Optional hook/callout capability
83

tuned.conf file
Adds concept of Inheritance (just like
httpd.conf)
84

tuned.conf file
Adds concept of Inheritance (just like
httpd.conf)
Profiles updated for RHEL7 features and
characteristics
Tuned: Profile Inheritance
Parents
throughput-performance
balanced
latency-performance
Children
network-throughput
desktop
virtual-host
virtual-guest
85
network-latency
Tuned: Profile Inheritance
Parents
balanced
latency-performance
Children
network-throughput
desktop
network-latency
Your-DB
Your-Middleware
virtual-host
virtual-guest
Your-Web
86
What do the guts look like ?
87
Tuned: Throughput Profiles - RHEL7

Tunable
Units
Balanced
Inherits From/Notes
sched_min_ granularity_ns
nanoseconds
auto-scaling
10000000
sched_wakeup_granularity_ns
nanoseconds
3000000
15000000
dirty_ratio
Percent
20
40
Percent
10
10
swappiness
Weight 1-100
60
10

Filesystem Barriers
network-throughput
deadline
Boolean
CPU Governor
Enabled
ondemand
Disk Read-ahead
KB
128
Disable THP
Boolean
Enabled
Energy Perf Bias
normal
performance
4096
performance
kernel.sched_migration_cost_ns
nanoseconds
500000
min_perf_pct (intel_pstate only)
Percent
auto-scaling
tcp_rmem
Bytes
auto-scaling
Max=16777216
tcp_wmem
Bytes
auto-scaling
Max=16777216
88
udp_mem
PagesRED HAT CONFIDENTIAL

auto-scaling | JEREMY EDER
Max=16777216
100
Tuned: Latency Profiles - RHEL7

Tunable
Units
Balanced
latency-performance
network-latency
Inherits From/Notes
latency-performance
nanoseconds
auto-scaling
nanoseconds
3000000
dirty_ratio
percent
20
10
percent
10
swappiness
Weight 1-100
60
10

Filesystem Barriers
10000000
10000000
deadline
Boolean
CPU Governor
Enabled
ondemand
performance
N/A
No
CPU C-States
N/A
Locked @ 1
Energy Perf Bias
normal
performance
Disable THP
Boolean
nanoseconds
min_perf_pct (intel_pstate only)
percent
net.core.busy_read
microseconds
50
net.core.busy_poll
microseconds
50
net.ipv4.tcp_fastopen
Boolean
89
kernel.numa_balancing
N/A
Yes
Boolean
5000000
100
Enabled
Disabled
Tuned: Virtualization Profiles - RHEL7

Tunable
Units
Inherits From/Notes
nanoseconds
10000000
nanoseconds
15000000
dirty_ratio
percent
40
percent
10
swappiness
Weight 1-100
10
virtual-host
virtual-guest
30
5

Filesystem Barriers
Boolean
CPU Governor
Disk Read-ahead
performance
Bytes
Energy Perf Bias
4096
performance
nanoseconds
5000000
min_perf_pct
(intel_pstate only)
90
percentRED HAT CONFIDENTIAL | JEREMY100

EDER
30
RHEL tuned package

Available profiles:
- balanced
- desktop
- latency-performance
- myprofile
- network-latency
- network-throughput
- throughput-performance
- virtual-guest
- virtual-host
Current active profile: myprofile
91
Power Management
92
CPU Tuning: C-states (idle states)
Power Saving
Performant
C-state Impact on Jitter
Latency (Microseconds)
250
200
C6
C3
C1
150
100
50
0
Max
Time (1-sec intervals)

93
C0
Power Consumption RHEL6 vs RHEL6@C0
94
C-state lock increases power draw over out of the box

Test
Efficiency [Wh] % Diff
Kernel Build
+12.5%
Disk Read
+32.2%
Disk Write
+25.6%
Unpack tar.gz
+23.3%
Active Idle
+41%
Use cron to set latency-performance tuned profile when necessary.

Or use BUSY_POLL
Set tuned profile in application init script
Turbostat shows P/C-states on Intel CPUs
turbostat in RHEL6.4, cpupowerutils package or kernel-tools in Fedora/RHEL7
Default
pk cor CPU
%c0
GHz
TSC
%c1
%c3
%c6
%c7
0.24 2.93 2.88
5.72
1.32
0.00
92.72
2.54 3.03 2.88
3.13
0.15
0.00
94.18
2.29 3.08 2.88
1.47
0.00
0.00
96.25
1.75 1.75 2.88
1.21
0.47
0.12
96.44
%c1
%c3
%c6
%c7
latency-performance
pk cor CPU
95
%c0
GHz
TSC
0 0.00
3.30 2.90 100.00
0.00
0.00
0.00
1 0.00
3.30 2.90 100.00
0.00
0.00
0.00
2 0.00
3.30 2.90 100.00
0.00
0.00
0.00
3 0.00
3.30 2.90 100.00
0.00
0.00
0.00
Profiling cpuidle and cpufreq
96
Fixed upstream cpuidle regression in June

Future CPUs such as Haswell add more C-states,
C8,C9,C10
Turbostat display a bit awkward for 40+ cores
Direction in future is to shrink the exit-latency between
C-states
http://www.breakage.org/2012/11/processor-max_cstat
e-intel_idle-max_cstate-and-devcpu_dma_latency/
Take-aways
97
Helpful Utilities
redhat-support-tool
sos
kdump
perf
Networking
NUMA
Supportability
hwloc
dropwatch
Intel PCM
ethtool
numactl
netsniff-ng (EPEL6)
numad
tcpdump
numatop (01.org)
Power/Tuning
wireshark/tshark
Storage
psmisc
strace
cpupowerutils (R6)
blktrace
sysstat
kernel-tools (R7)
iotop
systemtap
powertop
iostat
trace-cmd
tuna
util-linux-ng
tuned
98
Helpful Links
99
Red Hat Low Latency Performance Tuning Guide
Optimizing RHEL Performance by Tuning IRQ Affinity
Red Hat Performance Tuning Guide
Red Hat Virtualization Tuning Guide
STAC Network I/O SIG
Finteligent Low Latency Tuning w/KVM
Perf
Blog: http://www.breakage.org/ or @jeremyeder
Questions
100

Tampa Performance3

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Tampa Performance3

Diunggah oleh

Hak Cipta:

Format Tersedia

Performance Tuning and

Analysis of Red Hat

RED HAT CONFIDENTIAL | JEREMY EDER

Agenda: Performance Analysis of RHEL6/7

Performance Engineering Overview

Performance Analysis Utilities

RED HAT CONFIDENTIAL | JEREMY EDER

RED HAT CONFIDENTIAL | JEREMY EDER

Red Hat Performance Engineering

Benchmarks code path coverage

CPU linpack, lmbench

Memory lmbench, McCalpin Streams

Disk IO Iozone, aiostress scsi, FC, iSCSI

Filesystem IOzone, postmark ext3/4, xfs. gfs2,gluster

Network Netperf 10 Gbit, 40 Gbit IB, PCI3

Bare Metal, RHEL6/7 KVM

White box AMD/Intel, with our OEM partners

RED HAT CONFIDENTIAL | JEREMY EDER

Red Hat Confidential

Performance Projects / Tooling

RHEL6.5 numad tuna, and tuned

Tuna used to bind IRQ's / real-time like isolation

RED HAT CONFIDENTIAL | JEREMY EDER

RED HAT CONFIDENTIAL | JEREMY EDER

Performance Tuning Automation

NUMA Pinning (numactl)

IRQ Pinning (tuna)

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ?

Automatic not enough...

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ?

Automatic not enough...

Need to eek out the last X percent

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ?

Automatic not enough...

Need to eek out the last X percent

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ?

Automatic not enough...

Need to eek out the last X percent

OK...here's where it get's fun!

RED HAT CONFIDENTIAL | JEREMY EDER

RED HAT CONFIDENTIAL | JEREMY EDER

Userspace tool to read CPU

RED HAT CONFIDENTIAL | JEREMY EDER

List counters/tracepoints available

RED HAT CONFIDENTIAL | JEREMY EDER

grep for something interesting,

RED HAT CONFIDENTIAL | JEREMY EDER

System-wide 'top' view of busy

RED HAT CONFIDENTIAL | JEREMY EDER

Record system-wide (-a)

RED HAT CONFIDENTIAL | JEREMY EDER

Record system-wide (-a)

RED HAT CONFIDENTIAL | JEREMY EDER

Record system-wide (-a)

RED HAT CONFIDENTIAL | JEREMY EDER

Record system-wide (-a)

RED HAT CONFIDENTIAL | JEREMY EDER

Record system-wide (-a)

RED HAT CONFIDENTIAL | JEREMY EDER