Performance Tuning Guide for

Windows 2003 and Windows

2008

Data Domain, Inc.

2421 Mission College Boulevard, Santa Clara, CA 95054
866-WE-DDUPE; 408-980-4800
Version 1.0 Revision A
March 10, 2010
Data Domain Proprietary and Confidential

Copyright 2010 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC
CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND
WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication
requires an applicable software license.
EMC and Data Domain are registered trademarks, and Global Compression is a
trademark of EMC Corporation.

Performance Tuning Guide for

Windows 2003 and Windows 2008
This document explains how to achieve optimal performance with Windows 2003 and
Windows 2008 systems used as backup servers with a Data Domain system.
Note: This document applies to backup servers only, not to nearline or archive servers.
This document covers the topics shown in the following table.
Related Documentation

Page 4

Defining the Network Topology and a Performance Baseline

Page 4

Running Multiple Clients

Page 4

Enabling Window Scaling (Windows 2003 Only)

Page 4

Tuning the Network Adapter

Page 5

Suggested Tools for Monitoring Performance

Page 5

Troubleshooting Performance Problems

Page 6

Determining the Performance Baseline

Page 6

Identifying Bottlenecks

Page 6

Performance Tuning Guide for Windows 2003 and Windows 2008

Related Documentation
The Documentation page at https://my.datadomain.com/documentation provides
access to documents that have information related to use of this application with Data
Domain products. Be sure to read the Data Domain Operating System User Guide and
release notes, under Product Documentation. Also be sure to read the Data Domain
integration guide or application introduction for your backup application.
Also refer to the following Microsoft documents:

Performance Tuning Guidelines for Windows Server 2003

Performance Tuning Guidelines for Windows Server 2008

Defining the Network Topology and a Performance

Baseline
Collect information about your network environment as a baseline to use for comparison
later if you encounter performance problems. When you fully understand the network
topology, you can better detect and resolve any bottlenecks that appear. Collect
information about the routers, the gateway used, and the Maximum Transfer Unit (MTU)
used on the network path between the systems. Many Windows tools can help you
monitor networks, network adapters, network interfaces, and network-related
consumption of system resources. For suggestions, see Suggested Tools for Monitoring
Performance on page 5.
Note: Many factors affect performance, so the recommendations in this section cannot
apply to every situation. Experiment to find the best settings for your environment.

Running Multiple Clients

Running two or more clients on the same workstation may yield better throughputs for
backups and restores, especially for workstations that have large file systems on multiple
disks. Consider running a client for each file system, keeping in mind the resulting
increased memory usage and CPU utilization.

Enabling Window Scaling (Windows 2003 Only)

Note: Windows 2008 tunes window scaling automatically.
The TCP receive window specifies the amount of data that the sending host can send at
one time on a connection. Window scaling permits TCP to negotiate a scaling factor for
the TCP receive window size, which allows for a TCP receive window of up to 1MB. To
enable window scaling, do the following:
1. Use Regedt32 to open this registry:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters
2. Add the following values:

Name

Type

Data (recommended value)

Tcp1323Opts

REG_DWORD

Performance Tuning Guide for Windows 2003 and Windows 2008

Name

Type

Data (recommended value)

TCPWindowSize

REG_DWORD

DDOS 4.7 and later: 1048576

DDOS 4.6: 262144

TcpTimedWaitDelay REG_DWORD

30

Tuning the Network Adapter

Optimize network throughput and resource usage by tuning the network adapter if it
supports tuning. The optimal set of tuning parameters depends on the network adapter,
the workload, host-computer resources, and your performance goals.
Data Domain recommends that you disable the Windows feature that lets the system
offload TCP/IP transport tasks and connections to a Network Interface Card (NIC). This
feature is enabled by default. Many network adapters are not powerful enough to handle
the offloaded features at high throughput rates, so the net result is a loss of performance.
Some network adapters let administrators configure resources such as receive and send
buffers. Other network adapters actively manage their resources, and there is no need to
configure them. In either case, Data Domain recommends against manually tuning these
resources.
Enabling jumbo frames yields more efficient network communication between the server
and the Data Domain system, but only if the full network path, including the switch, is
configured with jumbo frames enabled.
For optimal backup and restore performance with specific backup applications, Data
Domain generally recommends tuning application-specific values such as the read/write
buffer sizes and multi-threading. For recommended values, see the Data Domain
integration guide or application introduction for your backup application.
No special tuning parameters are required for a 10 Gb client.

Suggested Tools for Monitoring Performance

Consider using the following tools to monitor performance, loads, and network traffic:

Use the Windows Performance Monitor (perfmon) to test system performance. In the
Performance Monitor, you can choose:

Objects to watch (for example, the processor or memory)

Counters to test (for example, available memory)

The default view of the data is a real-time line chart. However, you can log or format
the data as reports and generate alerts when counters reach specified limits.

Use a tool such as the freeware application Lmdd.exe to determine the baseline
performance of the network between the client and the backup server, or between the
backup server and the Data Domain system. Data Domain also recommends Iometer
and Iperf.

Use a network sniffing application to monitor network data and detect the
bottlenecks. Data Domain recommends the Microsoft Network Monitor (netmon)
application or the freeware application Ethereal.

Performance Tuning Guide for Windows 2003 and Windows 2008

Troubleshooting Performance Problems

Solving a performance problem is frequently a process of:
1. Determining the performance baseline.
2. Identifying a bottleneck.
3. Eliminating a bottleneck.

This section offers some suggestions for these tasks.

Determining the Performance Baseline

Set a realistic performance expectation before you start troubleshooting so that you can
recognize performance bottlenecks. Use historical data that you recorded before the
problem started to determine the baseline performance. Next, try the following steps.
1. Confirm that the problem is reproducible.
2. Start troubleshooting without the application running. After you establish the

baseline, continue testing with the application running.

3. Use Data Domain system autosupport reports and the output of the DD OS command
system show performance can help you locate the bottleneck. The autosupport

report includes historical data on the transfer rate and system load.

Identifying Bottlenecks
Systematically eliminate as many factors as possible to isolate the root cause of the
problem. First, determine whether the problem lies in the Data Domain system or
outside. Search for performance bottlenecks in the following sequence:
1. Data Domain system
2. Application (backup) server
3. Network
4. Hardware
5. Client

The following sections give troubleshooting tips for the first three of these possible
sources. To troubleshoot hardware and client problems, see the vendor documentation
for those products.

Performance Tuning Guide for Windows 2003 and Windows 2008

Troubleshooting the Data Domain System

First, rule out the Data Domain system as the source of the problem:
1. Log in to the Data Domain system as an administrative user and run this command:
# system show detailed-stats 2

The output is similar to the following example:

CPU0
CPU State
NFS
NFS
NFS
NFS
NFS
eth0 KB/s
Busy
max 'CDVMS' ops/s
proc recv send idle in
out
--------------------------------------------------------------------0%
0%
33%
43%
30%
36%
29%
29%

0%
0%
35%
48%
34%
38%
29%
30%

V
V

0
0
1433
1746
1597
1624
1246
976

0%
0%
97%
98%
95%
99%
95%
98%

0%
0%
1%
1%
1%
1%
1%
1%

0%
0%
0%
0%
0%
0%
0%
0%

0%
0%
2%
5%
4%
0%
4%
1%

0
0
30
32
50
60
52
66

1
0
0
2
1
1
3
1

2. From the Windows server, manually map a drive to the Data Domain share /backup

as drive Z.

3. While system show detailed-stats 2 is running, use the lmdd.exe command on

the Windows server to create a 10 GB file of zeros on Data Domain share via drive Z:
CMD:c:\lmdd.exe if=/dev/zero of=z:\allzero bs=32768 count=327680

The output on the Data Domain system shows the transfer rate for each interface in
KB/s. One lmdd thread should provide 50 MB/s. Running three lmdd instances
concurrently should fully utilize a gigabit link at about 110 MB/s.
4. Examine the NFS proc, recv, send, and idle columns. In the example in Step 1, the

Data Domain system is very busy processing requests (97-99%).

5. Check the State column to rule out normal system maintenance activity as the cause

of the problem. These activities can affect performance. See Table 1

Table 1: State Values and Descriptions

State Meaning

Details

Cleaning

During this state the system performs read

and write operations to the file system.

Degraded RAID

Disk reconstruction (repair of a failed disk),

or RAID is resyncing (after an improper
system shutdown and a restart), or RAID is
degraded (a disk is missing and no
reconstruction is in progress)

Verify data

Data is being checked for consistency.

Merge fingerprint index.

This is a normal system maintenance activity.

Summary vector checkpoint.

This is a normal system maintenance activity.

Performance Tuning Guide for Windows 2003 and Windows 2008

6. If the system State does not explain the bottleneck as a temporary problem, use

Table 2 to troubleshoot the problem.

Table 2: Troubleshooting Suggestions

Observed
Statistics

Possible Sources of
the Bottleneck

Troubleshooting Suggestions

High NFS proc

Data Domain system

Upload a support bundle to Data Domain.

See the next step in this procedure for
instructions.

Low NFS proc

Network
Backup server
Backup server clients

The Data Domain system is lightly loaded.

Look elsewhere for the source of the
bottleneck.

High NFS recv

or send

Network
Backup server

The network may be the source of the

bottleneck. If the performance is at or below
about 11-12MB/s, the network link may be
running at 100Mbit/s rather than 1Gb/s or
10Gb/s. Ensure that the backup server and
the switch ports are configured for the correct
speed.
Alternatively, the backup server may be
heavily loaded. Examine the backup server
CPU utilization. On a Windows system, use
Perfmon to check that no individual CPU is
running at 100%. The average CPU
utilization might be under 100% even though
one or more CPUs are fully loaded, because
the network stack is single-threaded.

High NFS proc,

recv, and send

Backup server

The backup server disk may be heavily

loaded, the backup software may require
tuning, or the backup server may be waiting
for data from slow backup clients.

High NFS idle

Backup server

Ensure that the backup software is

correctly configured and optimized.
Examine backup server disk utilization if
you are backing up local disks.
If you are backing up remote clients,
ensure that the clients are not the source of
the bottleneck.

7. If you determine that there is a problem with the Data Domain system, generate a

support upload bundle and contact Data Domain Support. Use these DD OS
commands to generate the support upload bundle:
# perf start trace
# perf stop trace trace.bin
# support upload bundle trace.bin

Otherwise, proceed to the next section and continue troubleshooting with the backup
server.

Performance Tuning Guide for Windows 2003 and Windows 2008

Troubleshooting the Backup Server

On the Windows server, use the Windows Performance Monitor program to test system
performance.
The following counters help you determine whether the problem is a memory issue.

Pages/sec: when the number of pages per second exceeds 50 per paging disk on your
system, this indicates that there is not enough RAM to meet your server's needs.

Available Bytes: when the value is less than 10% of the actual RAM, this indicates that
insufficient memory is available.

When the Processor counter runs near 100% for extended periods, drill down at the
process level by examining the Process (instance)\% Processor Time counter for various
process instances on your system. This will help to determine if the system load is
handled effectively or if the system is underutilized.
The Disk Performance counters help you evaluate the performance of the disk
subsystem.

LogicalDisk : Disk Reads/sec: when observing an individual application (rather than

a partition), this counter gives an indication of how often the applications on the
partition are reading from the disk.

PhysicalDisk : Disk Reads Bytes/sec: use this counter to determine the disk
throughput for the disk subsystem.

LogicalDisk : Disk Reads Bytes/sec: for the partition, use this counter to see an
indication of the data transfer rate. A small value indicates a large number of random
reads of smaller sections.

For each network adapter, check the following statistics:

Bytes received per second: this counter shows how many bytes of data are sent to the
NIC.

Bytes sent per second: this counter shows how many bytes are received from the NIC.

Packets received per second

Packets sent per second

Output queue length

This counter is the length of the output packet queue, in packets. If the value is
greater than 2, delays occur. Locate and eliminate the bottleneck if possible. Because
NDIS queues the requests, this length should alwaysbe0 for a properly-configured
system.

Packets received errors

This counter is the number of incoming packets that contain errors and cannot be
delivered to a higher-layer protocol. A zero value does not guarantee that there are no
receive errors. The value is polled from the network driver, and it can be inaccurate.

Packets outgoing errors

Performance Tuning Guide for Windows 2003 and Windows 2008

Troubleshooting the Network

Check for problems in the following areas:

Media speed and duplexing

Ensure that both systems are set up with the same speed and duplex settings. For a
point-to-point connection, set up Auto Negotiation so that the adapters can negotiate
the speed and duplex rating to the highest possible speeds. If one of the adapters is
not set to Auto Negotiate, use the Control Panel to manually configure both adapters
with the same settings.

MTU size
Ensure that all of the other systems that are connected to the same switch (VLAN)
have jumbo frames enabled, and that the switch itself supports jumbo frames.

Network adapter
To ensure that the network adapter is not at fault, check the following:

10

All network interfaces of systems on the same network must have the same
network address.

The MTU sizes of systems on the same physical network or VPN must match.

The values for Ierrs and Oerrs should always be zero. If they are not, check the
network hardware and interfaces for problems. On Ethernet networks, the
collision field is not supported and always displays 0 (zero).

Performance Tuning Guide for Windows 2003 and Windows 2008