White Paper:

The State of Cloud Storage

The State of Cloud Storage

2015 Industry Report

A Benchmark Comparison of
Speed, Availability and Scalability

Executive Summary
Both 2013 and 2014 were record-setting years for adoption of cloud services in the
enterprise. More than 50 percent of large enterprises reported using cloud services as an
integral part of their infrastructure, and it is safe to say that cloud services are no longer
just for early adopters. Cloud service provider (CSP) offerings have matured and are
quickly becoming an essential component of any leading enterprises critical infrastructure.
One of the most widely adopted and fastest growing cloud services is public cloud
storage. Some analysts project a 33 percent compound annual growth rate for public and
private cloud storage over the next five years, as enterprises find new ways to leverage
the unlimited capacity and enterprise-grade reliability offered by most leading CSPs. The
challenge for enterprises, however, has been how to take full advantage of what these
providers have to offer without increasing internal costs for support, maintenance and
custom development.
At Nasuni, we have created a unique service that enables enterprises to scale their
storage capacity without scaling their internal operations, to provide unlimited backups of their data, and to provide global access from any device. To do this, we leverage
integrations with the major CSPs for unlimited capacity and global footprint. Our goal is to
always provide the best infrastructure for our customers, and to that end, we regularly test
these CSPs services.
This year, we have restricted our tests to the three leading CSPs: Microsoft Azures Blob
Storage, Amazons Simple Storage Service (S3), and Googles Cloud Storage. We found
that the results are similar to last year: Microsoft has come out as the highest performer,
particularly in the benchmark tests. Amazon, as expected, is a close second, and edged
out Microsoft in some aspects of scalability and availability, but ultimately fell short in some
of the critical benchmark evaluations. Google continues to show promise albeit without a
significant difference in its status from 2013.

Cohen, Reuven, The Cloud Hits the Mainstream: More than Half of U.S.
Businesses Now Use Cloud Computing, Forbes, April 16, 2013.

Rebello, Jagdish, Enterprise cloud computing: future market size,

growth and competitive landscape, IHS Quarterly.

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Why Does Nasuni

Publish This Report?

provider is more a decision about the price and

performance trade-off than about features. We hope
that, by sharing our findings on the performance of
these providers, we are informing the market with
critical information to enable readers to distinguish
between these so-called commodity services.

We regularly benchmark the major cloud service

providers to ensure that we offer our customers the
most reliable and best performing cloud storage
available. A few years ago, we decided to publish
these results to 1) demonstrate the reasoning
behind our choice to work with specific technology
partners, and 2) provide some useful benchmarking
and analysis in a market that is becoming
increasingly commoditized.

How We Evaluate CSPs

One of the things that we take pride in doing well
at Nasuni is making the most out of what all the
cloud service providers have to offer. The Nasuni
Service employs many different cloud services
including NoSQL databases, elastic computing, and
messaging queues from a variety of providers. But
storage is central to what we offer, so, accordingly,
we are invested in evaluating the public cloud
storage capabilities of the major providers.
Our thinking around how to test public cloud storage
comes from our own experience providing missioncritical storage services to hundreds of enterprises.
Our benchmark tests reflect the patterns of use that
we see from our customers. The scalability tests are
designed to stress the critical aspects of the storage
environment, such as the durability of containers
and the behavior of the service under stress, as
well as to simulate some of our toughest use cases,
such as disaster recovery and initial migrations.
Where we set up our tests and how frequently we
test are also considerations that are designed to
replicate the global footprint of our customers and
help us to identify weak points not just in the storage
itself, but also in the network infrastructure that is
delivering it.

Although this report represents a detailed review of

Nasunis latest findings, it is important to note that
we conduct these tests primarily to identify CSPs
that meet the needs of Nasuni and its customers.
They do not necessarily represent the optimal
metrics for any provider tested. However, these
independent test results should be of interest to any
organization that is considering the use of cloud
storage as a component within a larger
IT infrastructure.

Comparison Metrics
As with most technology evaluations, three main
criteria govern the decision to purchase:
Functionality: what a service offers
Price: the cost of the service
Performance: how well that service is operates
For each of the cloud service providers, the
functionality is essentially the same: we are able to
create containers for objects, write objects into that
container, read objects from that container, and then
delete objects.
For most of the major CSPs, price is based on
some combination of data stored per month, and
the bandwidth consumed with both writes and
reads. Price competition has been happening for
years in this industry, and the real distinction in
pricing is around the capacity tiers of service. But,
for the most part, the price differences are minimal
and only matter on the margin. This leaves us with
performance as the main set of comparison metrics.

Sharing Critical Performance Information

Public cloud storage is often referred to as a
commodity that a CSP offers: a basic service that
differs little from its competitive alternatives. Since
most of the major cloud storage services are at
feature parity with each other, choosing the right

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Performance
Performance is the primary yardstick by which
Nasuni measures any publicly available CSP, testing
the operation and stability of CSPs over long
periods of time. In fact, Nasuni has been testing and
comparing CSPs since 2009. Before considering
any CSP for use in a production environment, it
must meet minimum performance benchmarks
across three areas:
Speed
Availability
Scalability
Speed
This simple test measures the raw ability of each
CSP to handle large numbers of writes, reads, and
deletes (W/R/D). We test each CSP with files of
varying sizes:
1 KB

10 MB

10 KB

100 MB

100 KB

1 GB

Due to dynamics in the marketplace, the list of

platforms evaluated continues to change from year
to year. The CSPs tested this year are Amazon S3,
Microsoft Azure, and Google Cloud Storage. HPs
Cloud Object Storage and IBMs SoftLayer were
considered, and even tested in a limited capacity;
however, we ultimately decided not to include them.
In HPs case, it was partly because there was a
change in their strategic direction and we were
unsure what their service would look like over the
coming year. With IBM, our experience with their
service was checkered with scheduled outages
on their part. Working around those proved to be
difficult and a concern for us. We hope to be able to
re-visit both services in the future.

Using different levels of concurrency:

25 Threads

10 Threads

50 Threads

This test runs for 12 hours, using multiple testing

machine instances and several non-serial test runs
to reduce the likelihood that external network issues
could bias the results.
Availability: This test takes place over a 30-day
period and measures each CSPs response time to
a single W/R/D process at 60-second intervals:
Write a randomly-generated 1 KB file.
Read a randomly-selected previously-written file.
Delete a selected file.
Reading and deleting a random file forces each CSP
to prove their ability to be responsive to all of the
data, all of the time, and not merely to the last piece
of cached data. This test calculates the entire time
required to complete the three requests, including

White Paper:
The State of Cloud Storage

Scalability: Similar to the availability test, this is

also an extended test that measures each CSPs
ability to perform consistently as the number of
objects under management increases. Performance
under increasing object counts is often the
Achilles heel of a cloud storage system, and this
test measures each CSPs ability to maintain
performance levels as the total number of objects
stored in a single container increases to hundreds
of millions. This is particularly important to Nasuni,
because unlimited scalability is a key feature of our
service. We have customers with large data sets
that have been keeping version snapshots for years.

Methodology

1 MB

1 Thread

any required retries. This ensures examination of

not only responsiveness, but also of CSP reliability
and latency.

We also looked to outside analysts to help validate

our decisions to focus our efforts this year. Gartner
has listed our three target CSPs in their most
recent Magic Quadrant report as the only Leaders
(Amazon and Microsoft) and Challengers (Google)
in the public cloud storage space. That additional
perspective helped us justify our decision to restrict
the scope of this years report.
Unlike last year, we did not invite the CSPs to
participate in the evaluation process.

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Test Setup

File Size &

Threads Tested
Number of
Concurrent Threads

Nasuni engineers conducted all tests between

October 2014 and February 2015, using virtual
machines across most of the major cloud-compute
platforms. The virtual machines had the
following specifications:
RAM: 3.5-4 GB
vCPUs: 1
Operating system: Ubuntu 14.04,
64 bit - Ubuntu 14.04 LTS

The test centers were also spread geographically

throughout the eastern and central regions of the
United States. We ran all tests using a variety of
times, locations, virtual machines, and dates to
minimize the risk of external network bias.
Benchmark Tests: The benchmark tests are
designed to evaluate the performance of CSPs
under file-server data load, and the file data set
represents the same distribution of file sizes used
by actual Nasuni enterprise customers across
thousands of installations over several years. Here is
our typical breakdown of files by size:
10KB

100KB

1MB

10MB

100MB

1GB

16.8% 24.6%

26.2%

9.7%

22.2%

0.4%

0.1%

Table 1: File size distribution

In addition, this distribution of file sizes closely

matches a well-documented breakdown that
Microsoft did over a number of years.
For each speed test, the test evaluated 23
combinations of file sizes and thread counts as
shown in Table 2.

White Paper:
The State of Cloud Storage

10
KB

100
KB

1
MB

10
MB

100
MB

1
GB

1
10
25
50

Table 2: File size and thread count combinations tested

Each CSPs benchmark tests were run using four

outside machines. For example, Amazon EC2 was
not used to test Amazon S3. Instead, S3 was tested
using Microsoft, HP, Google, and IBM.

1KB

1
KB

The results are averaged based on the weighting

of customer file-server data (Table 1) and are
then indexed to the performance of the top
performer. The results compare all the CSPs to
the performance of the top performer across all
file sizes and thread counts. This allows hundreds
of individual tests to be evaluated using a single
benchmark metric. Detailed raw results by CSP are
included in the appendix.
As the results show, the raw speed performance
varies significantly as object sizes and thread counts
vary. Specifically, small object sizes and smaller
thread counts highlight the transactional overhead
of any platform. The effect of transactional overhead
becomes most noticeable during writes, which
contains three steps:
Preparation
Transmission
Acknowledgment
For small files, Transmission is only a small portion
of the total transaction, so any inefficiency in the
performance of Preparation and Acknowledgment
has greater impact. Those object stores that are
built with efficient Preparation and Acknowledgment
steps perform best when handling small files.
However, as file sizes or thread counts increase,
the time associated with Transmission increasingly
dominates the overall time associated with
the transaction. Inefficiencies in Preparation or
Acknowledgement become less and less critical.
Many CSPs overly focus their efforts to improve
the efficiency of the Transmission stage of the

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Results

transaction, and thus perform better under the load

of larger object sizes or thread counts. This may be
fine for use-cases such as media archives, but for
file-server data, which is often dominated by small
files, performance on small files is critical.

Write Benchmark
Similar to last year, Microsoft was the top write
performer, excelling in 13 of the 23 individual
combinations tested, thereby making it the optimal
write target for file-based data. Amazon is a strong
second and Google performed at less than half of
the average response time of Microsoft.

Availability Test: Availability tests were run from

a single VM running in Rackspace. The metric
used is response time, which measures each
CSPs response time to a single W/R/D process at
60-second intervals. Because response time also
includes any time associated with retries or delays,
it is a more effective metric for availability than a
simple ping test.
Scalability Test: Scalability tests were conducted
using internal machines to reach the highest
scalability numbers in the shortest amount of time,
for example, Amazon EC2 writing to Amazon S3. As
object counts increase, the performance of some
CSPs degrades or becomes variable. Depending on
CSP architecture, some systems are designed to
scale across containers, not within them. This type
of architectural limitation can become a significant
bottleneck after months or years of usage.
An ideal scenario for anyone seeking to leverage
cloud storage is to partner with a CSP whose
performance and responsiveness are unchanging,
regardless of the number of objects under
management. Just as with traditional in-house
storage, customers expect a consistent level
of performance.

Figure 1: Indexed write speed with all file sizes

However, for files larger than 1MB, Amazon had

the overall best write performance. In Figure 2,
Google and Microsoft are on par with one another,
essentially tying for second place to Amazon.

Under this test, all of the CSPs were loaded with

new objects as quickly as possible: up to 100
million objects or 30 days, whichever came first. The
variance represents how much the speed of loading
objects changed over time, causing inconsistency
and variability as objects were loaded. This year
we added a new dimension write speed vs.
write speed variance so others can evaluate the
tradeoff of time-to-complete their large tasks vs. the
variability in the performance.
Figure 2: Indexed write speed with file sizes >1MB

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Read Benchmark

Delete Benchmark

In terms of read performance, Microsoft still

consistently outperforms the other CSPs. However,
Amazon trails Microsoft by less than it did in
the write performance tests. Google, however,
consistently performs at about half of the level of
Microsoft in both read and write.

Microsofts real performance superiority is seen in

the delete benchmark: it is more than twice as fast
at deleting files as Amazon and nearly 5x as fast as
Google. Different CSPs implement delete in different
ways. Some CSPs acknowledge the delete and
then do the work in the background. Others do the
actual delete operation before responding to the
request. This could explain some of the results in
this benchmark.

Figure 3: Indexed read speed with all file sizes

Similar to the write benchmarks, Amazon

significantly outperforms both Google and
Microsoft on the large file sizes.

Figure 5: Indexed DELETE speed with all file sizes

Figure 4: Indexed read speed with file sizes >1MB

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Availability
Amazon and Microsoft nearly tied on our availability
metrics, averaging a response time of .1 and .14
seconds, respectively, over a 30-day period. Google,
again, trailed the top two with an average response
time of .5 seconds, nearly 5x slower.

In addition to system and data availability, the

test also measures overall uptime or percent
of the time that the CSP is reachable. Unlike in
previous reports, each CSP had perfect scores
of 100 percent uptime.

Scalability
In previous reports, we measured scalability
resilience using three metrics: 1) the variance of
write speed in writing 100M objects, 2) the number
of write misses and 3) the number of read misses.
This year we decided to add a fourth dimension
to this analysis, the tradeoff between variance
(measured as (obj/s)2) and write speed.
Google had by far the lowest variance of the three,
an order of magnitude smaller than both Microsoft
and Amazon. Microsoft had the worst of the three
with exceptionally large variance.

Figure 6: Average response time (shorter bars are better)

Examining the results over the month of testing

also gives some insight into the variability of the
numbers. Amazon shows the most consistency and
the smallest daily uptime values. Microsoft is a close
second. Google, however, is both more variable
and slower.

Figure 8: Variance in write speed (shorter bars are better)

The read/write error analysis showed less of a

difference between CSPs. All CSPs had zero read
errors for 100M objects. Google and Microsoft
showed zero write errors, while Amazon had five
write errors for an error rate of .000005 percent.

Figure 7: Average daily response time

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

The average write speed/variance trade-off tells

a different story. Microsoft performs worst with
the lowest average write speed and the highest
variance. Amazon appears to be the top performer
with the highest write speed; however, that comes
at a price of a substantially higher variance over
Google, which has the lowest variance but also a
substantially lower average write speed. For most
applications, however, the ability to write quickly is
more important than the ability to write consistently,
which makes Amazon the preferred choice for
simply scaling quickly.

Amazon performed slightly better than Microsoft in

our availability test, both in overall average response
time and in daily average response time. However,
Google was a distant third, with almost five times
the response time.
Scalability results were mixed. Amazon had a
relatively low variance and the highest average write
speed, but also was the only CSP to show any write
errors. Although Microsoft had the highest variance,
it also posted the second highest speed and a
perfect record of zero read and zero write errors.
Overall, Microsoft and Amazon are the two clear
leaders in public cloud storage. This year, Microsoft
out-performed Amazon, specifically, in our critical
benchmark tests. However, Amazon continues
to demonstrate the robustness of its platform,
particularly in its ability to scale quickly and reliably.

Disclaimer

Figure 9: Variance (1/V) vs. average write speed

Conclusion
Similar to our 2013 report, we find that there are
only two significant competitors in the public cloud
storage market: Microsoft and Amazon. And, for the
second year in a row, Microsoft is the top CSP for
public cloud storage.

The tests reported upon in this document are conducted by us

using our own test tools under test conditions chosen by us.
The test conditions were chosen by us to reasonably represent
what our customers would experience using our Service with
their representative environments and workloads. The tests
have been designed by us to only look at the performance
aspects of the CSPs that we believe are relevant to our
customers it is intentionally narrow in scope. Nasuni is not in
the business of benchmarking CSPs, certifying test results or
selling performance metrics. We have attempted to make sure
the tests are fair and consistent within our selected parameters
and have worked with several of the vendors to confirm our
results. Our tests are not meant to indicate performance from
each CSP under ideal conditions to the CSP, and, in any
event, performance should only be one factor of many in a
CSP selection process.

In our most important suite of tests the

benchmark tests Microsoft consistently
performed better than Amazon. It delivered the
best speeds across small and medium-sized files
and, in some cases, beat Amazon by nearly 2x. For
large files, Amazon showed better write and read
performance, but Microsofts delete speeds topped
Amazons.

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Appendix Raw Results

Microsoft Azure

Write Benchmark Results (KB/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

62

531

3,485

9,654

18,075

10

35

3,242

17,502

28,947

27,182

25

472

4,316

19,745

29,838

28,896

50

519

4,606

19,973

30,014

30,411

100 MB

1 GB

100 MB

1 GB

100 MB

1 GB

Read Benchmark Results (KB/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

113

863

4,617

10,271

22,886

10

631

6,014

22,748

33,854

27,462

25

1,229

9,600

27,061

72,474

25,773

50

1,285

9,778

26,128

79,116

25,493

Delete Benchmark Results (Obj/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

111

107

105

105

129

10

755

799

806

795

695

25

1,267

1,250

1,294

1,032

659

50

1,272

1,294

1,276

1,244

435

Note: Microsoft Azure Blog Storage does not support objects larger than 64MB in a single upload.

10

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Amazon S3

Write Benchmark Results (KB/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

91

560

3,003

9,241

11,737

11,851

10

95

916

5,551

27,123

48,188

43,771

25

231

2,241

14,141

65,409

102,036

50

481

4,707

27,849

112,971

119,267

Read Benchmark Results (KB/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

19

175

1,480

8,729

20,067

32,651

36,042

10

190

1,881

13,783

62,196

89,801

86,477

25

474

4,560

31,833

83,684

82,460

50

996

9,354

56,899

86,346

75,570

Delete Benchmark Results (Obj/s)

Threads

11

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

17

18

18

17

18

16

13

10

174

166

182

175

180

157

25

470

448

473

461

441

50

957

955

968

945

850

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

Google Cloud Storage

Write Benchmark Results (KB/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

53

343

2,496

13,189

23,172

28,214

10

54

525

3,533

23,094

32,050

32,040

25

125

1,267

8,769

30,958

32,327

50

261

2,533

15,736

30,891

32,086

Read Benchmark Results (KB/s)

Threads

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

23

188

1,436

5,095

19,658

28,966

31,598

10

210

1,901

12,271

27,971

32,930

33,994

25

518

4,740

19,056

18,520

29,843

50

975

7,488

22,440

14,731

24,288

Delete Benchmark Results (Obj/s)

Threads

12

File Sizes
1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

10

10

87

88

61

86

77

60

25

212

209

149

218

192

50

441

419

296

310

264

White Paper:
The State of Cloud Storage

www.nasuni.com
info@nasuni.com

United States: 1.800.208.3418

International: 1.508.433.6200

About Nasuni
Files are everywhere and they are a pain. Nasuni eliminates this pain forever by delivering file storage for
distributed enterprises using a combination of cloud capacity, Nasuni software, and NAS appliances. Nasuni
gives customers unlimited storage with built-in data protection and DR, secure global file sharing and mobile
access, all managed from a single web console. Nasuni is cloud-based NAS for the distributed enterprise.
Our team is made up of enterprise storage, security and networking industry veterans with a shared vision of
transforming the way enterprise organizations view data storage. We believe that storage should be as easy
to purchase, consume and manage as the electricity that keeps the lights on.

Nasuni Corporation
2015, All Rights Reserved

313 Speen Street,

Natick, MA 01760-1538

1.508.433.6200
www.nasuni.com

WP_3.15