Scaling an early stage startup

by Mark Maunder <mark@feedjit.com>

Why does performance and scaling
quickly matter?
• Slow performance could cost
you 20% of your revenue
according to Google.
• Any reduction in hosting costs
goes directly to your bottom line
as profit or can accelerate
growth.
• In a viral business, slow
performance can damage your
viral growth.
 My first missteps
• Misconfiguration. Web
server and DB configured
to grab too much RAM.
• As traffic builds, the
server swaps and slows
down drastically.
• Easy to fix – just a quick
config change on web
server and/or DB.
 Traffic at this stage
• 2 Widgets per second
• 10 HTTP requests per
second.
• 1 Widget = 1 Pageview
• We serve as many
pages as our users do,
combined.
 Keepalive – Good for clients, bad
for servers.
• As http requests
increased to 10 per
second, I ran out of
server threads to
handle
connections.
• Keepalive was on
and Keepalive
Timeout was set to
300.
• Turned Keepalive
off.
 Traffic at this stage
• 4 Widgets per second
• 20 HTTP requests per second
Cache as much DB data as possible
• I used Perl’s Cache::FileCache to cache either DB
data or rendered HTML on disk.
• MemCacheD, developed for LiveJournal, caches
across servers.
• YMMV – How dynamic is your data?
 MySQL not fast
enough

• High number of writes & deletes on a large

single table caused severe slowness.
• Writes blow away the query cache.
• MySQL doesn’t support a large number of small
tables (over 10,000).
• MySQL is memory hungry if you want to cache
large indexes.
• I maxed out at about 200 concurrent read/write
queries per second with over 1 million records
(and that’s not large enough).
 Perl’s Tie::File to the early rescue
• Tie::File is a very
simple flat-file API.
• Lots of files/tables.
• Faster – 500 to
1000 concurrent
read/writes per
second.
• Prepending
requires reading
and rewriting the
whole file.
 BerkeleyDB is very very fast!

• I’m also experimenting with BerkeleyDB for

some small intensive tasks.
• Data From Oracle who owns BDB: Just over
90,000 transactional writes per second.
• Over 1 Million non-transactional writes per
second in memory.
• Oracle’s machine: Linux on an AMD Athlon™ 64
processor 3200+ at 1GHz system with 1GB of
RAM. 7200RPM Drive with 8MB cache RAM.
Source: http://www.oracle.com/technology/products/berkeley-db/pdf/berkeley-db-perf.pdf
 Traffic at this stage
• 7 Widgets per second
• 35 HTTP requests per second
 Created a separate
image and CSS server

• Enabled Keepalive on the Image server to be

nice to clients.
• Static content requires very little memory per
thread/process.
• Kept Keepalive off on the App server to reduce
memory.
• Added benefit of higher browser concurrency
with 2 hostnames.
Source: http://www.die.net/musings/page_load_time/
 Now using Home Grown Fixed
Length Records
• A lot like ISAM or MyISAM
• Fixed length records mean we
seek directly to the data. No more
file slurping.
• Sequential records mean
sequential reads which are fast.
• Still using file level locking.
• Benchmarked at 20,000+
concurrent reads/writes/deletes.
 Traffic at this stage
• 12 Widgets per second
• 50 to 60 HTTP requests per
second
• Load average spiking to 12
or more about 3 times per
day for unknown reason.
Blocking Content Thieves

• Content thieves were

aggressively crawling our
site on pages that are CPU
intensive.
• Robots.txt is irrelevant.
• Reverse DNS lookup with
‘dig –x’
• Firewall the &^%$@’s with
‘iptables’
 Moved to httpd.prefork
• Httpd.worker consumes more memory than
prefork because worker doesn’t share memory.
• Tuning the number of Perl interpreters vs number
of threads didn’t improve things.
• Prefork with no keepalive on the app server uses
less RAM and works well – for Mod_Perl.
 The amazing Linux
Filesystem Cache

• Linux uses spare memory to cache files

on disk.
• Lots of spare memory == Much faster I/O.
• Prefork freed lots of memory. 1.3 Gigs out
of 2 Gigs is used as cache.
• I’ve noticed a roughly 20% performance
increase since using it.
 Tools
• httperf for benchmarking your server
• Websitepulse.com for perf monitoring.
 Summary
• Make content as static as possible.
• Cache as much of your dynamic
content as possible.
• Separate serving app requests and
serving static content.
• Don’t underestimate the speed of
lightweight file access API’s .
• Only serve real users and search
engines you care about.