WCF binding Understanding and configuring buffer management

Problem:
Developing or consuming service-oriented applications? Ever came across HTTP
400s? a needless question though. It is not uncommon to find a WCF
CommunicationException with HTTP 400 (Bad Request). Quite a few of these
instances may happen to be returning an error message that would say The
maximum message size quota for incoming messages (65536) has been exceeded. To increase
the quota, use the MaxReceivedMessageSize property on the appropriate binding element .
In certain complex project architectures, you may not get to see the detailed
error message, especially, if you are just working on the UI end of it.
To find out a solution for this, let us understand how WCF does the buffer
management.
Background & Solution:
What?
In the simplest form, a buffer is a chunk of memory and buffer pool is a set of
one or more buffers.
It all starts when the BufferManager is created. BufferManager creates and
manages the buffers and buffer pools, depending on the values that you set in
the binding configuration in the applications config file. Lets first see what those
buffer-related values are.
MaxBufferPoolSize is the size of the largest buffer pool that the BufferManager
can manage.

If, in case, allocating buffer to an incoming message would mean that it

would exceed this limit, then, the message would take a new buffer out
of the heap and the garbage collector takes care of it once used up.
If this value is 0(zero), it means that each request would take a new
buffer out of the memory heap and will be taken care of, by the
garbage collector.

MaxBufferSize is the size of the largest individual buffer that the buffer manager
can allocate (to any buffer pool).

For streamed transfers, only the SOAP headers need to be buffered while
the message body can be streamed on-demand. In such cases, this would
just mean the maximum size of the SOAP headers. For buffered transfers,
it is the maximum size of the headers and message body.

MaxReceivedMessageSize is the maximum size of a message that can be

received on a channel.

For buffered transfers, this is the same as the MaxBufferSize.

For streamed transfers, this is either greater than or equal to
MaxBufferSize (for obvious reasons mentioned above)

How?
When the BufferManager is created, a series of buffer pools are also created,
with each pool holding up to a memory determined by MaxBufferPoolSize.
The buffers in each of these pools would have sizes in an incremental pattern of
the multiples of 128, until it reaches the maximum allowed message size (as set
by MaxBufferSize).
For example, if the MaxBufferSize is set to 450 bytes, the BufferManager would
create 3 pools as
1st pool that holds buffers of size 128 bytes each
2nd pool that holds buffers of size 256 bytes each
3rd pool that holds buffers of size 450 bytes (this pool should otherwise hold
buffers of size 512, but since the MaxBufferSize is lesser than 512, it would limit
it to 450)
Here is a logical depiction of the buffer pools from above example. The n below
represents the number of buffers each pool can hold.
It is equal to the MaxBufferPoolSize / buffer size that pool can hold.
Say, the MaxBufferPoolSize for the example is 1800.
4501
2561

1281
1282

1283
128n

1284

n=>1800/128 =14
=4

4502
2562

2563
256n

4503

2564

n=>1800/256 =7

450n

4504

n=>1800/450

Note that each buffer pools maximum size is the same, though the sizes of the
buffers being held by each of those differ.
When?
Buffers are not allocated when the pools are created. They are only allocated
when a message arrives to be processed.
1. Request arrives

2. BufferManager tries to find a pool that can hold buffers that are sufficient
for this message
3. If appropriate pool is identified with available buffers, the message gets
processed. Otherwise, the message is allocated a buffer out of the heap
(not managed by BufferManager).
4. Also, If the message size is more than the MaxBufferSize, it is allocated an
unmanaged buffer, on the memory heap i.e., buffer that is garbage
collected as when it is available for collection.
5. Once processed, the buffer manager tries to put the buffer that held the
message back into its pool for reuse. However, if by doing so, the pools
size exceeds the MaxBufferPoolSize, the buffer will not be returned back to
the pool.
Trade-off
If you need to process large messages, you can use a large MaxBufferSize. But,
using too much memory would also decrement performance. As a developer, you
should find a balance between time and memory.
As a thumb rule, if your server, normally has n number of concurrent requests,
then you may have to set the MaxBufferPoolSize as n times the MaxBufferSize.
Example configuration
Finally, let us see a sample configuration setting w.r.t. the settings we defined in
the What? Section of this post.
<binding name="WSHttpBinding_SampleFeedService" closeTimeout="00:00:30" openTimeout="00:00:30"
receiveTimeout="00:00:30" sendTimeout="00:00:30" bypassProxyOnLocal="false" transactionFlow="false"
hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="524288" maxReceivedMessageSize="2097152"
messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />
<security mode="None">
<transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
<message clientCredentialType="Windows" negotiateServiceCredential="true"
establishSecurityContext="true" />
</security>
</binding>