Postfix queue directories

The following sections describe Postfix queues: their purpose, what normal behavior looks like, and
how to diagnose abnormal behavior.

The "maildrop" queue

Messages that have been submitted via the Postfix sendmail(1) command, but not yet brought into
the main Postfix queue by the pickup(8) service, await processing in the "maildrop" queue.
Messages can be added to the "maildrop" queue even when the Postfix system is not running. They
will begin to be processed once Postfix is started.
The "maildrop" queue is drained by the single threaded pickup(8) service scanning the queue
directory periodically or when notified of new message arrival by the postdrop(1) program. The
postdrop(1) program is a setgid helper that allows the unprivileged Postfix sendmail(1) program to
inject mail into the "maildrop" queue and to notify the pickup(8) service of its arrival.
All mail that enters the main Postfix queue does so via the cleanup(8) service. The cleanup service
is responsible for envelope and header rewriting, header and body regular expression checks,
automatic bcc recipient processing, milter content processing, and reliable insertion of the message
into the Postfix "incoming" queue.
In the absence of excessive CPU consumption in cleanup(8) header or body regular expression
checks or other software consuming all available CPU resources, Postfix performance is disk I/O
bound. The rate at which the pickup(8) service can inject messages into the queue is largely
determined by disk access times, since the cleanup(8) service must commit the message to stable
storage before returning success. The same is true of the postdrop(1) program writing the message
to the "maildrop" directory.
As the pickup service is single threaded, it can only deliver one message at a time at a rate that
does not exceed the reciprocal disk I/O latency (+ CPU if not negligible) of the cleanup service.
Congestion in this queue is indicative of an excessive local message submission rate or perhaps
excessive CPU consumption in the cleanup(8) service due to excessive body_checks, or (Postfix ≥
2.3) high latency milters.
Note, that once the active queue is full, the cleanup service will attempt to slow down message
injection by pausing $in_flow_delay for each message. In this case "maildrop" queue congestion
may be a consequence of congestion downstream, rather than a problem in its own right.
Note, you should not attempt to deliver large volumes of mail via the pickup(8) service. High volume
sites should avoid using "simple" content filters that re-inject scanned mail via Postfix sendmail(1)
and postdrop(1).
A high arrival rate of locally submitted mail may be an indication of an uncaught forwarding loop, or
a run-away notification program. Try to keep the volume of local mail injection to a moderate level.
The "postsuper -r" command can place selected messages into the "maildrop" queue for
reprocessing. This is most useful for resetting any stale content_filter settings. Requeuing a large
number of messages using "postsuper -r" can clearly cause a spike in the size of the "maildrop"
queue.
The "hold" queue

The administrator can define "smtpd" access(5) policies, or cleanup(8) header/body checks that
cause messages to be automatically diverted from normal processing and placed indefinitely in the
"hold" queue. Messages placed in the "hold" queue stay there until the administrator intervenes. No
periodic delivery attempts are made for messages in the "hold" queue. The postsuper(1) command
can be used to manually release messages into the "deferred" queue.
Messages can potentially stay in the "hold" queue longer than $maximal_queue_lifetime. If such
"old" messages need to be released from the "hold" queue, they should typically be moved into the
"maildrop" queue using "postsuper -r", so that the message gets a new timestamp and is given
more than one opportunity to be delivered. Messages that are "young" can be moved directly into
the "deferred" queue using "postsuper -H".
The "hold" queue plays little role in Postfix performance, and monitoring of the "hold" queue is
typically more closely motivated by tracking spam and malware, than by performance issues.

The "incoming" queue

All new mail entering the Postfix queue is written by the cleanup(8) service into the "incoming"
queue. New queue files are created owned by the "postfix" user with an access bitmask (or mode)
of 0600. Once a queue file is ready for further processing the cleanup(8) service changes the queue
file mode to 0700 and notifies the queue manager of new mail arrival. The queue manager ignores
incomplete queue files whose mode is 0600, as these are still being written by cleanup.
The queue manager scans the incoming queue bringing any new mail into the "active" queue if the
active queue resource limits have not been exceeded. By default, the active queue accommodates
at most 20000 messages. Once the active queue message limit is reached, the queue manager
stops scanning the incoming (and deferred, see below) queue.
Under normal conditions the incoming queue is nearly empty (has only mode 0600 files), with the
queue manager able to import new messages into the active queue as soon as they become
available.
The incoming queue grows when the message input rate spikes above the rate at which the queue
manager can import messages into the active queue. The main factors slowing down the queue
manager are disk I/O and lookup queries to the trivial-rewrite service. If the queue manager is
routinely not keeping up, consider not using "slow" lookup services (MySQL, LDAP, ...) for transport
lookups or speeding up the hosts that provide the lookup service. If the problem is I/O starvation,
consider striping the queue over more disks, faster controllers with a battery write cache, or other
hardware improvements. At the very least, make sure that the queue directory is mounted with the
"noatime" option if applicable to the underlying filesystem.
The in_flow_delay parameter is used to clamp the input rate when the queue manager starts to fall
behind. The cleanup(8) service will pause for $in_flow_delay seconds before creating a new queue
file if it cannot obtain a "token" from the queue manager.
Since the number of cleanup(8) processes is limited in most cases by the SMTP server
concurrency, the input rate can exceed the output rate by at most "SMTP connection count" /
$in_flow_delay messages per second.
With a default process limit of 100, and an in_flow_delay of 1s, the coupling is strong enough to limit
a single run-away injector to 1 message per second, but is not strong enough to deflect an
excessive input rate from many sources at the same time.
If a server is being hammered from multiple directions, consider raising the in_flow_delay to 10
seconds, but only if the incoming queue is growing even while the active queue is not full and the
trivial-rewrite service is using a fast transport lookup mechanism.

The "active" queue

The queue manager is a delivery agent scheduler; it works to ensure fast and fair delivery of mail to
all destinations within designated resource limits.
The active queue is somewhat analogous to an operating system's process run queue. Messages
in the active queue are ready to be sent (runnable), but are not necessarily in the process of being
sent (running).
While most Postfix administrators think of the "active" queue as a directory on disk, the real "active"
queue is a set of data structures in the memory of the queue manager process.
Messages in the "maildrop", "hold", "incoming" and "deferred" queues (see below) do not occupy
memory; they are safely stored on disk waiting for their turn to be processed. The envelope
information for messages in the "active" queue is managed in memory, allowing the queue manager
to do global scheduling, allocating available delivery agent processes to an appropriate message in
the active queue.
Within the active queue, (multi-recipient) messages are broken up into groups of recipients that
share the same transport/nexthop combination; the group size is capped by the transport's recipient
concurrency limit.
Multiple recipient groups (from one or more messages) are queued for delivery grouped by
transport/nexthop combination. The destination concurrency limit for the transports caps the
number of simultaneous delivery attempts for each nexthop. Transports with a recipient
concurrency limit of 1 are special: these are grouped by the actual recipient address rather than the
nexthop, yielding per-recipient concurrency limits rather than per-domain concurrency limits. Per-
recipient limits are appropriate when performing final delivery to mailboxes rather than when
relaying to a remote server.
Congestion occurs in the active queue when one or more destinations drain slower than the
corresponding message input rate.
Input into the active queue comes both from new mail in the "incoming" queue, and retries of mail in
the "deferred" queue. Should the "deferred" queue get really large, retries of old mail can dominate
the arrival rate of new mail. Systems with more CPU, faster disks and more network bandwidth can
deal with larger deferred queues, but as a rule of thumb the deferred queue scales to somewhere
between 100,000 and 1,000,000 messages with good performance unlikely above that "limit".
Systems with queues this large should typically stop accepting new mail, or put the backlog "on
hold" until the underlying issue is fixed (provided that there is enough capacity to handle just the
new mail).
When a destination is down for some time, the queue manager will mark it dead, and immediately
defer all mail for the destination without trying to assign it to a delivery agent. In this case the
messages will quickly leave the active queue and end up in the deferred queue (with Postfix < 2.4,
this is done directly by the queue manager, with Postfix ≥ 2.4 this is done via the "retry" delivery
agent).
When the destination is instead simply slow, or there is a problem causing an excessive arrival rate
the active queue will grow and will become dominated by mail to the congested destination.
The only way to reduce congestion is to either reduce the input rate or increase the throughput.
Increasing the throughput requires either increasing the concurrency or reducing the latency of
deliveries.
For high volume sites a key tuning parameter is the number of "smtp" delivery agents allocated to
the "smtp" and "relay" transports. High volume sites tend to send to many different destinations,
many of which may be down or slow, so a good fraction of the available delivery agents will be
blocked waiting for slow sites. Also mail destined across the globe will incur large SMTP command-
response latencies, so high message throughput can only be achieved with more concurrent
delivery agents.
The default "smtp" process limit of 100 is good enough for most sites, and may even need to be
lowered for sites with low bandwidth connections (no use increasing concurrency once the network
pipe is full). When one finds that the queue is growing on an "idle" system (CPU, disk I/O and
network not exhausted) the remaining reason for congestion is insufficient concurrency in the face
of a high average latency. If the number of outbound SMTP connections (either ESTABLISHED or
SYN_SENT) reaches the process limit, mail is draining slowly and the system and network are not
loaded, raise the "smtp" and/or "relay" process limits!
When a high volume destination is served by multiple MX hosts with typically low delivery latency,
performance can suffer dramatically when one of the MX hosts is unresponsive and SMTP
connections to that host timeout. For example, if there are 2 equal weight MX hosts, the SMTP
connection timeout is 30 seconds and one of the MX hosts is down, the average SMTP connection
will take approximately 15 seconds to complete. With a default per-destination concurrency limit of
20 connections, throughput falls to just over 1 message per second.
The best way to avoid bottlenecks when one or more MX hosts is non-responsive is to use
connection caching. Connection caching was introduced with Postfix 2.2 and is by default enabled
on demand for destinations with a backlog of mail in the active queue. When connection caching is
in effect for a particular destination, established connections are re-used to send additional
messages, this reduces the number of connections made per message delivery and maintains good
throughput even in the face of partial unavailability of the destination's MX hosts.
If connection caching is not available (Postfix < 2.2) or does not provide a sufficient latency
reduction, especially for the "relay" transport used to forward mail to "your own" domains, consider
setting lower than default SMTP connection timeouts (1-5 seconds) and higher than default
destination concurrency limits. This will further reduce latency and provide more concurrency to
maintain throughput should latency rise.
Setting high concurrency limits to domains that are not your own may be viewed as hostile by the
receiving system, and steps may be taken to prevent you from monopolizing the destination
system's resources. The defensive measures may substantially reduce your throughput or block
access entirely. Do not set aggressive concurrency limits to remote domains without coordinating
with the administrators of the target domain.
If necessary, dedicate and tune custom transports for selected high volume destinations. The "relay"
transport is provided for forwarding mail to domains for which your server is a primary or backup MX
host. These can make up a substantial fraction of your email traffic. Use the "relay" and not the
"smtp" transport to send email to these domains. Using the "relay" transport allocates a separate
delivery agent pool to these destinations and allows separate tuning of timeouts and concurrency
limits.
Another common cause of congestion is unwarranted flushing of the entire deferred queue. The
deferred queue holds messages that are likely to fail to be delivered and are also likely to be slow to
fail delivery (time out). As a result the most common reaction to a large deferred queue (flush it!) is
more than likely counter-productive, and typically makes the congestion worse. Do not flush the
deferred queue unless you expect that most of its content has recently become deliverable (e.g.
relayhost back up after an outage)!
Note that whenever the queue manager is restarted, there may already be messages in the active
queue directory, but the "real" active queue in memory is empty. In order to recover the in-memory
state, the queue manager moves all the active queue messages back into the incoming queue, and
then uses its normal incoming queue scan to refill the active queue. The process of moving all the
messages back and forth, redoing transport table (trivial-rewrite(8) resolve service) lookups, and re-
importing the messages back into memory is expensive. At all costs, avoid frequent restarts of the
queue manager (e.g. via frequent execution of "postfix reload").

The "deferred" queue

When all the deliverable recipients for a message are delivered, and for some recipients delivery
failed for a transient reason (it might succeed later), the message is placed in the deferred queue.
The queue manager scans the deferred queue periodically. The scan interval is controlled by the
queue_run_delay parameter. While a deferred queue scan is in progress, if an incoming queue
scan is also in progress (ideally these are brief since the incoming queue should be short), the
queue manager alternates between looking for messages in the "incoming" queue and in the
"deferred" queue. This "round-robin" strategy prevents starvation of either the incoming or the
deferred queues.
Each deferred queue scan only brings a fraction of the deferred queue back into the active queue
for a retry. This is because each message in the deferred queue is assigned a "cool-off" time when
it is deferred. This is done by time-warping the modification time of the queue file into the future.
The queue file is not eligible for a retry if its modification time is not yet reached.
The "cool-off" time is at least $minimal_backoff_time and at most $maximal_backoff_time. The next
retry time is set by doubling the message's age in the queue, and adjusting up or down to lie within
the limits. This means that young messages are initially retried more often than old messages.
If a high volume site routinely has large deferred queues, it may be useful to adjust the
queue_run_delay, minimal_backoff_time and maximal_backoff_time to provide short enough delays
on first failure (Postfix ≥ 2.4 has a sensibly low minimal backoff time by default), with perhaps longer
delays after multiple failures, to reduce the retransmission rate of old messages and thereby reduce
the quantity of previously deferred mail in the active queue. If you want a really low
minimal_backoff_time, you may also want to lower queue_run_delay, but understand that more
frequent scans will increase the demand for disk I/O.
One common cause of large deferred queues is failure to validate recipients at the SMTP input
stage. Since spammers routinely launch dictionary attacks from unrepliable sender addresses, the
bounces for invalid recipient addresses clog the deferred queue (and at high volumes proportionally
clog the active queue). Recipient validation is strongly recommended through use of the
local_recipient_maps and relay_recipient_maps parameters. Even when bounces drain quickly they
inundate innocent victims of forgery with unwanted email. To avoid this, do not accept mail for
invalid recipients.
When a host with lots of deferred mail is down for some time, it is possible for the entire deferred
queue to reach its retry time simultaneously. This can lead to a very full active queue once the host
comes back up. The phenomenon can repeat approximately every maximal_backoff_time seconds
if the messages are again deferred after a brief burst of congestion. Perhaps, a future Postfix
release will add a random offset to the retry time (or use a combination of strategies) to reduce the
odds of repeated complete deferred queue flushes.