A Deep Dive into Transport Queues - Part 1

12 January 2010 by Johan Veldhuis

Submission queues? Poison message queues? Johan Veldhuis unlocks the mysteries of MS Exchange's Transport queues that used to temporarily store messages waiting until they are passed through to the next stage, and explains how to manage these queues.

You may wonder, why a deep dive into transport queues, is there so much to tell about it? Well, thats the first thing I thought too. But after some investigation and writing down some things I discovered that there is much to tell about them. In this article, I will discuss which transport queues exist and how they work together to deliver message(s) to a mailbox or another mail server. After that, I will explain which tools you can use to manage the transport queues. Lets start with the definition of a queue. What does the dictionary tells us about it? Queue means to form a line, and to wait for services. In Exchange, the queues are used to temporarily store messages waiting until they are passed through to the next stage. Exchange contains five types of queues: submission queue, mailbox delivery queue, remote delivery queue, poison message queue, and the unreachable queue. Depending on the routing of messages, they are placed in a specific queue. What are the queues used for? Below is an overview:

Submission queue, all messages received will always first be placed in the submission queue. The submission queue receives the messages via SMTP-receive, the Pickup directory, or the store driver. Once placed in this queue, the categorizer will scan messages. Once scanned the categorizer will place the message in the correct queue. This can be the delivery queue or the undeliverable queue. The categorizer makes this decision by checking the location of the recipient. The submissions queue can be found both on Edge and Hub servers, each Transport server can only have one submissions queue. Mailbox delivery queue, messages in this queue are waiting to be delivered to a mailbox server in the same site. The mailbox delivery queue can only be found on a Hub server. Unlike the submission queue a Hub server can have multiple mailbox delivery queues. Remote delivery queue, as the name already says this queue is used for delivering messages to remote servers by using SMTP. This queue can be both seen on an Edge and Hub server and both servers may have multiple queues. Queues will be automatically created when needed and will be deleted when there will be no message in the specific queue for 3 minutes. Messages with the same destination will be placed in the same queue. When having an Edge server this are external domains or external send connectors. When you dont have an Edge server these can also be seen on the Hub server. Besides this the Hub server contains queues for messages that need to be delivered to other Hub servers in other site(s). Poison message queue, Exchange will move messages that may screw up your Exchange environment after a server failure. All messages which are placed in this queue will be permanently suspended so they will not be automatically be processed further. An administrator can check this queue and may decide to process the message further, this is a manual task. If an administrator thinks the message may screw up his environment he may delete the message from the queue. Messages which will end up in this queue may be harmful in their content and/or format. Another reason can be an agent which caused the transport engine to fail when it processed the message. When this queue is empty you wont see it when using the queue viewer or the get-queue command. Undeliverable queue, this queue contains messages that cant be routed to their destination s. This can happen in several cases: the mail server responsible for receiving the mail is not available, the internetconnection has dropped, etc.

Figure 1. Overview of the transport pipeline

In both Exchange 2007 and Exchange 2010 the queues are stored in an ESE database. The default location of the database is C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue. By default, this directory will contain the following files:

As all ESE databases this database has also a log file and a checkpoint file. Changes are written to the log files and memory first before they are committed to the database. The reason for this is to improve the performance. After the log file is flushed to the database Exchange will update the Checkpoint file to contain the name of the latest log file flushed to the database. If a server fails the Checkpoint file is used for recovery purposes to determine where to start the actual recovery. In fact, this principle is the same as for the normal Exchange database and its log files. There is only one difference between ESE databases used for mailboxes and the ESE database used for the queue. Circular logging is on by default, on mailbox databases this is not recommend in most cases but can be enabled in certain circumstances for example during a migration of a large environment. This has one disadvantage on the queue database, it cant replay log files which are restored from a backup to recover the transport database.

Transport queue management

For managing the queues you can use the following tools:

Queue viewer Powershell, the Exchange Management Shell contains several commands to manage the queue: getqueue, suspend-queue, retry-queue, get-message, resume-message, remove-message and exportmessage.

Queue viewer
Lets start with the queue viewer the tool can be opened by performing the following steps:

Open the Exchange Management Console Select the toolbox In the right piece of the screen select the Queue Viewer

The queue viewer will open the local transport database by default if available, but you can also open the transport databases located on other HUB Transport servers in the same Exchange organization. What the GUI does is executing Powershell cmdlets to retrieve the information, in case of getting an overview of the queues the get-queue cmdlet is executed. One exception to this is the queue viewer that is opened on the Edge Transport server, this one cannot connect to other server but can only view the transport database of the local server.

Figure 2. The Queue Viewer as part of the "Tools" option in Exchange Management Console

There are two tabs: queues which will display the current queues and messages which will display the messages that are in a queue on the specific server. When the queue tab is selected you may see a lot of queues which may make it a little bit difficult to have a good overview. For this issue you can use the filter option which will let create a filter to display only the information you would like to see. Follow the steps below to create a filter:

Select a queue property, for example next hop domain Select a comparison operator, for example = Enter a value , for example Microsoft.com

This filter will only display queues which have as next hop domain microsoft.com. Another option is to create a filter which only filters queues which have a specific status. This can be useful if you only want to see queues that contain messages which are not delivered yet. To create an overview of messages in a specific queue create a filter manually or just right click the queue and select the option view messages. This will open a new tab and will automatically apply a filter which selects only messages from the specific queue.

Using the queue viewer its possible to suspend the queue. This wil l freeze all messages that are current in the queue and will pause processing them until you resume the queue. In some cases a queue will have the status retry, in this case the Exchange server wasnt yet able to deliver the message(s) to the next hop mail server. If a queue is in retry state it will try to deliver the message(s) after one minute specified time. If you do not want to wait before its the retry counter has expired just select the queue and select the action retry. This will immediately retry to send the messages. Besides the filter that can be used for the queues, you can also create a filter for the messages tab. With this filter will make it possible to filter out messages with specific SCL value, source IP-address, subject, etc. The current status can be seen when getting the properties of the message, just as a queue a message can have several states:

Active, if in the delivery queue the message is being deliver to the next hop mail server, if in the submission queue the categorizer is processing the message. Pending remove, the administrator choose to remove the message from the queue but the message is already in the delivery queue. A message will be deleted if it reenters the queue because of an error but will be delivered if no error occurs. Pending suspend, the administrator choose to suspend the message in a queue but the message is already in the delivery queue. The behavior is the same as with the pending remove status. Ready, the message is waiting to be processed. Retry, Exchange couldnt deliver the message the last time it tried it, Exchange will try to deliver the message again if the retry counter expires. Suspended, the processing of the message has been suspended and the message will not be processed further till the administrator resumes the message.

As an administrator you will have a few actions when right clicking a message: suspend, resume and remove messages with or without NDR.

Exchange Management Shell

Via the Exchange Management Shell you can perform the sa me actions. Theres one thing you can do with the Shell which you cant do via the GUI, export messages. But lets start with the commands you can use to manage the transport queues. To view the current queues you will need to use the command get-queue, for example:

get-queue |fl

Figure 3. Queue Information using the Exchange Management Shell

Above you see an example of the output of get-queue |fl command. In this case only the submission queue exists and has zero messages in it. As with using the queue a filter can be applied to the queue, for this we will need to add some extra parameters:

get-queue Filter {messagecount eq 100 and status eq Retry }

The above example will display all queues which have more than 100 messages in it and which has the status of retry. To change the queue status we can use the following commands: suspend-queue, resume-queue and retryqueue. The first will freeze the queue and prevents messages from being processed further, resume-queue will start processing the messages again and last but not least retry will retry to send the messages in the queue instead of waiting till the retry counter has expired. Below some examples of the commands:

suspend-queue Filter {messagecount eq 100 and status eq Retry }

Will suspend all queues which have more than 100 messages in it and have the status retry.

Resume-Queue -Identity Hub\microsoft.com -Filter {status eq "suspended"}

Will resume the queue with messages send to contacts at Microsoft and have the status suspended.

Retry-Queue -Filter {NextHopDomain -eq "microsoft.com" -and Status -eq "Retry"}

Will resume all queues which contain messages that are send to contacts at Microsoft and have the status retry. There is only one exception for the retry-queue command, it cant be used with the parameter -Resubmit $true on the poison queue. If you wish to like to reprocess a message which is in the poison queue you will need to use the resume-message which is explained later. When you want to view messages in specific queues you can use the command get-message:

get-message -filter {queue eq servername\destination and FromAddress like user@domain.com. This will return all messages which are in the queue named destination on server servername and have user@domain.com as sender.

Just like with the queues you can also suspend, resume and remove messages from the queue. As said earlier there is one extra feature which can only performed by using PowerShell, exporting messages. But lets start from with the first command: suspend-message.

Suspend-Message -Identity hub\Unreachable\3

The above command will suspend the message with ID 3 which is in the queue called unreachable located on hub.

Suspend-Message -Filter {Subject -eq "Newsletter"}

This command will suspend all messages which contain the word Newsletter in the subject and will suspend the message(s) until the administrator will resume the message(s) by using the resume-message command.

Resume-Message -Identity hub\Unreachable\3

Applies the same filter as the suspend-message command only with the resume-message the message will be processed further if the current state is suspend.

Resume-Message -Filter {Subject -eq "Newsletter"}

Earlier we suspended all messages with the word Newsletter in the subject, by using the resume-message it will start processing the message again.

Retry-Message -Filter {Status -eq "Retry"} -Resubmit $true

The last command is the export-message command, this will let you export a specific message or a complete queue to eml files which can be opened via Outlook.

Export-Message -Identity Hub01\Microsoft.com\1234 -Path "C:\Microsoft Export\export.eml"

The above command will export the message 1234 thats located in the Microsoft.com queue on the Hub01. It will export the message to a file called export.eml and will be placed in the Microsoft Export directory. Before running this command you will need to meet a few prerequisites:

The directory where the files will be placed in must exist You must have write permissions to the directory

The above example was an easy one so lets give an example of a more difficult one:
Get-Message Filter {FromAddress like @microsoft.com} server Hub01 |% { $temp = $temp.Replace(<, _);););); $temp = $temp.Replace(>, _););); Export-Message $_.Identity Path $tempmp }

Quite a large command with all those parameters but what does it do, lets start from the beginning:

First it will retrieve all messages in the queue which have as senderdomain microsoft.com and are in a queue on Hub01. Then for each message it will find that meets the criteria it performs the action between {}, I will describe this step by step. First the $Temp parameter is set to the directory where we would like to store the exported messages, in this case c:\Microsoft Export\ and we will use the value of the InternetMessageID as the filename for the exported message. Because the InternetMessageID contains < and > brackets and this are characters which may not be used in filenames we will replace them by _ Next step if to perform the export itself, the $_.Identity will be automatically filled with the results of the get-message command. After that we tell the command to convert the binary data so we can save the message content in the location specified after the Path parameter.

For now this was the first part of the article. In the next part of this article we will dive deeper and will explain how you can change configuration settings related to transport queues.

HomeSQL.NETSysAdminOpinionBlogsForumsBooks Sign in | Join | Help

Sysadmin Home > Exchange > A Deep Dive into Transport Queues (Part 2)
/w EPDw ULLTE3M
Av rating: Total votes: 13 Total comments: 0 send to a friend printer friendly version

A Deep Dive into Transport Queues (Part 2)

09 February 2010 by Johan Veldhuis

Johan Veldhuis completes his 'Deep Dive' by plunging even deeper into the mysteries of MS Exchange's Transport queues that are used to temporarily store messages which are waiting until they are passed through to the next stage, and explains how to change the way they work via configuration settings.

In part one I explained how the transport queues in Exchange work and which commands are available to manage the queues and messages in it. In this part we will dive deeper and will explain how you can change configuration settings related to transport queues.

Modifying the retry configuration for mail queues and mailboxes

Most configuration things can be done either by making changes via the GUI or via PowerShell commands. But there are some things in Exchange which can only be configured by changing a configuration file, with the retry configuration for mail queues this is the case. The configuration of these can be found in the edgetransport.exe.config file which is a XML file and can be located in C:\Program Files\Microsoft\Exchange Server\V14\Bin For the retry part there are a few keys:

<add key="QueueGlitchRetryCount" value="<Integer>" /> <add key="QueueGlitchRetryInterval" value="<hh:mm:ss>" /> <add key="MailboxDeliveryQueueRetryInterval" value="<hh:mm:ss>" />

With the Queue glitch retry count you can specify how many times Exchange will retry to send a message immediately in case of an issue reaching the destination server. Default this value is set to 4 but it can contain a value between 0 and 15. The queue glitch retry interval will let you specify the time between the glitch retries. Default this is set to 1 minute. The last key called mailbox delivery queue retry interval will let you specify how frequently the Hub transport server tries to deliver the messages located in the delivery queue. The default value for this key is set to 5 minutes. When you made changes to the file you will need to restart the Microsoft Exchange Exchange Transport (MSExchangeTransport) Service before the changes will take effect.

Transient failures
Related with the queue glitch retries are the transient failure retries. After the attempts configured with the queue glitch retry count Exchange will keep try sending the message. The connection attempts are configured as the transient retry count, the time between the attempts is configured as the transient failure retry interval. The transient retry count and interval can be configured both via PowerShell and the GUI. Lets start with the GUI: For the Hub:

Open the Exchange Management Console Expand the server configuration Select the Hub Transport server

In the upper part of the result screen right click on the server Select the limits tab Modify the value in the field Transient failure retry attempts Modify the value in the field Transient failure retry interval (seconds)

For the Edge:

Open the Exchange Management Console Select the Edge Transport server Click the properties link under the server name Select the limits tab Modify the value in the field Transient failure retry attempts Modify the value in the field Transient failure retry interval (seconds)

Via PowerShell you will need to execute the following command:

Set-TransportServer Hub01 TransientFailureRetryCount 8

This will set the attempts to 8 so Exchange will try to deliver the message 8 times after the queue glitch retries.

Set-TransportServer Hub01 -TransientFailureRetryInterval 00:01:00

With the command above we will tell Exchange to wait 1 minute between the connection attempts.

Outbound connection failure retry interval

With the outbound connection failure retry interval you can specify the retry interval for outgoing connection attempts that have previously failed. The connection attempts which have previously failed are controlled by the TransientFailureRetryCount and the TransientFailureRetryInterval parameters. The default value of this parameter on the hub, which is 10 minutes, differs from the default value of the Edge which is 30 minutes. On both server roles this setting can be changed via the EMC or EMS, lets start with the GUI. For the Hub:

Open the Exchange Management Console Expand the server configuration Select the Hub Transport server In the upper part of the result screen right click on the server Select the limits tab Modify the value in the field Outbound connection failure retry interval (minutes)

For the Edge:

Open the Exchange Management Console

Select the Edge Transport server Click the properties link under the server name Select the limits tab Modify the value in the field Outbound connection failure retry interval (minutes)

If you like to change it via PowerShell then execute this command:

Set-TransportServer Exchange01 -OutboundConnectionFailureRetryInterval 00:60:00

The command above will set the Outbound connection failure retry interval to one hour. To make the previous values more clear below a workflow which gives a graphical overview:

Figure 1 Workflow of queue

Modifying the retry, resubmit and expiration configuration for messages

Configuring the retry and expiration configuration for messages can be done by executing PowerShell commands and some by the GUI. Modifying the resubmit configuration can only be done by modifying the edgetransport.exe.config.

Message retry interval

With the message retry interval you can configure how much time will need to expire after which Exchange will retry to deliver a message that is in the retry state. By default this is 1 minu te and Microsoft doesnt recommend changing this value. If you would like to change it execute the following command:

Set-TransportServer Hub01 -MessageRetryInterval 00:05:00

With this command you will configure the server called Hub01 to deliver a message which is in retry state after 5 minutes.

Message expiration timeout

With the message expiration timeout you can configure how long Exchange will try to deliver the message before its removed from the queue, the default value for this is 2 days. This can be either done via PowerShell or via the GUI. For example:

Set-TransportServer Hub01 -MessageExpirationTimeout 4.00:00:00

This command will set the expiration timeout to 4 days. This means that after Exchange has tried to deliver the message for 4 days but has been unsuccessful the message will be deleted from the queue. If this happens the user will be informed with a message send to the user to inform him/her that the message can be delivered, this is also called a Non-delivery report (NDR). To modify the configuration of the expiration timeout via the GUI follow the steps mentioned below: For the Hub:

Open the Exchange Management Console Expand the server configuration Select the Hub Transport server In the upper part of the result screen right click on the server Select the limits tab Modify the value in the field Maximum time since submission

For the Edge:

Open the Exchange Management Console Select the Edge Transport server Click the properties link under the server name Select the limits tab Modify the value in the field Maximum time since submission

Resubmit Interval
With this parameter you can configure how much time needs to expire before a message is automatically resubmitted if the queue is in retry state. This can be either the mailbox delivery or remote delivery queue. This is only valid when the message itself isnt in the suspended state. As earlier mentioned the resubmit interval can only be changed by modifying the edgetransportconfig.exe.config. This file will contain the following line in the <appSettings> section:

<add key="MaxIdleTimeBeforeResubmit" value="<hh:mm:ss>" />

Delay DSN messages

Exchange will send a message automatically to a user when a message has not been delivered in a specific amount of time. Default a delay message is send to a user when the mail hasnt been delivered within 4 hours. The value can be changed either via the EMC or EMS: For the Hub:

Open the Exchange Management Console Expand the server configuration Select the Hub Transport server In the upper part of the result screen right click on the server Select the limits tab Modify the value in the field Notify sender when message is delayed more than (hours)

For the Edge:

Or

Open the Exchange Management Console Select the Edge Transport server Click the properties link under the server name Select the limits tab Modify the value in the field Notify sender when message is delayed more than (hours)

Set-TransportServer Hub01 -DelayNotificationTimeout 06:00:00

Which will configure the server to send a notification if the message hasnt been delivered after 6 hours. Keep in mind that you will need to specify a value which is greater than the TransientFailureRetryCount x TransientFailureRetryInterval. As already mentioned this functionality is enabled by default but can be disabled if you want to prevent this kind of messages being sent to users:

Set-TransportServer Hub01 -InternalDelayDSNEnabled $false

Which will prevent delay messages send to internal users.

Set-TransportServer Hub01 -ExternalDelayDSNEnabled $false

Result is the same as the previous command only for external users.

Configure the transport database

As explained in part one all queues are placed in an ESE database. To change the configuration you will need to modify the edgetransport.exe.config. This file contains the parameters listed below
parameter QueueDatabaseBatchSize description Specifies how many database I/O operations will be grouped together before theyre executed Specifies the maximum time in ms that the database will wait for multiple I/O operations to group before they will be executed. After the time has expired the I/O operations are executed without a warning, this happens only if one or more of the following conditions is true: The amount of I/O operations defined by the QueueDatabaseBatch Size hasnt been reached The time specified by the QueueDatabaseBatch Timeout parameter has passed Specifies the number of maximum connections which can be made to the ESE database Specified the amount of memory that can be used to cache transaction records. Specifies the maximum size of transaction log files The path where the queue database log files are stored. Specifies the maximum number of background cleanup work items that can be queued to the database engine thread pool at any time Enables or disables online defrag of the ESE database Specifies when the online defrag needs to be executed Specifies the maximum duration an online defrag may take The path where the transport database is stored default value 40

QueueDatabaseBatchTimeout

100ms

QueueDatabaseMaxConnections

QueueDatabaseLoggingBufferSize QueueDatabaseLoggingFileSize QueueDatabaseLoggingPath*

5242880 bytes 5242880 bytes Exchange TransportRoles\ data\Queue 32 Server\

QueueDatabaseMaxBackground CleanupTasks

QueueDatabaseOnlineDefrag Enabled QueueDatabaseOnlineDefrag Schedule QueueDatabaseOnlineDefrag TimetoRun QueueDatabasePath*

$true 1:00:00 (hh:mm:ss) 3:00:00 (hh:mm:ss) Exchange TransportRoles\ data\Queue Server\

*Before changing this parameters make sure the directory exists where you want to create the files. Besides this please check that the following users have full control on this directory: Network Service, System and Administrators.

Recover transport queues

How do you recover the transport queue if the Hub server fails? In this chapter I will describe how you can recover those messages and perform maintenance on the transport database. One scenario for example can be that the database will get so big so it cant be stored on the volume anymore. Another scenario may be that you cant bring a server back online which still has messages in the queue.

These are the requirements for recovering a database to another Hub server:

Exchange version must be the same, so same service pack and rollup. Must have the same transport role installed as the source server, its not recommend to recover a database from a Hub to an Edge. Ideally the Hub server must be placed in the same AD site.

Before starting the recovery we need to move the transport database and log files to another temporary location. Because the files may be in use you will first need to check if the Microsoft Exchange Transport service is stopped, this can be done via the services.msc management console. If the service is running you can stop it either via services.msc or via a command prompt by executing net stop MsExchangeTransport. Once the service is stopped you can move the files located in the queue database directory (c:\Program Files\Microsoft\Exchange\v14\Transport Roles\data\queue) to a temporary folder, for example Recovery. To continue mail delivery as soon as possible you could decide to start the Exchange Transport service again, if the cause of the failure is database related. During the startup of the service Exchange will check if a database exists and if not will create a new one. Just as with a mailbox database you will need to use eseutil to repair or perform maintenance on the transport database. The only difference between them is that the transport database uses circular logging which doesnt give you the option to replay the log files. Using the /r parameter you can start the repair of the database which will check the checkpoint file which log files have been committed to the database and which havent. The uncommitted log files will then be replayed to the database. If no checkpoint file exists, all log files are replayed, starting with the oldest first. Heres an example, the database and log files are placed in a temporary folder called Recovery. To start the recovery of the database we will need to run the following command:

Eseutil /r Trn /d c:\Recovery /8

Since all log filenames start with Trn we specify that as the name of the log file. Using the /d parameter we can specify the directory where the files are located and using the /8 parameter we will set the database page size to 8 kilobyte which will allow a faster recovery. Once the database has been fixed the last action will be to place it back to its original location so messages which are in the queue database can be delivered. As we have implemented a temporary queue database so that users can continue to receive messages we first need to make sure the temporary queue database is empty. For this we will need to pause the Exchange Transport service, this can be done either via the services.msc or via a command prompt by executing net pause MsExchangeTransport. Once it has stopped keep an eye on the queues using the Queue viewer or get-queue to monitor the delivery of current messages. When the queues are empty they will be displayed with a message count of 0. It can happen that after performing these steps there are still some messages in the unreachable and poison queue. To force the delivery of these messages run the following PowerShell command, keep in mind that the poison queue might contain messages which are harmful for your environment:

Retry-Queue -Identity "Unreachable/poison" -Resubmit $True

If messages still stay there then there is only one option and that is to export the messages:

Get-Message -Queue "Unreachable/poison" | Export-Message -Path "<ExistingLocalDirectory>"

To prevent NDRs being generated you may need to modify the message expiration time -out. This is only necessary in the following situations:

The recovered database has been offline for more than 2 days. The recovered database contains messages which cant be delivered to their final destination within 2 days.

You can modify the message expiration time-out by running the following Powershell command:

Set-TransportServer -MessageExpirationTimeout <LongerTimeOutIntervalIndd.hh:mm:ssFormat>

Now that all preparations have been made its time to place the recovered database back to its original location. You will need to begin by stopping the Microsoft Exchange Transport service, this can be done either via the services.msc or via the command prompt (net stop MsExchangeTransport). Once the service has been stopped, copy the files from the recovery database back to the original location, which is c:\Program Files\Microsoft\Exchange\v14\Transport Roles\data\queue by default. OK now that the recovered database is in place lets fire up the Microsoft Exchange Transport service ( net start MsExchangeTransport). Then keep an eye on the queues to see if mail will be transported correctly. It may be necessary to resubmit messages which are stuck in several queues with the retry state. You might choose to replace the recovered database again after all messages have been delivered. This can be the case if the database gets corrupted again or messages will not leave the queue and you are unable to remove them. In that case stop the Exchange Transport Service, delete all files from the queue directory and start the Exchange Transport service again.

Performing maintenance on the transport database

Sometimes it may be necessary to perform maintenance on the transport database, for example the size on the disk where the database is located does not have enough space left. In that case you must use Eseutil with the /d parameter, this will perform an offline defrag of the transport database. Keep in mind that this offline defrag will create another database and therefore you will need to have enough disk space or place the other database on another volume by providing the /t parameter.

Eseutil /d mail.que /td:\tmpdb.que

The command above will perform an offline defrag on the mail.que database file but will place the temp file on the D drive. Here ends the second part of a deep dive into transport queues which is also the last part of it. I hope you enjoyed reading if and if you got any questions dont hesi tate to contact me: j.veldhuis@dm-consultants.nl