Disclaimer
This document is for informational purposes only and is subject to change without notice. This document and its
contents, including the viewpoints, dates and functional content expressed herein are believed to be accurate as of its
date of publication. However, Epicor Software Corporation makes no guarantee, representations or warranties with
regard to the enclosed information and specifically disclaims any applicable implied warranties, such as fitness for a
particular purpose, merchantability, satisfactory quality or reasonable skill and care. As each user of Epicor software is
likely to be unique in their requirements in the use of such software and their business processes, users of this document
are always advised to discuss the content of this document with their Epicor account manager. All information contained
herein is subject to change without notice and changes to this document since printing and other important information
about the software product are made or published in release notes, and you are urged to obtain the current release
notes for the software product. We welcome user comments and reserve the right to revise this publication and/or
make improvements or changes to the products or programs described in this publication at any time, without notice.
The usage of any Epicor software shall be pursuant to an Epicor end user license agreement and the performance of
any consulting services by Epicor personnel shall be pursuant to Epicor's standard services terms and conditions. Usage
of the solution(s) described in this document with other Epicor software or third party products may require the purchase
of licenses for such other products. Where any software is expressed to be compliant with local laws or requirements
in this document, such compliance is not a warranty and is based solely on Epicor's current understanding of such laws
and requirements. All laws and requirements are subject to varying interpretations as well as to change and accordingly
Epicor cannot guarantee that the software will be compliant and up to date with such changes. All statements of
platform and product compatibility in this document shall be considered individually in relation to the products referred
to in the relevant statement, i.e., where any Epicor software is stated to be compatible with one product and also
stated to be compatible with another product, it should not be interpreted that such Epicor software is compatible
with both of the products running at the same time on the same platform or environment. Additionally platform or
product compatibility may require the application of Epicor or third-party updates, patches and/or service packs and
Epicor has no responsibility for compatibility issues which may be caused by updates, patches and/or service packs
released by third parties after the date of publication of this document. Epicor is a registered trademark and/or
trademark of Epicor Software Corporation in the United States, certain other countries and/or the EU. All other
trademarks mentioned are the property of their respective owners. Copyright Epicor Software Corporation 2014.
All rights reserved. No part of this publication may be reproduced in any form without the prior written consent of
Epicor Software Corporation.
100_700
Revision: July 03, 2014 12:13 p.m.
Total pages: 70
sys.ditaval
Contents
Contents
Introduction............................................................................................................................5
Purpose of this Guide.......................................................................................................................................5
Intended Audience...........................................................................................................................................5
How it is Organized.........................................................................................................................................5
Common Patterns...................................................................................................................9
Frequent User Complaints................................................................................................................................9
Everything Is Slow............................................................................................................................................9
Sometimes Everything Is Slow..........................................................................................................................9
Specific Program Is Slow................................................................................................................................10
Reboot Corrects Poor Performance................................................................................................................10
Unable to Reproduce by Epicor Technical Support..........................................................................................10
Standard Metrics...................................................................................................................11
SAN (I/O) Test Utility............................................................................................................13
Run the SQLIO Utility (Standard Metric)..........................................................................................................13
Contents
Scenarios................................................................................................................................................38
Load Distribution..................................................................................................................46
Application Servers........................................................................................................................................46
Task Agents...................................................................................................................................................46
Create Task Agent Rules................................................................................................................................47
Application Tuning...............................................................................................................49
Application Troubleshooting..........................................................................................................................49
Client Cache..................................................................................................................................................49
Locate the Client Cache..........................................................................................................................50
Memory Cached Programs.............................................................................................................................50
Client Customizations....................................................................................................................................50
Order Entry Performance Tuning....................................................................................................................51
MRP Performance Tuning...............................................................................................................................52
BAQ Performance Tuning...............................................................................................................................54
BAQ Server Settings................................................................................................................................54
BAQ Best Practices..................................................................................................................................56
SQL Syntax Issues....................................................................................................................................57
Introduction
Introduction
Intended Audience
The guide is intended for technical consultants, partners, and system administrators to help ensure the Epicor
ERP application performs as expected and to provide guidance on performance areas that should be addressed
before contacting Epicor consultants or Epicor Technical Support.
Individuals who perform all or some of these tasks will benefit from reviewing the Performance Tuning Guide.
How it is Organized
This guide explains how you can test the performance of your Epicor application.
The following are the main sections of this guide:
Performance Tuning Resources - Details the performance tuning resources available for use with Epicor
ERP. Review this section to learn more about each primary resource.
Common Patterns - Describes the common patterns of poor performance. Each pattern is described along
with suggestions for testing and potential solutions.
Standard Metrics - Epicor has established benchmark metrics for optimal performance. This section of the
guide contains a table that describes each benchmark metric and the tests you run to evaluate it.
SAN (I/O) Test Utility - You can use the SQLIO Utility to evaluate various parameters of an input/output
(I/O/) disk subsystem. Use this section of the guide to learn how to download and run this key Microsoft
utility.
Performance and Diagnostic Tool - Documents how you install and use the Performance and Diagnostic
Tool, the key program you will use to evaluate the performance of client and server installations. This section
documents how you activate the client logs and server logs for the Epicor application and then view them
within the Performance and Diagnostic Tool. It also describes the features, sheets, and fields available within
this tool.
Application Tuning - This section contains some application tips and techniques you can do inside the Epicor
application. You should only use the tips and techniques that match your use of the Epicor application.
Introduction
Baseline Performance Tests - Provides detailed instructions for performing primary tests to measure how
well your installed system performs. Epicor recommends you run these tests after the application is setup and
configured to verify the Epicor application has optimal performance. You can also run these tests periodically
later on to make sure the application performance has not degraded.
Common Patterns
Common Patterns
This section describes some common patterns for poor performance and the causes for these patterns.
If you are experiencing any of these common patterns, be sure to use the Performance and Diagnostic Tool as
needed to analyze the situation. This tool is discussed later in this guide.
Everything Is Slow
Everything is slow all of the time.
The most likely cause behind this pattern is the hardware sizing for the production environment is not correct.
Use the Hardware Sizing Guide to determine the recommended hardware; this guide evaluates data transaction
traffic, number of users, and the system software to determine the recommended hardware.
Common Patterns
10
Standard Metrics
Standard Metrics
Epicor has developed a Standard Metrics table that identifies the benchmarks for optimal performance. Compare
these benchmarks against the customer system to locate areas of poor performance.
These performance benchmarks can identify key areas of improvement. Once these performance values are
determined, customers and Epicor representatives can investigate potential ways these performance values can
be tuned to achieve better results. This section first contains the Standard Metrics table and then a second example
table that illustrates a specific company's metrics compared against these standard metrics.
Epicor Standard Performance Metrics
Performance Metrics
CPU Speed
(Config Check)
GetRowsKeepIdleTime
(GetRowsKeepIdleTime
Chart)
Network Test
(Network Diagnostics)
Configuration Check
(Config Check)
Customer Retrieval Test
(Baseline Test)
Sales Order Test
(Baseline Test)
400 milliseconds
This tests the performance of Sales Order Entry, Add 20 lines Sales Order < 72
a commonly used, important business function. seconds
This test does not check customizations or
BPMs.
This tests the performance of Purchase Order
Entry, a commonly used, important business
function. This test does not check
customizations or BPMs.
The following sections describe how you test for these standard metrics.
11
Standard Metrics
Expected Result
CompanyA
Production
CompanyA
Physical
26 Mb/Sec
N/A
< 5 Milliseconds
37 Milliseconds*
N/A
CPU Speed
3500 Milliseconds
2713 Milliseconds
GetRowsKeepIdleTime
< 30 Milliseconds
80 Milliseconds*
20 Milliseconds
N/A
N/A
N/A
N/A
Configuration Check
Pass
Pass
Pass
<1 Second
1.2 Seconds
.8 Seconds
< 72 Seconds
75 Seconds
64 Seconds
< 51 Seconds
49 Seconds
43 Seconds
* The bold values indicate a performance metric value that exceeds the value of an Epicor standard metric.
12
13
Use the results of each test to evaluate how your disk I/O subsystem works in comparison to similar subsystems.
14
performance of your Epicor ERP application. You can also export these results to Microsoft Excel for additional
review and analysis.
Through the Performance and Diagnostic Tool, you can evaluate:
The performance of one client versus another client on the same system.
The performance of business object methods on both the client and the server.
Overall performance of the server and the network.
Performance of business objects in one system against the same business objects on other systems.
Performance of customizations, personalizations, Business Process Management (BPM) methods, and business
activity querys (BAQs).
The configuration of the Epicor ERP application.
Important The Performance and Diagnostic Tool released with Epicor ERP version 10 is only compatible
with the 10.0.600 version or higher. If you need to evaluate the performance of Epicor ERP 9.05 or earlier,
download the Performance and Diagnostic Tool released with the 9.05 version.
Installation
Follow these steps to install the Performance and Diagnostic Tool.
Tool Requirements:
.NET 4.5 Runtime
Windows 2008 R2 or Higher
Tip The Performance and Diagnostic Tool can be installed on any desktop or laptop machine where you
normally install the Epicor client. When using the tool, you will be able to connect to application servers
on other machines.
1. From Windows Explorer, navigate to the C:\Epicor installation folder. Expand the folder for your current
version and then open the Utilities\ICE3 Performance Diagnostic Tool folder.
Example C:\Epicor\ERP10\ERP10.0.700\Utilities\ICE3 Performance Diagnostic Tool
15
Client Diagnostics
You analyze the performance of client installations through client tracing logs. This section of the guide describes
how you set up these logs and analyze their results in the Performance and Diagnostic Tool.
16
11. For the Client Trace File Path, either paste the path you copied from the Current Log File field or click the
Browse button to find and select client log file.
12. Optionally, select the Exclude System Calls check box to prevent system method calls from displaying with
diagnostic results.
The Performance and Diagnostic Tool will then not display values from the GetRowsKeepIdleTime business
object method. This method runs in the background while a current form is active, but it does not indicate
how long it takes other methods like Update, GetNew, and GetList to run. These method calls are generated
by user activity.
However if you are testing how the client interacts with the server when no user activity is taking place, you
will want to review the calls from the GetRowsKeepIdleTime method. In this case, do not select this option
so you can displays these calls in the client results.
13. Notice you can also change the Log File Date Format to select a date format used in a different locality
or enter your own Custom Format.
14. Click on the Generate Diagnostics button.
You can now analyze the trace log data within the Results and Summary sheets.
2. You can filter the log results based on values that populate each column. To do this, click on the Funnel
icon that displays on each column header:
17
3. When you click on an icon, a drop-down list displays. Each drop-down list contains a series of default filter
options followed by specific values from the trace log:
All - The default filter, this option displays all the values for the column from the client tracing log.
Custom - Select this option to launch the Enter filter criteria for Type window. Use this window to
define both an Operator and an Operand to filter the results. You can define as many filter conditions
as you need, creating And/Or relationships between each custom filter.
Blank - Use this option to only display any empty rows available on the client trace log.
Nonblanks - Select this option to display all the populated rows on the selected client trace log.
Column Value - You can filter the results by selecting a specific value available in the column.
4. To help organize the results, use the Group By feature on the grid. Click and drag a column header (for
example, Object Name) to the Drag a column header here to group by that column area. You can drag
multiple column headers to further structure the results as you need.
Example If you group by using the Business Object column, the grid displays all the business objects
recorded on the client trace in alphabetical order.
5. You can review the following items on each business method call:
Type
Object Name
Method Name
Start Date
End Date
Execution Time
File Name
Server Execution Time
Network Transport Time
Appserver Thread
6. Click on the Summary tab.
Each business object and method displays on this grid.
7. The Trace Date/Time and File Name fields display the selected trace log as well as the date and time it
was run.
8. In the Report Monitor section, notice the GetRowsKeepIdleTime method. This business object runs in
the background while a current form is active.
Tip If you shut off the System Monitor, the tracing log did not record time against the
GetRowsKeepIdleTime method.
9. The All grid displays the summaries of the method calls recorded on the tracing log.
10. The Calls column indicates how many times the method sent a call to the server.
18
11. The execution times , in milliseconds, for each call are calculated and display in the accompanying columns.
Additional information displays that identifies the object information:
Average
Longest
Least
Diagnostics ID
Type
Object Name
Total
12. Notice you can view the results in a Microsoft Excel file. To do this, click the Export to Excel button.
13. Save the .xls file to directory you can access.
You can now launch the .xls spreadsheet and review it in Microsoft Excel.
14. To remove the generated results, click the Clear Results button.
Clear Results
Click the Clear Results button to remove data from the Results and Summary sheets.
You can also continue generating data from different trace logs without clearing the data. The Summary sheet
then contains results from each trace log generation.
Custom Format
If the date format you need is not available on the Log File Date Format drop-down list, enter the Language
ID for the language you use in this field. The date format linked to this language loads into the Performance and
Diagnostic Tool.
Export to Excel
Click the Export to Excel button to export results to Microsoft Excel for further manipulation and analysis. The
exported data is based on which grid is currently visible. For example, if the Results grid is visible, the data on
this grid is exported.
19
Generate Diagnostics
Click the Generate Diagnostics button to generate the log file results and display these results in the Summary
and Results grids.
Results Sheet
The Results sheet contains all calls made from this client for the duration of the current trace log. This grid supports
Group By functionality, so you can organize the results by Object and/or Method name to view execution times
for all method calls.
Summary Sheet
The Summary sheet summarizes the results for each BusinessObject.Method. This sheet displays the number
of calls and the Average/Least/Longest duration of these method calls. The duration is measured in milliseconds.
You can also group the summary data by Diagnostic ID, Type, Object Name, and Total.
Likewise you can sort the results by selecting the column that contains the most useful information to determine
the type of calls made most frequently by this client -- or which method calls consume the most time.
20
Scenarios
This topic contains examples of how you can measure client performance through different methods.
Comparison of client performance between locations and/or client systems.
You may have specific users reporting performance issues from their systems. By running client diagnostics from
multiple desktops/laptops that perform similar Epicor functions, you can determine differences in client execution
times for the same method calls.
This may provide you with some things to consider with your network, especially when comparing LAN users
against WAN users. However CPU differences will probably not be determined, because the tool is capturing
execution time of the method calls, not the time taken to open forms or render the data within these forms.
Identification of business object performance issues.
You may suspect a business object performs poorly under certain conditions. For example, you notice that
transaction entry slows down as the volume of data increases. By monitoring execution time of method calls
during entry, you may be able to measure the effect.
Identification of performance over time.
By analyzing the execution times of similar method calls throughout the day, you should be able to identify slow
periods. These slow periods may be related to network overload or server activity during those periods.
Identification of Epicor usage from a client.
You can analyze the log from a client to determine what activities that client engages against the Epicor application.
This method also measures the frequency of calls to business object methods from that client.
Identification of customization issues.
Client customizations may cause unexpected or excessive calls to business object methods. For example, a poor
customization could cause multiple calls to methods not made by the base application. Compare the trace logs
that track the same functions with and then without client customizations; this method may identify performance
issues.
21
2. Click on the Settings tile; from the General Options list, select Tracing Options....
The Tracing Options window displays.
3. Select the Enable Trace Logging check box.
4. Now select the Track Changes Only check box. Only update, new, delete, and other method calls that
change the database are included on the trace log.
5. Select the Include Server Trace check box. This option causes calls from the client to the server to be
included in the trace log.
6. Click the Clear Log button.
7. Now launch the program that has poor performance, following the interface movement pattern and data
entry pattern that demonstrates the issue.
Tip Typically you should have the end user who reported the issue follow the pattern that causes the
poor performance. The data the user enters is also very important. The performance issue can be
triggered by many factors -- the data the end user enters, the options the end user selects, the
navigation path the end user follows, and so on.
Configuration Check
The Performance and Diagnostic Tool contains a utility to check the configuration of the application server. Use
this Config Check option to see what issues and potential issues you may have with the application server
configuration.
This feature checks a number of configuration items, including the CPU Speed and Configuration Check
standard metrics. After the Performance and Diagnostic Tool analyzes the configuration, this feature displays
recommended actions you can follow to fix various issues.
22
3. If you access the application server through an http link, select the Connect using Http check box. You
then enter the link in the http:// field.
4. By default you enter the net.tcp// connection.
Tip Get the correct URL that points to the application server from the .sysconfig file that runs this
instance of the Epicor application. You can find this value in <appSettings><AppServerURL>
configuration setting. In your Epicor application folder, the .sysconfig files are located in the client\config
folder.
5. Use the Connection Method drop-down list to indicate how this application server checks for authentication
certificates through Internet Information Services (IIS). When a user logs into the application, the selected
method checks whether the user can access the Epicor application. Available options:
UsernameWindowsChannel -- Authenticate using an Epicor Username and Password. Windows checks
for existing Epicor user accounts to authenticate logins. When you select this option, you need to enter
the User ID and Password for the account. You can select this option for both smart client and Epicor
Web Access (EWA) installations.
UsernameSSLChannel -- Use this option to authenticate using a Secure Sockets Layer (SSL) X509
certificate. Use this method for application servers that handle installations where users reside in different
23
domains. By using an SSL certificate, users from these different domains can log into the Epicor application.
You can select this option for both smart client and Epicor Web Access (EWA) installations.
Windows Authentication -- Select this option to authenticate using a Windows Username and Password.
You can select this method for application servers that handle client installations where users access the
application through the same domain. Any user who has a Windows Username and Password within
this domain can successfully log into the Epicor application. You can only select this option on smart
client installations.
Tip You can also find the Connection Method in the .sysconfig file. Locate the <EndpointBindi
ng> value to see the method used the application server.
6. Enter the User Id and Password for the Epicor user account used to access the application server.
7. Next enter the Client Directory for the folder that contains the client installation for the Epicor application.
You can enter this path directly or click the Browse (...) button to find and select it.
8. If you need to validate this connection through a certificate, select the Requires Certificate check box. The
accompanying directory path field activates; enter the location for the certificate in this field.
9. Click OK.
3. The Result column displays the generated evaluation of each rule against your application server
configuration. Available results include FAIL, WARNING, EXISTS, INFO, Not Available, and PASS.
24
4. Depending on the results, different instructional text displays in the Action Required column.
INFO - Displays some key information you should review to make sure your system is set up correctly.
EXISTS Notifies you that various items, like customizations and BPM directives, are active in the current
system. These items should be evaluated for performance.
PASS The configuration met or surpassed these rule requirements. No further action is needed.
WARNING Alerts you that potential performance issues may occur. Review these items to see if further
changes are needed.
FAIL The application server configuration did not meet the rule requirements. The ActionRequired
column displays a recommended action you can do.
Not Available - Displays any item that was not available to test. You may need to correct some setup
configuration items and re-run the configuration check.
5. The Config Check Details sheet displays the various rule keys run to evaluate each configuration rule.
Expand one of the rules to see the specific calls.
6. Notice you can view the results in a Microsoft Excel file. To do this, click the Export to Excel button. The
information from all the Configuration Check sheets display in the exported spreadsheet.
7. Save the .xls file to directory you can access.
You can now launch the .xls spreadsheet and review it in Microsoft Excel.
8. To remove the generated results, click the Clear Results button.
Network Diagnostics
You can use the Performance and Diagnostic Tool to verify the baseline network and server performance are
running at optimal levels. Use this feature to evaluate the Network Test standard metric.
To do this, run multiple tests to gauge the overall performance of your network, and compare these results
against the network standard metric. Just like the Configuration Check, you need to update the fields on the
Settings > Options window so the tool connects to the application server.
25
Expected Results:
Server Time (blue/lower bar) < 0.5 Seconds
Network Time (green/upper bar) < 0.4 Seconds over a LAN
Network Time (green/upper bar) < 7 Seconds without compression or < 1.5 seconds with compression over
a WAN
Server Diagnostics
You analyze the performance of server installations through server tracing logs. This section of the guide describes
how you set up these logs and analyze the results.
Write Permission
Be sure the user account you use to access the server machine has write permission to the selected folder. Do
the following to verify you have access.
1. Launch Internet Information Servers (IIS) Manager. Depending on your operating system, you launch
this tool in different ways:
a. If you are on Windows SQL Server 2008 R2 or Windows 7, click Start. In the Search field, enter Internet
Information Services; when this program appears in the results, select it.
b. If you are on Windows SQL Server 2012 or Windows 8, press the <Windows> + F button to display the
Charms bar; from the Apps screen, select Internet Information Services (IIS) Manager.
26
27
To avoid running into disk space issues, you can control the number of log backups you want to maintain. Use
the logCount attribute within the server log to specify how many backups you want to keep. By using the default
option logCount="0", an unlimited number of backups is created.
Each log also includes a series of server detail values. Depending on the purpose for creating the log, you may
not need to track all of these details. The <appserver>.config file includes a setting that prevents these details
from recording in the log.
If you wish to change this default size, do the following:
1. Launch Windows Explorer.
2. Navigate to the web.config file; this file is located in the server folder in your Epicor installation. For example:
C:\Epicor\ERP10\ERP10.00.000\Server
3. Open this file in Notepad or a similar text editor.
4. Locate the <configuration> section.
5. For the <Trace> setting, locate the traceConfigurationFile attribute and review the name of the application
server configuration file you use.
Example <Trace logLocation="" traceConfigurationFile="appserver.config"
></trace>
9. Update the maxLogFileSize value to the log size file you need.
You must enter a whole number followed by KB, MB, or GB (these values are not case-sensitive). For example,
you can enter "16KB", "300mb", "3GB", and so on. You can also enter "none", "unlimited", or "-1" to
indicate the server log should not have a maximum file size. The default size is 10MB; the minimum log file
size is 4KB.
When you have an application server log run for a long period of time, the size of the log file can grow very
large. Usually this will not be a problem, as the Performance and Diagnostic Tool can evaluate most logs
500 MB in size or less. However if you want to view the log in Notepad or a similar text editor, you should
keep each log between 5 - 10 MB in size. Notepad can easily open log files this size, and they typically
contain about the right amount of data for you to review.
When the log file size reaches this maximum limit, the Epicor Administration Console creates a new log file
with the UTC date and time values placed on the end of the filename. Because of this, the active log files
all use the same file name.
10. To prevent the Machine, Process, and Thread ID server details from appearing in the log, locate the
<includeServerDetails> setting. Change this setting to "False" and these items will not populate in the
log results.
28
Tip These server details may be important if you are evaluating the performance of a server farm.
For example, you can use the Machine values to review the performance of a specific server. However
if you are not working in a server farm environment, suppress these details.
29
30
f. ERP DB Hits - Activate this check box to track how the Epicor application interacts with the database.
You can review each database hit as well as how long it took each hit to complete.
g. Data Logging - This option will be available in a future version.
8. Indicate which Advanced Logging information you want to include on the application server log. These
options record calls from the overall system server, and may impact performance while active. Available
options:
a. System DB Hits - Select this check box to record all the hits the database receives from SQL Server. Use
these values to determine the performance of SQL Server.
b. System Table Methods - Activate this check box to track the method calls being placed against the
system tables.
9. When you finish making your selections, click Apply and then OK.
10. The Server Manager dialog box displays, asking if you want these log settings to activate. If this is a good
time to begin generating results in the application server log, click Yes.
The application server is now active and recording the selected trace call options. The server log should run for
a period of time, such as a single day or a couple days. When you are ready to review the results, you display the
application server log in the Performance and Diagnostic Tool.
Important After you gather the system information you need, be sure to return to the Epicor Administration
Console and de-activate your log setting options. This reduces unnecessary calls to the server and improves
performance.
31
2. You can filter the log results based on values that populate each column. To do this, click on the Funnel
icon that displays on each column header:
32
3. When you click on an icon, a drop-down list displays. Each drop-down list contains a series of default filter
options followed by specific values from the trace log:
All - The default filter, this option displays all the values for the column from the client tracing log.
Custom - Select this option to launch the Enter filter criteria for Type window. Use this window to
define both an Operator and an Operand to filter the results. You can define as many filter conditions
as you need, creating And/Or relationships between each custom filter.
Blank - Use this option to only display any empty rows available on the client trace log.
Nonblanks - Select this option to display all the populated rows on the selected client trace log.
Column Value - You can filter the results by selecting a specific value available in the column.
4. To help organize the results, use the Group By feature on the grid. Click and drag a column header (for
example, Object Name) to the Drag a column header here to group by that column area. You can drag
multiple column headers to further structure the results as you need.
Example If you group by using the Workstation ID column, the grid displays all the workstation
entries together in alphabetical order.
5. You can review the following items on each business method call:
UTC
Type
Object Name
Method Name
Execution Time
ERPUser
Client Workstation ID
Exceeds MS Threshold - If this check box is selected, it indicates this server call took longer to execute
than the Millisecond Threshold value you entered.
MoreInformation
TotalMilliseconds
6. The Summary Analysis sheet calculates the total performance results for each business object method.
Notice you can group the results by various columns on this grid.
7. You can review several items on each business method call. Some key columns on this sheet include:
Frequency - This value indicates how often the business object method sent a call to the server.
% - Freq - The percentage this call was run compared to other method calls captured in this log.
Total Execution Time
Average Execution Time
Longest
Least
% - Time - The percentage about how long this call was run compared to other method calls captured
in this log.
8. Review the Errors and Messages sheet to analyze error messages.
33
9. You can review several items on each error. Columns on this sheet include:
LogLine
ClientID
ObjectName
MethodName
StartDateTime
ERPUser
10. Notice you can view the results in a Microsoft Excel file. To do this, click the Export to Excel button.
11. Save the .xls file to directory you can access.
You can now launch the .xls spreadsheet and review it in Microsoft Excel. The information from all the Server
Diagnostic sheets display in the exported spreadsheet.
12. To remove the generated results, click the Clear Results button.
34
35
The chart displays using the time interval you selected. The left side of the grid indicates how long it took
to run the GetRowsKeepIdleTime call, while the bottom of the grid indicates the time of the day when the
call occurred.
9. Optionally, you can click the Export to Excel button to save this graph as an .xslx spreadsheet.
You can then use Microsoft Excel to refine this graph as you need. You and your Epicor consultant use
this spreadsheet to evaluate the overall performance of your system.
Fields
This topic documents the fields and sheets available for analyzing the server log files.
Some fields on the interface have a context menu, which is indicated by a triangle in the upper right corner of
the field. To open the context menu, right-click on the field.
Add Files
Click the Add Files button to select one or more server log files for analysis. Depending on what you are evaluating,
multiple server.log files may have been generated. You can then analyze these files together within the Performance
and Diagnostic Tool.
You can also click the Down Arrow next to the Add Files button to select server log files from multiple server
locations. You may need to do this on load balanced systems where the application is located across multiple
servers.
Clear Results
Click the Clear Results button to remove the log file information from the Results, Summary, Errors and Messages,
and GetRowsKeepIdleTime Chart sheets.
36
Clear Selected
Click the Clear Selected button to remove the log file paths you have currently loaded into the Performance and
Diagnostic Tool.
Epicor Version
Use this drop-down list to select the application version run with the selected server log(s). The default version
is 10.0.1.
Export to Excel
Click the Export to Excel button to export results to Microsoft Excel for further manipulation and analysis. The
exported data is based on which grid is currently visible. For example, if the Results grid is visible, the data on
this grid is exported.
Generate Diagnostics
Click the Generate Diagnostics button to create the server log file data results.
GetRowsKeepIdleTime Chart
Use the GetRowsKeepIdleTime chart to display the performance time of each GetRowsKeepIdleTime method call
in the server log. This method call is run by the System Monitor to check for server related activity like uploading
reports. You can then pinpoint specific times of the day when there was increased server activity which affected
performance.
Ignore GetRowsKeepIdleTime
Click this check box to hide all GetRowsKeepIdleTime method call information from the server analysis results.
This method call is run by the System Monitor to check for server related activity like uploading reports. Because
this method is called frequently by every client on your system, select this check box to prevent this active calls
from appearing in the log results.
Millisecond Threshold
Any BusinessObject.Method calls greater than the value entered in this field have their Exceeds Threshold
check boxes automatically selected. This value indicates these calls are over the threshold value (in milliseconds)
you defined. Use this value to perform Sort By or Group By actions to review longer duration method calls.
Results Sheet
The Results sheet displays all calls received by the application server during the run of the server log. You can use
Group By functionality to group the results by Object and/or Method to see the execution times for all method
calls.
Summary Sheet
The Summary sheet summarizes the results for each BusinessObject.Method, showing the number of calls and
the Total/Average/Least/Longest duration of those calls. These duration values are measured in milliseconds. You
37
can sort the results by column to determine the types of calls made most frequently against the server, or which
calls consume the most time.
Scenarios
This topic contains examples of how you can measure server performance through different methods.
Identification of server performance compared to other servers.
Use this method to identify performance differences for similar business object method calls on different servers.
You do this by comparing the elapsed times for the method calls on one server against another server. These
time differences may be related to CPU, database, customizations, operating systems, hardware configurations,
and other differences.
Analysis of activity by workstation.
If you group by on the Workstation column, you can identify the method calls made by specific workstations.
If a user reports slow performance, you can identify what this user was doing at the time and whether the elapsed
times for method calls run using expected elapsed times.
Analysis of system performance over time.
If you group by using the Object Name and Method Name columns, you can observe the performance of
specific method calls throughout the day. You may be able to discover slow periods by observing overall differences
in execution time patterns for those methods.
Analysis of most common activities and performance.
If you sort by the Frequency column on the Summary sheet, you can view the most frequent method calls and
their execution times. Sort by the Total Time column to see what activities consume the most time on the server.
Analysis of server verses network elapsed time for methods.
On the Results sheet, if you group by the WorkStation column, you can see the execution times of specific
method calls on that workstation. You should also be able to identify the same calls from the Client Diagnostics
for that workstation and compare the differences. For example, if you see an elapsed time of 800 milliseconds
for a server call and you see 1500 milliseconds for that same call on a client, you can conclude that about 700
milliseconds was spent transferring data over your network. If you do not like this network performance, you
can analyze network traffic using a network analysis tool.
Identification of customization issues.
You can analyze the elapsed time it takes to call specific methods compared to other server installations. This
may indicate issues with customizations, business process method (BPM) method/data directives, or C# code
customizations.
38
within Microsoft SQL Server Management Studio , and these templates display data specific for locking/blocking
and deadlocks.
To do this, click Options > Download Resources. You then download the following .zip files and set them up
to work with SQL Server Management Studio.
SQLProfilerTemplates.zip -- Download this file to evaluate locking and blocking on your system.
DeadlockAnalysis.zip -- Download this file to evaluate deadlocks.
39
40
41
In the above example, COMPANY/JoeS from the USERCOMP84 machine is accessing the Epicor ERP database.
The access currently given to USERCOMP84 is not recommended (this connection is the culprit), as it can cause
locking and blocking. In this case, Epicor would recommend that only the COMPANY/EPICADMIN login from the
APPSERVER machine have access to this data.
If the log records locking and blocking issues while users run custom or modified reports using an ODBC
connection, consider creating a table view for the report. Since database tables can become locked while the
report runs, the table view can prevent locking and blocking situations. To create a table view:
1. Open the report in a third party application like Microsoft SQL Server Report Builder, Excel, or another
ODBC source.
2. Create a view for each table that has ODBC access.
42
Deadlock Tracing
A deadlock occurs when two users or sessions have locks on separate business objects, and each business object
process tries to establish a lock on the business object in use by the other user/session.
SQL Server automatically detects and resolves deadlocks. If a deadlock occurs, one process ends (the victim),
while the other process runs (the culprit). The victim transaction is rolled back. While determining the victim is
relatively easy, what you need to discover is the culprit process that caused the deadlock. This culprit could be
another Epicor ERP application process running at nearly the same time or a third party application preventing
the victim process to initialize.
You can monitor when deadlocks occur using the SQL Server Deadlock Graph. While this graph runs, deadlock
events are recorded in the SQL Profiler trace log.
43
4. Click OK.
The DeadlockAnalysis.zip file is downloaded to your computer.
5. Unzip the DeadlockAnalysis.zip file in a directory location you need.
44
45
Load Distribution
Load Distribution
You can improve how the Epicor ERP application performs by assigning different tasks to specific application
servers that can handle the load. This distributes the load more evenly across your system resources.
If you have less than two hundred users, the Epicor ERP application can run efficiency through one server machine.
However if you have more than 200 users, use VMWare, or your organization requires special processing that
requires significant system resources, consider setting up task agent rules to more evenly distribute the load. For
example, you could assign a process that requires significant resources to run on a more powerful server, like
Material Requirements Planning, and then assign reports that generate less data to a server with fewer resources.
By distributing the load between these application servers, you reduce performance bottlenecks and match a
report/process with a machine best suited to run it.
The Hardware Sizing Guide can help you determine whether you should assign load to multiple application
servers. After you identify the hardware and usage scenario that best matches your organization, you can then
decide whether distributing the load will improve performance. To set up this functionality, you first create
application servers for each server machine available on your system.
Within the Epicor ERP application, you set up security groups and task agent rules. Security groups determine
which users have access to specific tasks. Task agent rules use security groups to determine which application
server will handle the processing for a specific task or group of tasks.
Tip If you do not have security groups set up yet, review the Security chapter in the Epicor Implementation
User Guide. This user guide is available on EPICWeb: https://epicweb.epicor.com/edu/user-guides
Application Servers
An application server manages how a specific instance of the Epicor application runs. Through each application
server, you can configure licenses, companies, sessions, and users for a specific database.
You can set up multiple application servers to run the same database and balance the load. For example, you
create two application servers for the same database, but these application servers are linked to different server
machines through their endpoint bindings. One application server is set up to run Epicor Web Access (EWA) on
one server machine, while another application server is set up to run a smart client through Net.TCP on a different
server machine. Likewise you could set up another application server that links to a machine which only handles
SSRS reporting tasks.
You add the application server that will interact with your server machine through the Epicor Administration
Console. This management tool is located on your server machine. For information on how to add application
servers, review application help in the Epicor Administration Console.
Task Agents
The task agent handles all scheduled tasks for an application server.
The task agent activates any program added to a recurring schedule. Users add programs to recurring schedules
through the Schedule drop-down lists available on programs throughout the Epicor application. You create these
schedules in the Epicor application using System Agent Maintenance, and you also use this program to create
task agent rules to distribute the load.
46
Load Distribution
Tip To learn how to assign tasks to automatic schedules, review the System Agent Maintenance topics in
the application help or review the Automatic Data Processing chapter in the Epicor Implementation User
Guide. This user guide is available on EPICWeb: https://epicweb.epicor.com/edu/user-guides
You create a task agent using the Task Agent Service Configuration program. You launch this program from
within the Epicor Administration Console. When you select the application server, the center pane on the Epicor
Administration Console displays the settings for the application server. Click the Task Agent Configuration button
to set up the task agent for the selected application server.
2. Click the Agent ID button to find and select the system agent. Only one system agent is available.
3. Click on the Actions menu and select the Edit Task Agent Rules... option.
The Task Agent Rules window appears.
4. Click the New button.
5. Select the Company for which this task agent rule will generate tasks.
Only companies assigned to the current user account display on this drop-down list. The task agent rule will
then handle processing for the selected company.
6. Optionally select the Security Group for the task agent rule.
Whenever a user assigned to the selected security group runs a report or process linked to this task agent
rule, the application server linked to this rule generates the system activity.
7. Next define the Rule Type option. This value indicates what tasks are handled by the task agent rule.
Available options:
Specific Task - Indicates this task agent rule will only run against a specific process. After you select this
rule type, you next select the process from the Process Id drop-down list.
All Tasks - Indicates all processes are run against this task agent rule. Any time a process is launched by
users within a company or a security group, this task agent rule handles the processing.
Specific Report - Indicates this task agent rule will only run against a specific report. After you select
this rule type, you next select the report from the Process Id drop-down list.
All Reports - Indicates all reports are run against this task agent rule. Any time a report is launched by
users within a company or a security group, this task agent rule handles the report generation.
8. If you select either the Specific Task or the Specific Report rule type, you next select the Process Id for the
item you want the task agent rule to run. Depending on the rule type, either reports or processes display
on this drop-down list.
47
Load Distribution
9. Enter the Appserver URL for the application server that will run the activity for this task agent rule. This
value links the task agent rule to the application server's location.
Tip You can find this value by opening the system configuration (.sysconfig) file for the client
installation. Locate the <appSettings><AppServerURL> node and copy this value.
10. Use the Endpoint Binding drop-down list to indicate how this application server checks for authentication
certificates through Internet Information Services (IIS). Select the Endpoint Binding defined on the application
server; the same options display on this drop-down list.
11. If you select either the UsernameWindowsChannel or UsernameSSLChannel endpoint binding option, enter
the User ID and Password for the account that can access this application server.
12. When you finish adding the task agent rule, click Save.
Continue to add the task agent rules you need. If you need to remove a task agent rule, highlight it on the grid
and click the Delete button.
Now the next time the system agent activates a schedule or a user launches a process or report, the tasks are
distributed to the application servers defined on the task agent rules.
48
Application Tuning
Application Tuning
This section contains some application tips and techniques you can do inside the Epicor application. You should
only use the tips and techniques that match your use of the Epicor application.
Application Troubleshooting
Be sure to disable all schedules before you move from the training database to your live database. If you do
not, two MRP processes run at the same time, slowing down performance.
Epicor recommends that you do not run MRP processing during the work day. Schedule MRP processing
during off hours.
Be sure that your Business Activity Queries (BAQs) are properly written. If a BAQ pulls unnecessary data or a
calculation is improperly constructed, each time this BAQ is run it slows performance.
Avoid creating several Business Process Management (BPM) procedures that generate email messages at the
same time (synchronously). If these procedures run frequently, performance is reduced. Instead, set up your
BPM procedures to run asynchronously.
Avoid activating too many change logs. Only run change logs for the specific fields you need. When too many
logs are running, they slow performance.
Performance slows when several large reports generate at the same time. To resolve this, move your report
generation processes to separate application servers. The previous Load Distribution section describes how
you set up task agent rules to distribute the load as needed.
Likewise, when multiple applications run at the same time, performance slows. Try to only use the programs
you need when you need them. Close them if they are no longer active.
Client Cache
The Client Cache is the local Disk Cache used by the Epicor client machine. As the Epicor client needs various
items stored on the server and in the database, these items are downloaded and stored in the local Disk Cache
for better performance.
The local client cache stores items like custom context menus, themes, processing calling xref files, customizations,
personalizations, list of business objects access for security retrieval, and so on. Storing these items in the client
cache typically improves client performance, but sometimes too many items can accumulate in this folder and
cause slow performance instead.
As a standard practice, users should periodically clean out their local client cache to remove rarely used items.
To clear the cache:
1.
2.
From the Main Menu, click Options > Clear Client Cache.
3.
49
Application Tuning
Now when you restart the Epicor application, only items currently needed to launch the application are loaded
into the client cache. The client machine should now have improved performance. However as time goes on,
more items will download to the client cache, and the user should run the Clear Client Cache option again.
Client Customizations
Use this technique to test the performance of a client customization against the base program version.
Make sure the user account currently logged into the application has Customization privileges. Then turn on
customization mode from the Menu by either clicking the Settings tile and selecting Developer Mode or move
your mouse pointer over the bottom of the screen and click the Developer Mode button on the toolbar.
50
Application Tuning
Navigate to the customized program and launch it. When the Select Customization window appears, select
the Base Only check box. Click OK and time how long it takes to launch the program. Close the program and
launch it again, selecting the customization. Time how long it takes the customized version of the program to
launch.
51
Application Tuning
records linked to inventory or release jobs are selected on a sales order detail line. To clear the check box, select
the option. Now when users create a new sales order, these records will not automatically be pulled into the
sales order, improving performance by eliminating this additional processing.
EDI Entry Process
You can customize Sales Order Entry to display an Entry Process field that improves the performance of sales
order processing by linking the order directly to EDI or Service Connect. This additional field does not display on
the default interface, but you can create a customization that displays this field on either the detail line or release
sheets. When users enter the EDI value in this field, it indicates that the Electronic Data Interchange processes
runs against the order detail lines or releases, improving how quickly detail lines and releases are generated for
this sales order.
To do this, create a new customization and navigate to either the Lines > Detail or Releases > Detail sheets.
Launch the Toolbox to add an EpiTextBox field to the form. Depending on which sheet you are on, you bind
this EpiTextBox field to either OrderDtl.EntryProcess or OrderRel.EntryProcess table columns. Save the
customization and make it available to users. When users create a sales order that is populated through EDI, they
enter "EDI" in this field. This improves how quickly EDI data is imported into the new sales order.
Tip If this is how your organization typically creates sales orders, you could also create a Business Process
Management (BPM) directive that automatically populates this field with the EDI value each time the user
creates a new detail line or release.
Be sure to test how long it takes to process sales orders both with or without the customization. If you notice
better performance, make this customization available to all the users in the company.
52
Application Tuning
53
Application Tuning
54
Application Tuning
5. For the BAQ Query Max Result Rows field, enter the highest number of rows that can be returned by a
business activity query.
By entering a value in this field, you restrict how many rows can be returned by each BAQ. This prevents
the query from pulling in an unlimited number of records, restricting situations where a runaway BAQ
consumes too many system resources to generate query results.
6. Now in the BAQ Query Timeout field, enter how many seconds can elapse before the application server
stops the query.
By entering a value in this field, you define how long each BAQ is allowed to run. When a query attempts
to generate results and reaches this time limit, the application server stops the query and sends the user a
time out message.
7. Click Apply and then OK.
8. The Server Manager dialog box displays, asking if you want to restart the application server. If this is a
good time to restart the server, click Yes.
55
Application Tuning
Now when this application server processes BAQs, the queries generate using these row and timeout limits.
56
Application Tuning
Sorting Performance
Sorting data by a selected column is a powerful feature, but be aware that some significant processing time may
be required to display the reordered results. This situation is especially true when you sort a large amount of
data. The query tool has to first return all of the records into memory before it can re-order their sequence through
the selected column.
All of this processing occurs on the server, so the data calls need to move across the network before they arrive
at your client workstation. So if you sort on a large amount of data, be patient the reordered results are on
their way.
Runaway BAQs
If you suspect a business activity query is causing poor performance, use the server logs and the Performance
and Diagnostic Tool to determine which BAQ is causing an issue. You do this by accessing the web.config file
and then setting the server log to Verbose. Run the process that launches the BAQ. When the BAQ completes
its run, open the server log in the Performance and Diagnostic tool. Navigate to the Server Diagnostics > Results
sheet. Group the results by Object Name and review the Execution Time values.
ABS(OLUTE)
ABS
abs
ASC(II)
ASC
ascii
INT(EGER)
INTEGER
convert
MAX(IMUM)
MAXIMUM
MIN(IMUM)
MINIMUM
57
Test Setup
Be sure to run all tests on the delivered base Training database.
You should not have any active Business Process Management (BPM) directives or customizations running against
the business objects you will test. If you receive significant variations in the results, this may indicate that BPM
directives or customizations are degrading the application performance.
These tests are intended primarily to measure server and database performance. To eliminate variations caused
by the network, run all of these all tests over a LAN (Local Area Network) with a ping time (32 bytes) to the
Application Server < 1 ms. If you test with a higher network latency, it may result in some measurable difference
in the observed results.
58
Test Procedure
The following test procedure provides you with a repeatable path that uses standard data delivered within the
Training database.
The results from the tests are captured in the client trace log. You can then analyze these results using the
Performance and Diagnostic Tool.
59
60
If you see more object calls than above and have much slower performance, it may because the Sales Order Entry
form properties are not cached. If this is the case, you may see additional Lib type calls as shown below:
61
62
13. Click on the Client Trace Analysis option on the Plug-Ins pane.
The Client Trace Analysis interface displays.
14. Either paste or browse to the Client File Trace Path to load in the client trace log you generated.
15. Click the Generate Diagnostics button to capture the performance results.
16. Click on the Summary tab and expand the GetByCustID Method Name.
Expected Results:
Observed time to move between customers: < 1 second
Example beakdown by business object method from the Summary sheet:
63
6. Verify that the first five column headers on the Lines > List sheet match the column headers in the table
below.
If not then rearrange the columns to match this sequence:
7. Create a spreadsheet that contains these twenty sales order detail lines (or copy and paste these lines from
this electronic document).
Line
Open
Job Type
Part
Manufactured
AV TA
12
Manufactured
CA10
10
Manufactured
CA12
12
Manufactured
CA20
20
Manufactured
DCD 300 OR
20
Manufactured
DCD 400 KB
100
Manufactured
600
Manufactured
DSS 1030
10
Manufactured
G1000
1,000
10
Manufactured
MRP100 SK
100
11
Manufactured
PC1
420
12
Manufactured
PC7
480
13
Manufactured
SD TA
79
14
Manufactured
4600 1
200
15
Manufactured
00P1
10
16
Manufactured
00P2
20
17
Manufactured
00M1a
100
18
Manufactured
00M3
30
19
Manufactured
0LM1
10
20
Manufactured
0LM2
20
8. Clear the Trace Log as described in the previous Activate the Trace Log section.
9. Copy the twenty detail line from your spreadsheet into your clipboard. Do not select the column headers.
10. Right click above the column headers in the Lines>List sheet; select Paste Insert from the context menu.
11. Wait while the twenty order detail lines are loaded into the Sales Order Entry form.
12. Write to the Trace Log as described in the previous Activate the Trace Log section. Note the Log File name;
this value uses the TraceDataxxx.log file format.
13. Launch the Performance and Diagnostic Tool.
64
Type
Part
Rev
Description
Our Quantity
UOM
6. Create a spreadsheet that contains these twenty purchase order detail lines (or copy and paste these lines
from this electronic document).
Line
Type Part
UOM
516X075B
10
EA
1032x075
20
EA
1032X100
20
EA
38X075B
10
EA
65
Line
Type Part
UOM
38X100B
10
EA
38X125B
10
EA
38X200
20
EA
516X075B
30
EA
516X100B
30
EA
10
516X150B
30
EA
11
516X200B
30
EA
12
84005 042
30
EA
13
8400S 053
EA
14
8400S 060
EA
15
8400S 116
EA
16
8400S 610
EA
17
516FW
EA
18
516LN
30
EA
19
10w30Lube
10
EA
20
8mm 1.25HNUT
10
EA
7. Clear the Trace Log as described in the previous Test Setup section.
8. Copy the twenty detail line from your spreadsheet into your clipboard. Do not select the column headers.
9. Right click above the column headers in the Inventory sheet; select Paste Insert from the context menu.
10. Wait while the twenty purchase order detail lines are loaded into the Purchase Order Entry form.
11. Write to the Trace Log as described in the previous Activate the Trace Log section. Note the Log File name;
this value uses the TraceDataxxx.log file format.
12. Launch the Performance and Diagnostic Tool.
13. Navigate to Client Diagnostics.
14. Browse to the Client File Trace Path to locate the client trace file you wrote as described previously.
15. Click the Generate Diagnostics button to capture the performance results.
Expected Results:
Total observed time for 20 lines: 51 Seconds
This value is measured from the Results tab as the difference between the Start Time of the first method and the
End Time of the last method.
66
67
Index
Index
A
B
baq best practices 56
baq performance tuning 54, 56
baqs, application server settings 54
baseline performance test 58
best practices, baq 56
blocking and locking 39
business activity queries, application server settings 54
business activity query 56, 57
business activity query performance tuning 56
C
clear results 19, 36
clear selected 37
client cache 49, 59
client cache, locate 50
client customizations 50
client diagnostic scenarios 21
client diagnostics 16
client logs, analyze 17
client logs, generate 16
client trace path 19
client tracing log options 20
common patterns 9, 10
configuration check 22
configuration check settings 23
configuration settings file 61
configure the sp_lock3.sql file 40
create sql job 40
custom format 19
customer retrieval 62
customer retrieval test 62
D
deadlock graph setup 43
deadlock graph, activate the sql 44
deadlock graph, stop 44
deadlock tracing 43
developer mode 50
F
fields client diagnostics 19
fields server diagnostics 36
frequent user complaints 9
G
generate application server logs 29
generate client logs 16
generate diagnostics 20, 37, 63, 65
getrowskeepidle 58
getrowskeepidletime chart 34, 37
H
hardware sizing guide 7
how it is organized 5
I
ignore getrowskeepidletime 37
intended audience 5
L
lan 58
load distribution 46
locking and blocking 39
locking and blocking, additional resources 43
locks log, review 42
locks log, start 42
log details, server 27
log file date format 20
log size, server 27
M
memory cached programs 50
millisecond threshold 37
mrp performance tuning 52
N
network diagnostics 25
68
network test 25
Index
O
open sales order entry form 59
order entry performance tuning 51
S
sales order detail line performance 63
sales order line entry test 63
san (i/o) test utility 13
send client logs to epicor 21
server diagnostic scenarios 38
server diagnostics 26
server log size and details 27
server logs, access 31
server logs, analyze 32
server logs, generate 29
sheets client diagnostics 19
sometimes everything is slow 9
sp_lock3.sql, configure the 40
U
unable to reproduce 10
V
vmware best practices guides 8
W
write permission 26
69