Anda di halaman 1dari 16

COST/BENEFIT ANALYSIS

with AHP
Using the Analytic Hierarchy Process (AHP)
To Identify Performance Scenarios for Enterprise Applications

Prepared by;
Sena Aksoy &
Cansu Bulat

Ergonomics,
03/12/2010
What is the AHP?
The Analytic Hierarchy Process (AHP) is a quantitative
technique for multiple criteria decision making. It
provides a convenient way to quantify the qualitative
aspects thus eliminating subjectivity in the outcome.
The technique begins by clearly identifying the
objectives, criteria and alternatives for a given problem
situation. For each of the criteria (qualitative or
quantitative) a vector is created that gives the relative
ranking of its alternatives. Weights of the criterias are
determined from the problem context. These inputs
are then used to relatively rank the alternatives.
Using AHP in performance modeling
• The success of any performance modeling exercise hinges on
verification of the target systems architecture using a set of
performance scenarios. In determining the performance scenarios,
it is required to rank the application workloads based on these
multiple criterions.

• These performance scenarios represent the applications workloads,


critical in terms of business and performance. One example of a
performance scenario for an on-line trading application could be a
login page, as it is frequently used, business critical and has
stringent performance objective of very low response time.
Likewise, based on the application context there could be several
other criterions to be considered such as the workload type,
number of external interfaces or resource demand. Furthermore
one criterion may be more important than the other.
Methodology for Identification of
Performance Scenario using AHP
This section depicts the methodology to determine the performance
scenarios using the AHP technique. The steps involved in the method are
as follows:

Step 1: Determine and Rank the Criterions for the Application Context

Several publications and books identify attributes of workloads which


influence its selection for performance modeling:
• Business Criticality (can be enunciated as user-type, revenue-impact, etc)
• Performance goals such as response time.
• Workload intensity (measured as hits/sec or number of concurrent users,
payload size)
• Resource Intensiveness (system utilization)
• Workload Type (Online/Batch/Messaging/Report Printing).
• External Interfaces
• Ease to build and test.
Methodology for Identification of
Performance Scenario using AHP
• AHP theory classifies these criterions as benefit criteria and cost criteria.
• Benefit criteria are those criterions where higher values are desired. In the
presented list, business criticality, workload intensity, number of external
interfaces and resource intensiveness are benefit criterions, as the
application workloads with higher values.
• Cost criterion have lower desired values. It is implicit that response time is
a cost criterion.
• Based on the application context, one criterion may be more important
than the other. A pair-wise comparison and the eigenvector of priorities is
used to rank the criterions for application context. Consider a simple case
of ranking benefit criterions for a camera:
Methodology for Identification of
Performance Scenario using AHP
The relative importance of one criterion over another
employs a pair-wise comparison. Pair-wise comparison
examines the relative importance of each option against
each of the other option in pairs. This can be represented
using a square matrix of options.

AHP technique uses eigenvectors to mathematically


determine the ranks of options from the pair-wise
matrix [Haas]. The eigenvector of priorities is calculated
by squaring the pair-wise comparison square matrix
and normalizing the row totals. The eigenvector of the
absolute ranks of benefit and cost criterions is calculated
separately from their corresponding pair-wise comparison
matrices.
Methodology for Identification of
Performance Scenario using AHP
Step 2: Rank the Application Scenarios for each criterion

Once the criteria values for scenarios are available, the next step is to compute the relative ranks
of the application workloads for each criterion. For quantitative criteria scenario ranks are
derived by normalizing their values, to make them comparable with other criteria values.

However, for qualitative criteria, AHP advocates pair-wise comparison of scenarios. If there are
"n" scenarios and "m" qualitative criterions then (n* (n-1)/2) * m user inputs would be
required. For an application with just 10 scenarios and 1 criterion, 45 inputs from users need
be taken. Since most applications consist of a large number of application scenarios and
multiple criteria, getting inputs from the user for pair-wise comparison may not be feasible.

Alternatively, AHP recommends the ratings approach. According to this, a scale of intensities (i.e.
low, medium, high) is established for qualitative criteria like business criticality or resource
intensiveness. Pair-wise comparison of these intensities is done to get the relative
importance of one over another and their relative ranks are computed. Then user rates the
scenarios on these scales of intensities for the corresponding criteria [EPA 2002]. These scale
intensities are then replaced by their priorities for further processing. Thus only n*m+ (i*(i-
1)/2)*m (where i is the number of scale intensities for m criteria) user inputs would be
required. For the case described above, the number of user inputs required for pair-wise
comparison would be 3, besides the scale ratings of scenarios for each criterion.
Methodology for Identification of
Performance Scenario using AHP
For the camera example discussed above, the ranking for style criterion, which is
qualitative, is computed as follows using the ratings approach.
Methodology for Identification of
Performance Scenario using AHP
Step 3: Compute overall ranking of Application Scenarios using cost-benefit analysis.
• In the first two steps, the scenario ranks for each criterion and criteria weights have
been computed. In step three, we will aggregate the scenario ranks for all cost and
benefit criterions separately. In the camera example, the overall ranks of the options is
obtained by taking the ratio of their benefit and cost ranks.

• To achieve this, the application scenarios


ranks for all benefit criteria, obtained in
step 2 are represented together in a matrix
form. This matrix is multiplied with the
eigenvector of benefit criteria created in
Step 1 to get vector of the total rank of scenarios based on all benefit criteria. Thus, if
there are "n" scenarios ranked for
"m" criteria, then a matrix of order (n * m) is formed and multiplied with (m*1) criteria
vector, consisting of ranks of these "m“ criterions to get a (n*1) scenario ranks vector.
Methodology for Identification of
Performance Scenario using AHP
Similarly, a vector of scenario ranks based on all cost criteria weights is
derived.

The final step is computation of an overall rank for scenarios by taking the ratio of
their benefit and cost ranks [Saaty 1994]. The scenarios with higher values of ratio
are selected.
Case Study
• In this section we present the re-engineering of a shipment
application. The existing application had performance problems and
needed to support a substantial increase in workload due to
business requirements. Performance modeling was initiated for the
new application architecture.

• As a result, a minimum set of application scenarios was required.


Therefore with the objective of determining critical and frequently
used workloads, a workload modeling study was conducted. The
study helped to baseline the arrival rate, arrival pattern, and
response time for the 200 odd application workloads. Table 1
shows a snap shot of the workload modeling results and the
projected workload metrics for the target system based on inputs
from the business stakeholders.
Case Study
Case Study
A mix of OLTP, batch, message and report printing
workloads was observed. It was decided to take
performance scenarios of each workload type for
modeling. Here the application of AHP method
For identifying OLTP performance scenario is
discussed. Table 2 shows criterions, eigenvector
values and normalized ranks (step 1.2) determined
from the pair-wise comparison (step 1.1).

While the business criticality, throughput and


resource intensiveness were considered benefit
criterions, response time was regarded a cost
criterion. Following the procedure shown in step 2,
ranking of scenarios for the qualitative criterion
(business criticality and resource intensiveness)
was determined (using the ratings approach).
A scale of low, medium and high intensities was established and pair-wise comparison (step 2.1) of
these scales was used to get eigenvector values (step 2.2) and normalized ranks (Table 3).
Case Study
For quantitative criteria (response time, throughput) normalized ranks of the scenarios was
obtained. From the procedure described in step 3, the weighted benefit and cost rank of
each scenario was calculated by matrix multiplication. Subsequently, the overall ranks of
scenarios was obtained by taking the ratio of their benefit and cost ranks (step 3.3). Table 4
shows these benefit/cost ranks in terms of percentage values.
Case Study
From the above analysis it was determined that the ratio of 9.42 in scenario 4 indicates
the most representative performance scenario for the OLTP workload type.

Additional scenarios from other workloads (batch, message and report printing, etc)
can be picked having high values of benefit/cost ratio. Using the same process, the
different workload types resulted in selection of the performance scenarios as shown
in table 5.
Conclusion
The AHP methodology described in this article can facilitate
and accelerate the selection of the most valuable
performance scenarios. This can focus performance modeling
on a key few performance critical scenarios, rather than
evaluating hundreds of applications scenarios. Some critical
factors for choosing performance scenarios have been
suggested. However, more criteria can be taken into account
based on system, user and business requirements. By
following the outline of the steps described, selection of
performance scenarios can be made more objectively, based
on scientific method of AHP.

Anda mungkin juga menyukai