DEPLOYING AN AUTOMATED MODELING PROCESS

FOR SALES TARGETING

A Case Study for AT&T Advertising & Publishing

Gene Edmiston
Senior Manager, Marketing Analysis and Reporting
AT&T Advertising & Publishing

Don Kridel
Associate Professor of Economics
University of Missouri-St. Louis

ABSTRACT

A web-based service that uses advanced dynamic model building

techniques to conduct intelligent profiling and modeling is utilized for sales
targeting within AT&T Advertising and Publishing. This automated
modeling approach is designed to cost-effectively assist businesses in their
targeting activities. In particular, many companies with a direct sales force
require a timely methodology that can be used to more effectively target
sales force activity. In many cases, the relative paucity of internal data
makes a “purely internal” modeling effort ineffective. In the current paper,
we investigate alternative approaches to acquisition modeling for sales
targeting; the output of the models will be ranked lists that will be used for
sales force targeting. The process described involves a wide-variety of
alternative model specifications to be considered. The results in this case
indicate that the automated modeling approach outperforms more
traditional approaches by a substantial margin—both in terms of time and
cost.

Parts of this paper were presented at DMA*07, Chicago, IL. Oct. 17, 2007.

DEPLOYING AN AUTOMATED MODELING PROCESS

 FOR SALES TARGETING
A Case Study for AT&T Advertising & Publishing1

INTRODUCTION
Kridel and Dolk [7] describe automated dynamic model-building techniques
derived from research in the areas of active decision support and automatic
model generation ([3], [4], [5]) to generate context-specific customer lists
for B2C small and medium businesses (SMBs).2 Here we investigate a
similar approach for a much larger B2B company (AT&T Advertising &
Publishing, henceforth AT&T-AP). In this approach, the general objective
is to automate parts of the econometric modeling process. Specifically, the
automated process determines the appropriate independent variables, their
associated functional form, and uses the final model for scoring (the scores
are then used to provide a smart or targeted list). Heuristics combined with
data analysis and statistical testing are used to generate a knowledge base
that is used to evaluate “candidate models”; the process iteratively refines
the estimated equations until the process converges to a final model; from
the estimated model a prospect list is generated and provided to the sales
force. In this paper, we seek to describe the process in some detail.3
In particular, we discuss:
• Modeling scope, e.g., “national”, regional, or local models;
• Independent variable “scope”, e.g., the inclusion of internal (AT&T-
AP), external (national database) variables or both;
• Project cost (in $ and time) of an automated vs. a manual process;
• Results, i.e., calculating performance metrics (lift and ROI) that
display overall effectiveness of the project.

INTRODUCTION
1 When the work began, AT&T Advertising & Publishing (AT&T-AP) was still Southwestern
Bell Directory Operations (SBCDO). SBCDO had three regions: West (CA), Southwest (TX,
AR, KS, MO, OK) and Midwest (MI, OH, WI, MI). Recently, AT&T has acquired Bell South
and the regions have been re-aligned. No Bell South directories are included in this analysis.
Bell South books were added to the modeling project in 2008.
2 Kridel and Dolk [8] describe a case study for an SMB using automating modeling for a small
scale direct mail (DM) campaign. In this case, modeled prospects outperform ad hoc targeting
( a list based on “simple selects” derived from a customer profile) by almost six to one (7.9%
response rate vs. 1.4% response rate).
3 Initial modeling and analysis was performed in the Fall of 2004. The benchmark analyses
were performed for selected markets during the 2005 and 2006 contract periods.
Yellow Pages have been a resource for shoppers for nearly 130 years and
today spans multiple platforms - print, Internet and mobile devices.
Currently there more than 200 directory publishers, both telephone
company-affiliated and independents; these publishers issue more than
7,000 Yellow Pages directories in the U.S. and Canada. Most major cities
have multiple directories that directly compete. Like other media, the
proliferation of advertising choices has strained the economics
performance of most Yellow Page Publishers. Historically, Yellow Pages
companies were associated with the local telephone companies. In recent 4

years, this association has changed as a result of

• spin-offs (e.g., RHD/DEX and Idearc),
• increased head-to-head entry (Verizon providing Yellow Pages in
AT&T service areas, e.g., St Louis), and
• competitors entering various markets (e.g., Yellow Book).

AT&T-AP is the largest directory publisher in terms of revenue. AT&T’s

core print product is under pressure as consumer usage of the Yellow
Pages has begun to shift to the other media (in particular, the Internet).
With the increased penetration of broadband access, the Internet provides
a richer medium for information search than traditional print search. In
particular, web-based search provides
• The ability to search using a wider range of attributes;
• The ability to drill down on specific topics for more detailed
information and cross-comparisons ;
• Supports direct transactional capabilities.

4 Indeed, the initial AT&T Consent Degree had Yellow Pages going to AT&T (with the other
“competitive” services like long-distance and equipment) but it has shifted (along with
intraLATA toll) back to the Bell Operating Companies (BOCs) as there was widespread fear
that the BOCs were not sustainable as “local-only” companies.
This trend has significantly impacted Yellow Pages revenue and retention.
Thus ATT-AP will face increasing challenges as usage and circulation
declines and/or flatten. Historically, the AT&T Sales Force has focused on
increasing the spending of current customers. As penetration rates have
fallen, more attention must be directed at non-customers (OPs).

As in many businesses with a large direct sales force, the issue of

providing good leads to that sales force is important. Without a targeted
list, the sales force has to either develop its own ad hoc targets or dilute
itself by attempting to contact all (too many) prospects.5 While good
salesmen generally have effective informal targeting methods, these do not
perform as well as formal targeting models.

The difficulty with formal modeling is that it is typically expensive--both in

terms of $ and time. AT&T-AP is interested in targeting OPs or “old
prospects” (businesses listed in the Yellow Pages that do not purchase any
advertising); there are several million OPs throughout the AT&T-AP
territory spread over several hundred markets (or books).

Effective targeting of OPs is essential since these prospects outnumber

paying customers by a significant factor (between five and ten to one). As
a result, the sales force effort is a “binding constraint”; better targeting will
allow better utilization of sales force time and effort. Finally, as AT&T
expands its business lines from an advertising perspective (e.g., search
engine marketing and IYP--Internet Yellow Pages), there is yet another
drain on marketing and sales resources.

MODELING ANALYSIS
Since there is was a dearth of in-house modeling resources, the original
plan was for the project to follow other analysis tasks at AT&T-AP. That is
to say, develop models at the region (or company level) utilizing an outside
consultant to supplement in-house expertise.

5 The internal “folklore” was that all OPs were contacted; as the project unfolded, it became
clear that the sales offices were utilizing ad hoc targeting based on heading and length of time
as an OP.
During the evaluation phase, three primary questions were to be
addressed:

(1) Would modeling improve upon the existing ad hoc targeting

process; and, if so, by how much?

(2) What independent variables (and associated data cost) would be

required to effectively build targeting models, i.e., would effective
models require the use of:
a. internal (AT&T) data;6
b. external (national database) “customer” data; 7
c. or both.

(3) What modeling scope would be most effective, e.g., (one

company-wide model, three regional (Southwest, West,
Midwest) models, or “local” models (one for each directory)?

Ten trial markets were chosen for this analysis.8 These markets were
analyzed and alternative model specifications were tested. Models were
developed utilizing only internal (AT&T-AP) data as explanatory variables,
only external (Experian national B2B) data as explanatory variables, and
both internal and external data as explanatory variables.

6 Internal AT&T data available were: account age, whether the OP had ever advertised before
(and when), measure of “risk” (bad debt or delinquent), channel (premise, t-sales, other), and
headings information. Availability differed slightly by AT&T region.
7 The Experian NBD was utilized for the analysis; virtually the entire set of variables was
analyzed. Variables of particularly interest were number of employees, sales, SIC code, age of
business, Oxxford life-cycle, etc.
8 These were large (annual revenue > $5M) directories spread across the three AT&T AP
regions.
A representative model (internal & external model) is presented in Table 1.

Table 1: Representative Internal & External Model

T-
Categ Transfo Coeffici Statisti
Variable ory rm ent Mean c
Number of
Employees X -0.00111 14.74 -1.62
Number of 13065.
Employees n/a X2 0.00000 52 1.37
Year in Business
Code C 0.48052 0.06 6.16
Primary 2-digit SIC 50 -0.14676 0.05 -1.20
Primary 2-digit SIC 81 0.76671 0.02 6.43
Primary 2-digit SIC 80 0.63941 0.05 7.60
Primary 2-digit SIC 7 0.86068 0.02 6.99
Primary 2-digit SIC 65 -0.32440 0.04 -2.22
Primary 2-digit SIC 64 0.36049 0.02 2.49
Primary 2-digit SIC 17 0.74982 0.04 8.99
Primary 2-digit SIC 76 0.74604 0.01 5.24
Primary 2-digit SIC 75 0.47330 0.03 4.18
Primary 2-digit SIC 42 0.28364 0.01 1.48
Primary 2-digit SIC 59 0.34603 0.06 4.17
Primary 2-digit SIC 87 -0.68559 0.06 -4.68
Primary 2-digit SIC 55 0.42118 0.02 3.21
Primary 2-digit SIC 86 -0.52404 0.06 -3.80
Primary 2-digit SIC 52 0.58078 0.01 3.86
Primary 2-digit SIC 82 0.50863 0.01 2.79
Legal Business
Structure S 0.38866 0.01 2.50
Prospect Age LOG -2.85276 0.87 -48.71
Constant -2.18715 1.00 -55.12
Figure 1 displays the lift-chart results (by decile, for one of the ten initial test
markets) for each of the three modeling alternatives (scopes).

As can be seen in the figure, “internal-data only” models did not perform
any better than did random assignment (there is very slight lift in the first
three deciles).

“External-data only” models performed considerably better—offering

sizable lift in the first two deciles.

The combined (internal-data and external-data) model performed the best.

The lift is considerably larger in the first 4 deciles than are the alternatives.

Using the results displayed above, it became clear that modeling “would
pay” (assuming models could be developed at a reasonable cost) and that
the use of external data was critical to the success of the project. Indeed,
the estimated ROI for “internal data” only models was slightly negative.
It became clear very early that a company model was not feasible as there
were too few common variables across the regions. Further, it quickly
became apparent that local market conditions were more important than
had been initially suspected. Using an “aggregated” or “regional” model
from the ten test markets produced results that were approximately 10-15%
“worse” (than using the local models).9

Extrapolating to a wider roll-out and more “aggregate” regional models

(where there would be 30 to 35 markets per region), it was clear that local
models would significantly out-perform regional models. Clearly, given
internal resources and limited budgets for external consulting, alternative
methods were required. In this case, AT&T evaluated a few different
providers of automated services.10

THE AUTOMATED PROCESS

Kridel and Dolk [7] describes the data model underlying automated process
at a relatively high-level. To summarize here, the process begins with
customer list from AT&T-AP. The list is then matched (and geo-coded) to a
national database to obtain firmographics. In this case, the prospects (non-
customers) are also provided by AT&T-AP.

The modeling process (driven by the knowledge base) develops a logistic

regression model which is used for scoring. The ranked prospect list, along
with profiles and maps, are available for downloading and/or on-line
viewing and analysis.

9 The resulting models were quite different by market (book). In particular, the SIC codes of
importance were quite different across books.
10 AT&T-AP selected CopperKey (KAST) as its vendor of choice in December 2004.
 EVALUATION

Since early 2005, selected books have been tracked and analyzed. A
sample of books for 2005 and 2006 are displayed in Figure 2 and 3 below.
While there is some variability across books and years, the results are
similar in the sense that response rates have increased dramatically.

Figure 3 displays lift for selected books that were tracked in the first half of
2005.11

Response rates increased on average approximately 285%. Typically, this

meant OP conversion rates increasing from the 0.25% - 0.5% range to the
0.75% - 1.5% range.

11 Lift is calculated as: (Model Response – Base Response) / Base Response. Generally, the
top-four deciles were used for the targeted lists; non-matched or random selections were used
for base response rates.
Figure 4 displays lift for selected books that closed in the first quarter of
2006; results are summarized to the region.

As can clearly be seen, the tested markets from Regions One and Three
outperform the Region Two test markets (which nonetheless exhibited lift of
slightly over 100%).12

12 There are a variety of reasons that suggest why the Region 2 performed more poorly. In
particular, the way prospects/customers are assigned differs by region. In addition, Region 2
has more aggressively used discounting strategies to increase revenue among existing
advertisers—internal studies suggest that the resulting “over-development of large ads” causes
a “barrier-to-entry” which leads to “slower” conversion of OPs.
Figure 5 displays the impact on response rates for the same selected
books that closed in the first quarter of 2006.

The business-as-usual response rates varied slightly by region (more by

book-size) and increase in response rates varied considerably by region
(and less-so by book size).
Figure 6 compares the actual response rates to the predicted response
rates (from the model) by decile for a selected book in 2006.

Using the same nine books that are summarized in Figures 4 and 5, we
also note that average spending (per converted OP) has increased
approximately 50%. Further, using only the direct cost of the vendor’s
contract, ROI is calculated to be approximately 3000%.13

Of the increased response, the authors estimate that slightly less than one-
half of the improved performance is due to local or “book-specific” models
(as opposed to more aggregate region-specific models).

This suggests that using local models (via automated analytics)

significantly outperforms regional models (built “manually”) with an
estimated ROI of slightly over 3000%.

13 The average cost per local model (book) for the contract is slightly above $2000; incremental
revenue is calculated to be approximately $600,000 for the nine books summarized in Figures
4 and 5. In other words, incremental revenue in only these nine books that were tracked
exceeds the cost of the total contract by approximately a factor of 3.
 CONCLUSIONS

There is no surprise that targeting is shown to be effective in increasing

response rates. Targeting through automated analytics is shown (in this
case) to outperform more traditional methods. This improved performance
relates primarily to the ability to model at a much-lower geographic level.

The automation of the analytical process outlined in this paper is not a

perfect substitute for full-time modeling but it does significantly reduce the
amount of time required from a full-time analyst to target market effectively.
Because of this internal analyst can devote much more time to work on
higher priority (and/or less standard) projects that require less standardized
analytical methods and/or process, as well as higher revenue impacting
models or analyses that require more time and attention.
 REFERENCES

[1] Ben-Akiva, M. and S. Lerman (1985), Discrete Choice Analysis, MIT Press.

[2] Bhargava, H. K., R. Krishnan, and R. Muller, "Decision Support on Demand:

Emerging Electronic Markets for Decision Technologies" Decision Support Systems,
Vol. 19, pp. 193-214, 1997.

[3] Castillo, D.G., D.R. Dolk, and D.J. Kridel, “GOST: An Active Modeling System for
Costing and Planning NASA Space Programs”, Journal of Management Information
Systems, Winter 1991-92, Vol. 8, No. 3, pp. 151-169.

[4] Dolk, D.R., and D.J. Kridel, “An Active Decision Support System for Econometrics”,
Decision Support Systems, 7, 1991, pp. 315-328.

[5] Dolk, D.R., and D.J. Kridel, “Towards a Symbiotic Expert System for Econometric
Modeling”, Current Research in Decision Support Technology, edited by R.W. Blanning
and D.R. King, IEEE Computer Society Press, 1993, Chapter 7.

[6] Gupta, S. and P. Chintagunta, "On Using Demographic Variables to Determine

Segment Membership in Logit Mixture Models," Journal of Marketing Research, 31,
1994, pp. 128-136.

[7] Kridel, D.J. and D.R. Dolk, “An On-line Marketing Consultant for Small and Medium
Businesses”, Proceedings of the 32nd WDSI Conference, Lihue, HI, April 2003.

[8] Kridel, D.J. and D.R. Dolk, “Using Intelligent Profiling to Generate Smart Lists: An
Empirical Test”, Proceedings of the 33rd WDSI Conference, Manzanillo, Mexico, April
2004.

[9] Rossi, P., R. McCulloch, and G. Allenby, “The Value of Purchase History Data in
Target Marketing”, Marketing Science, Vol. 15, No. 4, 1996, pp. 321-340.

[10] Shaffer, G. and Z. Zhang, “Competitive Coupon Targeting”, Marketing Science,

Vol. 14, No. 4, 1995, pp. 395-416.

[11] Spohrer, J, P. Maglio, J. Bailey, and D. Gruhl, “Towards A Science of Service

Systems”, Computer, Jan. 2007.

[12] Spohrer, J, S. Vargo, N. Caswell, and P. Maglio, “The Service System is the Basic
Abstraction for Service Science”, Proceedings of the 41st Hawaiian International
Conference on System Sciences, Jan. 2008, pp. 1-10.

[13] Vargo, S. L. and Lusch, R. F. Evolving to a new dominant logic for marketing.
Journal of Marketing, 68, 1, 2004, 1-17.