Douglas C. Schmidt
schmidt@cs.wustl.edu
Department of Computer Science
Washington University, St. Louis, MO 63130
This paper appeared in the Handbook of Programming from network and host failures, minimizing the impact of com-
Languages, Volume I, edited by Peter Salus, MacMillan Com- munication latency, and determining an optimal partitioning of
puter Publishing, 1997. application service components and workload onto processing
elements throughout a network.
The accidental complexities associated with communica-
1 Introduction tion software stem from limitations with conventional tools
and techniques. For instance, low-level network programming
Communication software for next-generation distributed ap- interfaces like Sockets are tedious and error-prone. Likewise,
plications must be flexible and efficient. Flexibility is needed higher-level distributed computing middleware like CORBA,
to support a growing range of multimedia datatypes, traf- DCOM, and Java RMI lack key features, such as asynchronous
fic patterns, and end-to-end quality of service (QoS) require- I/O and end-to-end QoS guarantees. Moreover, conventional
ments. Efficiency is needed to provide low latency to delay- higher-level middleware implementations are not yet opti-
sensitive applications (such as avionics and call processing) mized for applications with stringent performance require-
and high performance to bandwidth-intensive applications ments [2, 3].
(such as medical imaging and teleconferencing) over high- Another source of accidental complexity arises from the
speed and mobile networks. widespread use of algorithmic design [4] to develop commu-
This paper outlines the key sources of complexity for com- nication software. Although graphical user-interfaces (GUIs)
munication software and describes how patterns and frame- are largely built using object-oriented (OO) design, commu-
works can alleviate much of this complexity. To focus the nication software has traditionally been developed with al-
discussion, the paper explains how patterns and frameworks gorithmic design. However, algorithmic design yields non-
have been applied to develop high-performance, concurrent extensible software architectures that cannot be customized
Web servers. rapidly to meet changing application requirements. In an era
of deregulation and stiff global competition, it is prohibitively
expensive and time consuming to repeatedly develop applica-
1.1 Sources of Complexity for Communication tions from scratch using algorithmic design techniques.
Software
Despite dramatic increases in computing power and network 1.2 Alleviating the Complexity of Communica-
bandwidth, however, the cost of developing communication tion Software with OO Frameworks and
software remains high and the quality remains relatively low. Patterns
Across the industry, this situation has produced a “communi-
cation software crisis,” where computing hardware and net- OO techniques provide principles, methods, and tools that sig-
works get smaller, faster, and cheaper; yet communication nificantly reduce the complexity and cost of developing com-
software gets larger, slower, and more expensive to develop munication software [5]. The primary benefits of OO stem
and maintain. from its emphasis on modularity, reusability, and extensibil-
The challenges of communication software arise from in- ity. Modularity encapsulates volatile implementation details
herent and accidental complexities [1]. Inherent complexities behind stable interfaces. Reusability and extensibility enhance
stem from fundamental challenges of developing communica- software by factoring out common object structures and func-
tion software. Chief among these are detecting and recovering tionality. This paper illustrates how to produce flexible and ef-
1
ficient communication software using OO application frame- develop complex communication middleware and applications
works and design patterns. [12]. The following are common pitfalls associated with the
A framework is a reusable, “semi-complete” application use of native OS APIs:
that can be specialized to produce custom applications [6].
Excessive low-level details: Developers must have intimate
A pattern represents a recurring solution to a software devel-
knowledge of low-level OS details. For instance, develop-
opment problem within a particular context [7]. Patterns and
ers must carefully track which error codes are returned by
frameworks can be applied together synergistically to improve
each system call and handle these OS-specific problems in
the quality of communication software by capturing success-
their applications. These details divert attention from the
ful software development strategies. Patterns capture abstract
broader, more strategic application-related semantics and pro-
designs and software architectures in a systematic format that
gram structure.
can be readily comprehended by developers. Frameworks cap-
ture concrete designs, algorithms, and implementations in par- Continuous re-discovery and re-invention of incompatible
ticular programming languages. higher-level programming abstractions: A common rem-
The examples in the paper focus on developing high- edy for the excessive level of detail with OS APIs is to de-
performance concurrent Web servers using the ACE frame- fine higher-level programming abstractions. For instance, a
work [8]. ACE is an OO framework that provides components Reactor [13] is a useful component for demultiplexing I/O
that implement core concurrency and distribution patterns [9] events and dispatching their associated event handlers. How-
related to the domain of communication software. The frame- ever, these abstractions are often re-discovered and re-invented
work and patterns in this paper are representative of solutions in an ad hoc manner by each developer or project. This pro-
that have been successfully applied to communication systems cess hampers productivity and creates incompatible compo-
ranging from telecommunication system management [9] to nents that cannot be reused readily within and across projects
enterprise medical imaging [10] and real-time avionics [11]. in large software organizations.
This paper is organized as follows: Section 2 presents an
High potential for errors: Programming to low-level OS
overview of patterns and frameworks and motivates the need
APIs is tedious and error-prone due to their lack of typesafety.
for the type of communication software framework provided
For example, many networking applications are programmed
by ACE; Section 3 outlines the structure of the ACE frame-
with the Socket API [14]. However, endpoints of communi-
work; Section 4 illustrates how patterns and components in
cation in the Socket API are represented as untyped handles.
ACE can be applied to develop high-performance Web servers;
This increases the potential for subtle programming mistakes
and Section 5 presents concluding remarks.
and run-time errors [15].
Lack of portability: Low-level OS APIs are notoriously
2 Applying Patterns and Frameworks non-portable, even across releases of the same OS. For in-
stance, implementations of the Socket API on Win32 plat-
to Communication Software forms (WinSock) are subtly different than on UNIX platforms.
Moreover, even WinSock on different versions of Windows
2.1 Common Pitfalls of Developing Communi- NT possesses incompatible bugs that cause sporadic failures
cation Software when performing non-blocking connections and when shut-
2.1.1 Limitations of Low-level Native OS APIs ting down processes.
Steep learning curve: Due to the excessive level of detail,
Developers of communication software confront recurring
the effort required to master OS-level APIs can be very high.
challenges that are largely independent of specific application
For instance, it is hard to learn how to program the thread can-
requirements. For instance, applications like network file sys-
cellation mechanism correctly in POSIX Pthreads. It is even
tems, email gateways, object request brokers, and Web servers
harder to learn how to write a portable application using thread
all perform tasks related to connection establishment, service
cancellation mechanisms since they differ widely across OS
initialization, event demultiplexing, event handler dispatch-
platforms.
ing, interprocess communication, shared memory manage-
ment, static and dynamic component configuration, concur- Inability to handle increasing complexity: OS APIs define
rency, synchronization, and persistence. Traditionally, these basic interfaces to mechanisms like process and thread man-
tasks have been implemented in an ad hoc manner using low- agement, interprocess communication, file systems, and mem-
level native OS application programming interfaces (APIs), ory management. However, these basic interfaces do not scale
such as the Win32 or POSIX, which are written in C. up gracefully as applications grow in size and complexity. For
Unfortunately, native OS APIs are not an effective way to instance, a Windows NT process only allows 64 thread local
2
storage (TLS) keys. This number is inadequate for large-scale and frameworks help alleviate the continual re-discovery and
server applications that utilize many DLLs and thread local re-invention of communication software concepts and compo-
objects. nents by capturing solutions to standard communication soft-
ware development problems [7].
2.1.2 Limitations of Higher-level Distributed Object
Computing Middleware 2.2.1 The Benefits of Patterns
It is possible to alleviate some of the pitfalls with native
Patterns are particularly useful for documenting the structure
OS APIs by using higher-level distributed computing middle-
and participants in common micro-architectures for concur-
ware. Common examples of higher-level distributed comput-
rency and communication such as Reactors [13], Active Ob-
ing middleware include CORBA [16], DCOM [17], and Java
jects [23], and Brokers [24]. These patterns are generaliza-
RMI [18]. Higher-level distributed computing middleware re-
tions of object-structures that have proven useful to build flex-
sides between clients and servers and eliminates many tedious,
ible and efficient event-driven and concurrent communication
error-prone, and non-portable aspects of developing and main-
software frameworks and applications.
taining distributed applications by automating common net-
work programming tasks such as object location, object acti- Traditionally, communication software patterns have either
vation, parameter marshaling, fault recovery, and security. been locked in the heads of the expert developers or buried
However, higher-level distributed computing middleware is deep within the source code. Allowing this valuable informa-
often only a partial solution, for the following reasons: tion to reside only in these locations is risky and expensive,
however. For instance, the insights of experienced designers
Lack of portability: Conventional higher-level middleware will be lost over time if they are not documented. Likewise,
is not widely portable. For instance, the Object Adapter com- substantial effort may be necessary to reverse engineer pat-
ponent in the CORBA 2.0 specification is woefully under- terns from existing source code. Therefore, explicitly captur-
specified [19]. Therefore, servers written in CORBA are not ing and documenting communication software patterns is es-
portable among ORB products from different vendors. Like- sential to preserve design information for developers who en-
wise, DCOM is targeted for Win32 platforms and Java RMI is hance and maintain existing software. Moreover, knowledge
targeted for applications written in Java. of domain-specific patterns helps guide the design decisions
Lack of features: Conventional higher-level middleware fo- of developers who are building new applications.
cuses primarily on communication. Therefore, it does not
cover other key issues associated with developing distributed
2.2.2 The Benefits of Frameworks
applications. For instance, conventional higher-level middle-
ware does not specify important aspects of high-performance Although knowledge of patterns helps to reduce development
and real-time distributed server development such as shared effort and maintenance costs, reuse of patterns alone is not suf-
memory, asynchronous I/O, multi-threading, and synchroniza- ficient to create flexible and efficient communication software.
tion [20]. While patterns enable reuse of abstract design and architec-
Lack of performance: Conventional higher-level middle- ture knowledge, abstractions documented as patterns do not
ware incurs significant throughput and latency overhead [2, directly yield reusable code. Therefore, it is essential to aug-
21]. These overheads stem from excessive data copying, ment the study of patterns with the creation and use of appli-
non-optimized presentation layer conversions, internal mes- cation frameworks. Frameworks help developers avoid costly
sage buffering strategies that produce non-uniform behavior re-invention of standard communication software components
for different message sizes, inefficient demultiplexing algo- by implementing common design patterns and factoring out
rithms, long chains of intra-ORB virtual method calls, and lack common implementation roles.
of integration with underlying real-time OS and network QoS
mechanisms [22]. 2.2.3 Relationship Between Frameworks and Other
Reuse Techniques
2.2 Overcoming Communication Software Pit-
Frameworks provide reusable software components for appli-
falls with Patterns and Frameworks
cations by integrating sets of abstract classes and defining stan-
Successful developers and software organizations overcome dard ways that instances of these classes collaborate [25]. The
the pitfalls described above by identifying the patterns that resulting application skeletons can be customized by inherit-
underly proven solutions and by reifying these patterns in ing and instantiating from reuseable components in the frame-
object-oriented application frameworks. Together, patterns works.
3
The scope of reuse in a framework can be significantly perform their processing by borrowing threads of control from
larger than using traditional function libraries or conventional self-directed application objects. This is illustrated in Fig-
OO class libraries. The increased level of reuse stem from the ure 1 (A), where the application-specific logic manages the
fact that frameworks are tightly integrated with key communi- event loop. In contrast, frameworks are active, i.e., they man-
cation software tasks such as service initialization, error han- age the flow of control within an application via event dis-
dling, flow control, event processing, and concurrency control. patching patterns like Reactor [13] and Observer [7]. The
In general, frameworks enhance class libraries in the fol- callback-driven run-time architecture of a framework is shown
lowing ways: in Figure 1 (B). This “inversion of control” is referred to as The
Hollywood Principle [26], i.e., “don’t call us, we’ll call you.”
Frameworks define “semi-complete” applications that em-
body domain-specific object structures and functionality: In practice, frameworks and class libraries are complemen-
Class libraries provide a relatively small granularity of reuse. tary technologies. Frameworks often utilize class libraries in-
For instance, the classes in Figure 1 (A) are typically low- ternally to simplify the development of the framework. For
level, relatively independent, and general components like instance, portions of ACE use the string and vector contain-
Strings, complex numbers, arrays, and bitsets. In contrast, ers provided by the C++ Standard Template Library [27] to
manage connection maps and other search structures. In ad-
dition, application-specific callbacks invoked by framework
NETWORKING event handlers frequently use class library components to per-
APPLICATION
SPECIFIC
form basic tasks such as string processing, file management,
LOGIC MATH and numerical analysis.
INVOKES ADTS
To illustrate how OO patterns and frameworks are being
USER
successfully applied to communication software, the remain-
EVENT
DATA der of this paper examines the structure and use of the ACE
LOOP INTERFACE
BASE
framework [8].
4
This section outlines the structure and functionality of the most versions of UNIX (e.g., SunOS 4.x and 5.x; SGI IRIX
ACE framework. Section 4 illustrates how components and 5.x and 6.x; HP-UX 9.x, 10.x, and 11.x; DEC UNIX, AIX 4.x,
patterns in ACE can be applied to build high-performance, DG/UX, Linux, SCO, UnixWare, NetBSD, and FreeBSD),
concurrent Web servers. real-time operating systems (e.g., VxWorks, Chorus, LynxOS,
and pSoS), and MVS OpenEdition. Due to the abstraction pro-
3.1 The Structure and Functionality of ACE vided by ACE’s OS adaptation layer, a single source tree is
used for all these platforms.
ACE is a relatively large framework, containing over 135,000
lines of C++ code divided into 450 classes. To separate con- 3.1.2 The ACE C++ Wrapper Layer
cerns and to reduce the complexity of the framework, ACE is
designed using a layered architecture. Figure 2 illustrates the It is possible to program highly portable C++ applications di-
relationships between ACE components. rectly atop ACE’s OS adaptation layer. However, most ACE
The lower layers of ACE contain an OS adapter and C++ developers use the C++ wrappers layer shown in Figure 2.
wrappers that portably encapsulate core OS communication The ACE C++ wrappers simplify the development of applica-
and concurrency services. The higher layers of ACE ex- tions by encapsulating and enhancing the native OS concur-
tend the C++ wrappers to provide reusable frameworks, self- rency, communication, memory management, event demulti-
contained distributed service components, and higher-level plexing, dynamic linking, and file system APIs with typesafe
distributed computing middleware components. Together, C++ interfaces.
these layers simplify the creation, composition, and configu- The use of C++ alleviates the need for developers to pro-
ration of communication systems. The role of each layer is gram to the weakly-typed OS C APIs directly, which improves
outlined below. application robustness. For instance, since the C++ wrappers
are strongly typed, compilers can detect type system violations
3.1.1 The OS Adaptation Layer at compile-time rather than at run-time, which is often the case
with the C-level OS APIs. ACE uses C++ inlining extensively
The OS adaptation layer constitutes approximately 13% of to eliminate performance penalties that would otherwise be in-
ACE, i.e., 18,000 lines of code. This layer resides directly curred from the additional typesafety and levels of abstraction
atop the native OS APIs written in C. The OS adaptation layer provided by the OS adaptation layer and the C++ wrappers.
shields the other layers in ACE from platform-specific depen- The C++ wrappers provided by ACE are quite comprehen-
dencies associated with the following OS APIs: sive, constituting 50% of its source code. Applications can
Concurrency and synchronization: ACE’s adaptation combine and compose these wrappers by selectively inherit-
layer encapsulates OS concurrency APIs for multi-threading, ing, aggregating, and/or instantiating the following compo-
multi-processing, and synchronization; nents:
Interprocess communication (IPC) and shared memory: Concurrency and synchronization components: ACE ab-
ACE’s adaptation layer encapsulates OS APIs for local and stracts native OS multi-threading and multi-processing mecha-
remote IPC and shared memory; nisms like mutexes and semaphores to create higher-level OO
concurrency abstractions like Active Objects [23] and Poly-
Event demultiplexing mechanisms: ACE’s adaptation
morphic Futures [33].
layer encapsulates OS APIs for synchronous and asyn-
chronous demultiplexing I/O-based, timer-based, signal- IPC and filesystem components: The ACE C++ wrappers
based, and synchronization-based events; encapsulate local and/or remote IPC mechanisms [15] such as
sockets, TLI, UNIX FIFOs and STREAM pipes, and Win32
Explicit dynamic linking: ACE’s adaptation layer encapsu-
Named Pipes. ACE wrappers also encapsulate the OS filesys-
lates OS APIs for explicit dynamic linking, which allows ap-
tem APIs, as well.
plication services to be configured at installation-time or run-
time. Memory management components: The ACE memory
File system mechanisms: ACE’s adaptation layer encapsu- management components provide a flexible and extensible ab-
lates OS file system APIs for manipulating files and directo- straction for managing dynamic allocation and deallocation of
ries. shared memory and local heap memory.
5
SELF-CONTAINED MIDDLEWARE
DISTRIBUTED JAWS ADAPTIVE APPLICATIONS
SERVICE WEB SERVER
TOKEN GATEWAY THE ACE ORB
COMPONENTS
SERVER SERVER (TAO)
SERVICE CORBA
FRAMEWORKS ACCEPTOR CONNECTOR
HANDLER HANDLER
OS ADAPTATION LAYER
C PROCESSES/ STREAM SOCKETS/ NAMED SELECT/ DYNAMIC MEMORY SYSTEM
APIS THREADS PIPES TLI PIPES IO COMP LINKING MAPPING V IPC
PROCESS/THREAD COMMUNICATION VIRTUAL MEMORY
SUBSYSTEM SUBSYSTEM SUBSYSTEM
C++ wrappers. These framework components support flexible In general, the ACE framework components facilitate the
configuration of concurrent communication applications and development of communication software that may be updated
services [8]. The framework layer in ACE contains the fol- and extended without modifying, recompiling, relinking, or
lowing components: even restarting running systems [8]. This degree of flexibility
Event demultiplexing components: The ACE Reactor [13] is achieved in ACE by combining C++ language features like
and Proactor [30] are extensible, object-oriented demultiplex- templates, inheritance, and dynamic binding with design pat-
ers that dispatch application-specific handlers in response to terns like Abstract Factory, Strategy, and Service Configurator
various types of I/O-based, timer-based, signal-based, and [7, 31].
synchronization-based events.
Service initialization components: The ACE Connector 3.1.4 Self-contained Distributed Service Components
and Acceptor components [28] decouple the active and pas-
In addition to its C++ wrappers and framework components,
sive initialization roles, respectively, from application-specific
ACE provides a standard library of distributed services that are
tasks that communication services perform once initialization
packaged as self-contained components. Although these ser-
is complete.
vice components are not strictly part of the ACE framework,
Service configuration components: The ACE Service Con- they play two important roles:
figurator [31] supports the configuration of applications whose
services may be assembled dynamically at installation-time Factoring out reusable distributed application building
and/or run-time. blocks: These service components provide reusable imple-
mentations of common distributed application tasks such as
Hierarchically-layered stream components: The ACE
naming, event routing, logging, time synchronization, and net-
Streams components [8] simplify the development of com-
work locking.
munication software applications that are composed of
hierarchically-layered services, e.g. user-level protocol stacks. Demonstrating common use-cases of ACE components:
ORB adapter components: ACE can be integrated seam- The distributed services also demonstrate how ACE compo-
lessly with single-threaded and multi-threaded CORBA imple- nents like reactors, service configurators, acceptors and con-
mentations via ORB adapters [10]. nectors, active objects, and IPC wrappers can be used effec-
6
in args
tively to develop flexible and efficient communication soft-
ware. CLIENT
operation()
SERVANT
out args + return value
State
I/O Strategy Cached Virtual
Framework Filesystem
many tedious and error-prone aspects of distributed applica-
tion development, including:
Asynchronous Completion Token Tilde ~
Service Configurator
Authentication, authorization, and data security; Expander
/home/...
Protocol
Service location and binding;
Acceptor
State
Presentation conversion issues involving network byte- Protocol Pipeline Strategy
Framework Framework
ordering and parameter marshaling. Pipes and Filters Active Object Strategy
7
4 Developing High-performance Web threads or processes (for concurrent Web servers) or managing
sets of socket handles (for single-threaded concurrent servers).
Servers with Patterns and Frame- Each request is processed by a handler, which goes through a
work Components lifecycle of parsing the request, logging the request, fetching
file status information, updating the cache, sending the file,
The benefits of applying frameworks and patterns to commu- and cleaning up after the request is done. When the response
nication software is best introduced by example. This section returns to the client with the requested file, it is parsed by an
describes the structure and functionality high-performance HTML parser so that the file may be rendered. At this stage,
Web servers developed using the patterns and framework com- the requester may issue other requests on behalf of the client,
ponents in ACE. Many error handling details are omitted to e.g., in order to maintain a client-side cache.
keep the code examples concise. In addition, the examples
focus on features that are portable across multiple OS plat-
forms, though noteworthy platform-specific features of ACE 4.2 Overview of an OO Web Server Communi-
(such as asynchronous I/O and I/O completion ports) are de- cation Software Architecture
scribed where appropriate.
Figure 6 illustrates the general object-oriented communication
software architecture for the Web servers covered in this sec-
4.1 Overview of a Web System tion. The roles performed by the components in this architec-
Figure 5 illustrates the general architecture of a Web system.
The diagram provides a layered view of the architectural com-
HTTP HTTP
1: GET ~schmidt Handler Handler
WWW HTTP/1.0 WWW
CLIENT SERVER HTTP Sock Sock
2: index.html Stream Stream
Handler
HTTP
PROTOCOL Sock Acceptor
HTML
HANDLERS Stream
PARSER Event Sock
GUI Dispatcher Acceptor
DISPATCHER
REQUESTER
8
The HTTP Acceptor: This component is a factory that ac- 4.3.1 Strategic Patterns in JAWS
cepts connections from clients and creates HTTP Handlers
to process the requests from clients. There is typically one The following patterns are strategic because they are ubiqui-
HTTP Acceptor per server, though certain concurrency tous to the domain of communication software. Therefore,
strategies (such as Thread Pool) allocate multiple Accep- they significantly influence the software architecture of Web
tors to leverage OS multi-threading capabilities. An HTTP servers.
Acceptor contains an ACE SOCK Acceptor, which en- The Acceptor pattern: This pattern [28] decouples passive
capsulates the passive connection establishment capabilities of connection establishment from the service performed once the
TCP/IP sockets. connection is established. Figure 8 illustrates the structure of
the Acceptor pattern in the context of Web servers. The HTTP
The SOCK Acceptor and SOCK Stream are C++ wrap-
pers provided by ACE. They shield applications from non- HTTP Handler
portable, tedious, and error-prone aspects of developing com- HTTP HTTP
munication software using the native OS socket interfaces Handler Acceptor
written in C. Other ACE components will be introduced peer_stream_ peer_acceptor_
throughout this section, as well. open() CREATE & handle_input()
ACTIVATE
IES
OTIF
4.3 Design Patterns for Web Server Communi- Event N
9
Handler inherit from the abstract Event Handler in-
terface. The difference is that this Event Handler de-
fines completion hooks rather than initiation hooks. There-
fore, when asynchronous invoked accept and read oper-
APPLICATION- APPLICATION-
APPLICATION- APPLICATION-
INDEPENDENT DEPENDENT
Figure 11: Structure of the Active Object Pattern
overlapped_result =
HTTP Async
GetQueuedCompleteStatus(); Handler Write
overlapped_result->complete() method requests (such as get request) into Method
HTTP
Objects that are stored on an Activation Queue. The
Event_Handler
Acceptor Scheduler, which runs in a separate thread from the client,
handle_accept()
handle_read_file()
dequeues these Method Objects and transforms them
handle_write_file() Async Async back into method calls to perform the specified HTTP pro-
handle_timeout() Op Accept cessing. The Active Object pattern is used in the Thread-
get_handle() open()
n cancel() per-Request model in Section 4.4.2, the Thread Pool models
A
1 n in Section 4.4.3, and the Thread-per-Session model in Sec-
1
tion 4.4.4,
Completion 1 Timer_Queue
Dispatcher 1 n
1
schedule_timer(h) The Half-Sync/Half-Async pattern: This pattern [29] de-
handle_events() Handles cancel_timer(h) couples synchronous I/O from asynchronous I/O in a system
register_handle() expire_timer(h)
in order to simplify concurrent programming effort without
1 1
degrading execution efficiency. Figure 12 illustrates the struc-
ture of the Half-Sync/Half-Async pattern in the context the
Figure 10: Structure of the Proactor Pattern queue-based Thread Pool model in Section 4.4.3. In this de-
sign, the Reactor is responsible for reading HTTP requests
from clients and and enqueueing valid requests on a Message
Queue. This Message Queue feeds the pool of Active
Objects that process the requests concurrently.
10
Active
is domain-independent and thus widely applicable, the prob-
SYNC TASK
Active Object Active lem it addresses does not impact Web server software architec-
LEVEL
Object Object ture as pervasively as strategic patterns like the Active Object
and Reactor. A thorough understanding of tactical patterns is
4: dequeue(msg)
essential, however, to implement highly flexible software that
5: svc(msg) is resilient to changes in application requirements and platform
environments.
QUEUEING
: Message
LEVEL
11
REGISTERED
2: accept()
OBJECTS 5: recv(request) 3: make_handler()
2: HANDLE INPUT 6: process(request)
APPLICATION
HTTP 3: CREATE HANDLER HTTP
HTTP
LEVEL
Handler 4: ACCEPT CONNECTION Handler
HTTP HTTP Acceptor
5: ACTIVATE HANDLER Handler Handler
Event Event
HTTP Event Event Handler
Handler Handler Handler
HTTP Acceptor
Handler
Reactor
4: handle_input()
1: handle_input()
FRAMEWORK
LEVEL
Initiation
Dispatcher
6: PROCESS HTTP REQUEST
Reactor
SERVER
OS EVENT DEMULTIPLEXING INTERFACE
1: CONNECT
KERNEL
LEVEL
CLIENT
CLIENT CLIENT
12
way to implement the Event Dispatcher in high-
performance Web servers [32]. This model is most effective
2: HANDLE INPUT on OS platforms (such as Windows NT and Solaris 2.6) that
HTTP 3: CREATE HANDLER
Handler permit simultaneous calls to the accept function on the same
4: ACCEPT CONNECTION
HTTP 5: SPAWN THREAD acceptor socket. On platforms that do not allow this (such as
Handler most SVR4 implementations of UNIX) it is necessary to ex-
HTTP HTTP plicitly serialize accept with an ACE Mutex synchroniza-
Acceptor
Handler tion object.
Reactor There are several variations of the Thread Pool model. Fig-
ure 16 and Figure 17 illustrate the handle-based and queue-
based synchronous Thread Pool models, respectively. Fig-
ure 18 illustrates the asynchronous Thread Pool model. Each
6: PROCESS HTTP REQUEST of these variants is outlined below:
13
Client requests can execute concurrently until the number invokes the dequeue method of the request queue, which
of simultaneous requests exceed the number of threads in the blocks awaiting client requests.
pool. At this point, additional requests are queued in the ker- Once an HTTP request has been dequeued by a thread in
nel’s socket listen queue until a thread in the pool finishes the pool this thread performs the necessary computation and
its processing and becomes available. To reduce latency, the filesystem operations to service the request. The requested
Thread Pool can be configured to always have threads avail- data is then transmitted synchronously to the client. After
able to service new requests. However, the number of threads the data transmission completes the thread returns to the pool
needed to support this policy can be very high during peak and reinvokes dequeue method to retrieve another HTTP re-
loads as threads block in long-duration synchronous I/O oper- quest.
ations. In contrast with the handle-based Thread Pool model, the
One drawback with the handle-based Thread Pool model is queue-based Thread Pool design makes it possible to accept
that the size of the socket listen queue is relatively small (i.e., (or reject) all incoming connections rapidly and prioritize how
around 8 to 10 connections on most OS platforms). Therefore, each client is processed. The primary drawback stems from
high volume servers that receive hundreds of Web hits per sec- the extra context switching and synchronization required to
ond may not be able to accept connections fast enough to keep manage the queue in the Web server.
the kernel from rejecting clients. Moreover, it is not possible
to prioritize which connections are dropped since the kernel The Asynchronous Thread Pool: As shown in Figure 18,
does not distinguish among different clients. this model uses the ACE Proactor, which manages an I/O
completion port. An I/O completion port is a thread-safe
The Queue-based Synchronous Thread Pool: As shown
in Figure 17, this model uses the Half-Sync/Half-Async pat-
tern, which combines the Reactor and Active Object patterns.
In this model, the ACE Reactor thread accepts connections I/O 1: INITIATE ASYNC ACCEPT
Completion 2: RUN EVENT LOOP
HTTP 4: ACCEPT COMPLETES
Port 5: QUEUE COMPLETION
Handler
2: HANDLE INPUT
Message 3: ENQUEUE REQUEST 6: DEQUEUE COMPLETION Async Async
Queue & PROCESS
Active HTTP REQUEST Async Read Accept
HTTP Handler
Object
Handler HTTP
Write Async
4: DEQUEUE & HTTP Handler Accept
PROCESS HTTP
Active Handler HTTP
REQUEST
HTTP Handler
Object Handler Proactor
Acceptor
Active
Active
Object
Object Reactor
7: PROCESS HTTP REQUEST
3: HTTP
REQUEST
5: PROCESS HTTP REQUEST SERVER
1: HTTP
REQUEST CLIENT
SERVER
CLIENT
CLIENT
CLIENT
CLIENT
CLIENT
Figure 17: Queue-based Synchronous Thread Pool Web Figure 18: Asynchronous Thread Pool Web Server Model
Server Model
queue of I/O completion notifications that resides in the OS
from clients (via the HTTP Acceptor) and manages all the kernel (in contrast, the queue-based Thread Pool managed
HTTP Handlers. the thread in user-space). Each I/O operation is initiated and
When HTTP requests arrive from clients they are vali- “handed off” to the kernel, where it runs to completion. There-
dated briefly by the associated HTTP Handler in the ACE fore, the initiating thread does not block. When these opera-
Reactor thread and then enqueued in the thread-safe ACE tions complete asynchronously, the kernel queues the resulting
Message Queue that joins the “async” and “sync” layers notifications at the appropriate I/O completion port.
in the Web server. Each Active Object in the thread pool Like the synchronous Thread Pool model, the asynchronous
14
Thread Pool is created during Web server initialization. Un-
like the synchronous model, however, the threads wait on an
3: SPAWN THREAD 2: CREATE, ACCEPT,
I/O completion port rather than waiting on accept. The OS PER CONNECTION AND ACTIVATE
queues up results from all asynchronous operations (e.g., asyn- HTTP_HANDLER
chronous accepts, reads, and writes) on the I/O completion HTTP HTTP
port. The result of each asynchronous operation is handled by Handler Handler HTTP HTTP
Handler
Acceptor
a thread selected by the OS from the pool of threads waiting
Reactor
on the completion port. The thread that dequeues the com-
pletion notification need not be the same one that initiated the
operation.
4: PROCESS HTTP REQUEST
The asynchronous Thread Pool model is typically less re- 1: HTTP
REQUEST
source intensive and provides more uniform latency under SERVER
heavy workloads than synchronous Thread Pool models [32]. CLIENT
15
example applications, including JAWS. ACE has been used in [13] D. C. Schmidt, “Reactor: An Object Behavioral Pattern for
research and development projects at many universities and Concurrent Event Demultiplexing and Event Handler Dispatch-
companies. For instance, it has been used to build avionics ing,” in Pattern Languages of Program Design (J. O. Coplien
systems at Boeing [11]; telecommunication systems at Bell- and D. C. Schmidt, eds.), pp. 529–545, Reading, MA: Addison-
Wesley, 1995.
core [13], Ericsson [37], and Motorola [9]; medical imaging
systems at Siemens [31] and Kodak [10]; and many academic [14] M. K. McKusick, K. Bostic, M. J. Karels, and J. S. Quarter-
man, The Design and Implementation of the 4.4BSD Operating
research projects.
System. Addison Wesley, 1996.
[15] D. C. Schmidt, T. H. Harrison, and E. Al-Shaer, “Object-
References Oriented Components for High-speed Network Programming,”
in Proceedings of the 1st Conference on Object-Oriented Tech-
[1] F. P. Brooks, The Mythical Man-Month. Reading, MA: nologies and Systems, (Monterey, CA), USENIX, June 1995.
Addison-Wesley, 1975. [16] Object Management Group, The Common Object Request Bro-
[2] A. Gokhale and D. C. Schmidt, “Measuring the Performance ker: Architecture and Specification, 2.2 ed., Feb. 1998.
of Communication Middleware on High-Speed Networks,” in [17] D. Box, Essential COM. Addison-Wesley, Reading, MA, 1997.
Proceedings of SIGCOMM ’96, (Stanford, CA), pp. 306–317,
[18] A. Wollrath, R. Riggs, and J. Waldo, “A Distributed Object
ACM, August 1996.
Model for the Java System,” USENIX Computing Systems,
[3] D. C. Schmidt, S. Mungee, S. Flores-Gaitan, and A. Gokhale, vol. 9, November/December 1996.
“Alleviating Priority Inversion and Non-determinism in Real- [19] D. C. Schmidt and S. Vinoski, “Object Adapters: Concepts
time CORBA ORB Core Architectures,” in Proceedings of the and Terminology,” C++ Report, vol. 11, November/December
Fourth IEEE Real-Time Technology and Applications Sympo- 1997.
sium, (San Francisco, CA), IEEE, December 1997.
[20] D. C. Schmidt, A. Gokhale, T. Harrison, and G. Parulkar,
[4] G. Booch, Object Oriented Analysis and Design with Ap- “A High-Performance Endsystem Architecture for Real-time
plications (2nd Edition). Redwood City, California: Ben- CORBA,” IEEE Communications Magazine, vol. 14, February
jamin/Cummings, 1993. 1997.
[5] M. E. Fayad and D. C. Schmidt, “Object-Oriented Application [21] A. Gokhale and D. C. Schmidt, “Measuring and Optimizing
Frameworks,” Communications of the ACM, vol. 40, October CORBA Latency and Scalability Over High-speed Networks,”
1997. Transactions on Computing, vol. 47, no. 4, 1998.
[6] R. Johnson and B. Foote, “Designing Reusable Classes,” [22] D. C. Schmidt, D. L. Levine, and S. Mungee, “The Design and
Journal of Object-Oriented Programming, vol. 1, pp. 22–35, Performance of Real-Time Object Request Brokers,” Computer
June/July 1988. Communications, vol. 21, pp. 294–324, Apr. 1998.
[7] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Pat- [23] R. G. Lavender and D. C. Schmidt, “Active Object: an Ob-
terns: Elements of Reusable Object-Oriented Software. Read- ject Behavioral Pattern for Concurrent Programming,” in Pat-
ing, MA: Addison-Wesley, 1995. tern Languages of Program Design (J. O. Coplien, J. Vlissides,
[8] D. C. Schmidt and T. Suda, “An Object-Oriented Framework and N. Kerth, eds.), Reading, MA: Addison-Wesley, 1996.
for Dynamically Configuring Extensible Distributed Commu- [24] F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad, and
nication Systems,” IEE/BCS Distributed Systems Engineering M. Stal, Pattern-Oriented Software Architecture - A System of
Journal (Special Issue on Configurable Distributed Systems), Patterns. Wiley and Sons, 1996.
vol. 2, pp. 280–293, December 1994. [25] R. Johnson, “Frameworks = Patterns + Components,” Commu-
[9] D. C. Schmidt, “A Family of Design Patterns for Application- nications of the ACM, vol. 40, Oct. 1997.
level Gateways,” The Theory and Practice of Object Systems [26] J. Vlissides, “The Hollywood Principle,” C++ Report, vol. 8,
(Special Issue on Patterns and Pattern Languages), vol. 2, no. 1, Feb. 1996.
1996.
[27] A. Stepanov and M. Lee, “The Standard Template Library,”
[10] I. Pyarali, T. H. Harrison, and D. C. Schmidt, “Design Tech. Rep. HPL-94-34, Hewlett-Packard Laboratories, April
and Performance of an Object-Oriented Framework for High- 1994.
Performance Electronic Medical Imaging,” USENIX Comput-
[28] D. C. Schmidt, “Acceptor and Connector: Design Patterns for
ing Systems, vol. 9, November/December 1996.
Initializing Communication Services,” in Pattern Languages of
[11] T. H. Harrison, D. L. Levine, and D. C. Schmidt, “The De- Program Design (R. Martin, F. Buschmann, and D. Riehle,
sign and Performance of a Real-time CORBA Event Service,” eds.), Reading, MA: Addison-Wesley, 1997.
in Proceedings of OOPSLA ’97, (Atlanta, GA), ACM, October [29] D. C. Schmidt and C. D. Cranor, “Half-Sync/Half-Async: an
1997. Architectural Pattern for Efficient and Well-structured Con-
[12] D. C. Schmidt and C. Cleeland, “Applying Patterns to Develop current I/O,” in Pattern Languages of Program Design (J. O.
Extensible ORB Middleware,” Submitted to the IEEE Commu- Coplien, J. Vlissides, and N. Kerth, eds.), Reading, MA:
nications Magazine, 1998. Addison-Wesley, 1996.
16
[30] T. Harrison, I. Pyarali, D. C. Schmidt, and T. Jordan, “Proac-
tor – An Object Behavioral Pattern for Dispatching Asyn-
chronous Event Handlers,” in The 4th Pattern Languages of
Programming Conference (Washington University technical re-
port #WUCS-97-34), September 1997.
[31] P. Jain and D. C. Schmidt, “Service Configurator: A Pattern
for Dynamic Configuration of Services,” in Proceedings of the
3rd Conference on Object-Oriented Technologies and Systems,
USENIX, June 1997.
[32] J. Hu, I. Pyarali, and D. C. Schmidt, “Measuring the Impact
of Event Dispatching and Concurrency Models on Web Server
Performance Over High-speed Networks,” in Proceedings of the
2nd Global Internet Conference, IEEE, November 1997.
[33] R. H. Halstead, Jr., “Multilisp: A Language for Concurrent
Symbolic Computation,” ACM Trans. Programming Languages
and Systems, vol. 7, pp. 501–538, Oct. 1985.
[34] D. C. Schmidt and J. Hu, “Developing Flexible and High-
performance Web Servers with Frameworks and Patterns,” ACM
Computing Surveys, vol. 30, 1998.
[35] J. C. Mogul, “The Case for Persistent-connection HTTP,” in
Proceedings of ACM SIGCOMM ’95 Conference in Computer
Communication Review, (Boston, MA, USA), pp. 299–314,
ACM Press, August 1995.
[36] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, and T. Berners-Lee,
“Hypertext Transfer Protocol – HTTP/1.1,” Standards Track
RFC 2068, Network Working Group, January 1997. Available
from http://www.w3.org/.
[37] D. C. Schmidt and P. Stephenson, “Experiences Using Design
Patterns to Evolve System Software Across Diverse OS Plat-
forms,” in Proceedings of the 9th European Conference on
Object-Oriented Programming, (Aarhus, Denmark), ACM, Au-
gust 1995.
17