Introduction To Algorithms - Design Techniques and Analysis M. Alsuwaiyel - Chapter - 1

Table of Contents
1. INTRODUCING ALGORITHMS ........................................................................................ 11

1.1 Introduction .................................................................................................................................................................... 11
1.1.1 Describing Algorithms.............................................................................................................................................. 12
1.1.2 Algorithm-Design Guidelines ................................................................................................................................... 16
1.1.3 Proof Techniques ...................................................................................................................................................... 18
1.2 Problem-Solving Techniques......................................................................................................................................... 20
1.2.1 Brute-Force Search ................................................................................................................................................... 20
1.2.2 Problem Reduction.................................................................................................................................................... 25
1.3 Running-Time and Space Complexity Analysis........................................................................................................... 28
1.3.1 Asymptotic Notations ............................................................................................................................................... 31
1.3.2 Classification of Algorithms Based on Order of Running Time ............................................................................... 36
1.3.3 Space Complexity Analysis ...................................................................................................................................... 39
1.4 Searching......................................................................................................................................................................... 40
1.4.1 Sequential Search...................................................................................................................................................... 40
1.4.2 Binary Search............................................................................................................................................................ 41
1.5 Sorting ............................................................................................................................................................................. 45
1.5.1 Selection Sort ............................................................................................................................................................ 45
1.5.2 Insertion Sort............................................................................................................................................................. 47
1.5.3 Mergesort .................................................................................................................................................................. 49
1.5.4 Quicksort................................................................................................................................................................... 54
1.5.5 Sorting on Multiple Fields ........................................................................................................................................ 60
1.5.6 A Lower Bound for Sorting by Comparison............................................................................................................. 63
2. FUNDAMENTAL DATA STRUCTURES........................................................................... 74

2.1 Graphs and Trees ........................................................................................................................................................... 75
2.1.1 Elementary Graph Definitions .................................................................................................................................. 76
2.1.2 Trees.......................................................................................................................................................................... 77
2.1.3 Data Structures for Graphs........................................................................................................................................ 79
2.2 Priority Queues............................................................................................................................................................... 85
2.2.1 Heapsort .................................................................................................................................................................... 93
2.2.2 Application of Queues: Discrete Event Simulation .................................................................................................. 96
2.3 Union-Find for Disjoint Sets.......................................................................................................................................... 98
2.3.1 Applications of Union-Find ADT ........................................................................................................................... 104
2.4 Hashing.......................................................................................................................................................................... 106
2.4.1 Open Addressing..................................................................................................................................................... 107
2.4.2 Separate Chaining ................................................................................................................................................... 108
2.4.3 HashTable Implementation and Usage ................................................................................................................... 109
2.4.4 Analysis of Separate Chaining ................................................................................................................................ 116
2.4.5 Analysis of Open Addressing.................................................................................................................................. 118
2.4.6 Selecting a Hash Function....................................................................................................................................... 120
2.4.7 String Search: The Rabin-Karp Algorithm ............................................................................................................. 123
2.5 Height-Balanced Trees................................................................................................................................................. 126
2.5.1 2-3 Trees ................................................................................................................................................................. 127
2.5.2 Treaps...................................................................................................................................................................... 131
3. INDUCTION..................................................................................................................... 146
3.1 Structure of a Proof by Induction............................................................................................................................... 147

3.1.1 Examples of Induction Proofs................................................................................................................................. 147
3.1.2 Pitfalls in Induction Proofs ..................................................................................................................................... 149
3.1.3 Using Induction for Counting ................................................................................................................................. 151
3.2 Induction, Recursion and Iteration: faces to the same coin ..................................................................................... 153
3.2.1 Writing an Induction Step: Extend or Reduce......................................................................................................... 154
3.2.2 Strong Induction Versus Strengthening the Induction Hypothesis ......................................................................... 154
3.2.3 Converting Recursion into Iteration........................................................................................................................ 155
3.3 The All-Same Problem................................................................................................................................................. 158
3.4 Constructing a Tournament Schedule........................................................................................................................ 160
3.5 The Coin-Change Problem .......................................................................................................................................... 162
3.5.1 Using Strong Induction for the Coin-Change Problem ........................................................................................... 163
3.6 Basics Problems on Sets and Sequences ..................................................................................................................... 164
3.6.1 Set Union ................................................................................................................................................................ 164
3.6.2 Even-Odd Partitioning ............................................................................................................................................ 165
3.6.3 Even-Odd Partitioning Using Strong Induction ...................................................................................................... 167
3.6.4 Sequence Partitioning ............................................................................................................................................. 167
3.6.5 Maximum-Sum Subsequence.................................................................................................................................. 169
3.7 Finding the k-th Smallest Element.............................................................................................................................. 171
3.7.1 Finding the k-th Smallest Element Using Induction ............................................................................................... 171
3.7.2 Finding the k-th Smallest Element Using Partitioning ............................................................................................ 172
3.8 Numeric Computations................................................................................................................................................ 174
3.8.1 Polynomial Evaluation............................................................................................................................................ 175
3.8.2 Fast Integer-Exponentiation .................................................................................................................................... 177
3.9 Distribution-Based Sorting .......................................................................................................................................... 180
3.9.1 Counting Sort .......................................................................................................................................................... 180
3.9.2 Radix Sort (Bucket Sort)........................................................................................................................................ 181
3.9.3 Radix-Exchange Sort (Matesort)............................................................................................................................. 183
3.10 Longest Path in a Tree ............................................................................................................................................... 187
4. DIVIDE AND CONQUER................................................................................................. 197

4.1 Solving Recurrence Equations .................................................................................................................................... 198
4.1.1 The Substitution Method......................................................................................................................................... 198
4.1.2 The Induction Method............................................................................................................................................. 200
4.1.3 The Characteristic Equation Method....................................................................................................................... 201
4.1.4 Recursion Trees and the Master Theorem............................................................................................................... 205
4.1.5 Average-Case Analysis of Quicksort ...................................................................................................................... 207
4.2 Constructing a Tournament Schedule - revisited ...................................................................................................... 209
4.3 The MinMax Problem.................................................................................................................................................. 210
4.4 Finding the Majority Element..................................................................................................................................... 212
4.5 The Skyline Problem .................................................................................................................................................... 216
4.6 Polynomial Multiplication ........................................................................................................................................... 220
4.7 Matrix Multiplication................................................................................................................................................... 223
4.7.1 Strassens Matrix Multiplication............................................................................................................................. 223
4.7.2 Winograds Matrix Multiplication .......................................................................................................................... 224
5. DYNAMIC PROGRAMMING........................................................................................... 234

5.1 Computing the Binomial Coefficients......................................................................................................................... 234
5.2 The 0/1-Knapsack Problem ......................................................................................................................................... 238
5.3 Recovering Solution Components ............................................................................................................................... 241
5.4 The Subset-Sum Problem ............................................................................................................................................ 242
5.5 Memoization ................................................................................................................................................................. 244
5.6 Longest Common Subsequence................................................................................................................................... 246
5.7 Weighted-Interval Scheduling..................................................................................................................................... 249
5.8 Principle of Optimality ................................................................................................................................................ 251
5.9 Shortest Paths ............................................................................................................................................................... 253
5.9.1 Floyds Shortest-Paths Algorithm........................................................................................................................... 254
5.9.2 Bellman-Ford Shortest-Paths Algorithm................................................................................................................. 260
5.9.3 DAG Reductions..................................................................................................................................................... 264
5.10 The Traveling Salesman Problem ............................................................................................................................. 269
5.11 Polygon Triangulation ............................................................................................................................................... 272
6. GREEDY TECHNIQUE ................................................................................................... 280

6.1 DijKstras Single-Source Shortest-Paths Algorithm ................................................................................................. 283
6.2 Minimum Spanning Trees ........................................................................................................................................... 287
6.2.1 Prims MST Algorithm ........................................................................................................................................... 289
6.2.2 Kruskals MST Algorithm ...................................................................................................................................... 293
6.2.3 Applications of MSTs ............................................................................................................................................. 297
6.3 Graph Coloring ............................................................................................................................................................ 299
6.3.1 Greedy Coloring...................................................................................................................................................... 302
6.3.2 Graph Coloring Heuristics & Metaheuristics.......................................................................................................... 305
6.4 Scheduling ..................................................................................................................................................................... 309
6.4.1 Minimizing Time in the System.............................................................................................................................. 309
6.4.2 Interval Scheduling ................................................................................................................................................. 310
6.4.3 Scheduling with Deadlines...................................................................................................................................... 312
6.4.4 Scheduling All Intervals.......................................................................................................................................... 313
6.5 Huffman Coding........................................................................................................................................................... 315
7. GRAPH ALGORITHMS................................................................................................... 327

7.1 Depth-First Search ....................................................................................................................................................... 327
7.2 Breadth-First Search.................................................................................................................................................... 333
7.2.1 Maze Traversal........................................................................................................................................................ 336
7.3 Topological Sorting ...................................................................................................................................................... 339
7.3.1 Critical-Path Analysis ............................................................................................................................................. 342
7.4 Bipartite Matching ....................................................................................................................................................... 344
7.5 Network Flow................................................................................................................................................................ 349
7.5.1 Maximum Flows and Minimum Cuts ..................................................................................................................... 356
7.6 Applications of Matching and Network Flow ............................................................................................................ 358
7.6.1 Reducing Maximum-Cardinality Bipartite-Matching to Max-Flow ....................................................................... 358
7.6.2 Relating Bipartite Matching to Vertex Cover ......................................................................................................... 359

7.6.3 System of Distinct Representatives and Halls Marriage Theorem ........................................................................ 362
7.6.4 Stable Matching ...................................................................................................................................................... 366
7.6.5 Variations of Stable Matching and Network Flows ................................................................................................ 368
8. BACKTRACKING ........................................................................................................... 378

8.1 Graph K-Coloring ........................................................................................................................................................ 379
8.1.1 A Backtracking Algorithm for k-Coloring .............................................................................................................. 381
8.2 Recursive Slot Filling ................................................................................................................................................... 384
8.2.1 Permutation Generation .......................................................................................................................................... 385
8.2.2 Generating the Next Permutation............................................................................................................................ 387
8.3 The Eight-Queens Problem ......................................................................................................................................... 390
8.4 A Generic Structure for Backtracking ....................................................................................................................... 393
8.5 Subset Generation ........................................................................................................................................................ 395
8.5.1 A Backtracking Algorithm for the Subset-Sum Problem........................................................................................ 396
8.5.2 A Backtracking Algorithm for the 0/1-Knapsack Problem..................................................................................... 397
8.6 Solving and Constructing Sudoku Puzzles ................................................................................................................. 398
8.6.1 Solving Sudoku Puzzles by Backtracking............................................................................................................... 398
8.6.2 Construction of Sudoku Puzzles ............................................................................................................................. 401
8.6.3 Ranking of Sudoku Puzzles .................................................................................................................................... 403
8.7 Branch-and-Bound....................................................................................................................................................... 407
9. THEORY OF NP.............................................................................................................. 415

9.1 Introduction .................................................................................................................................................................. 415
9.1.1 Problem Reduction and Polynomial-Time Reducibility ......................................................................................... 419
9.1.2 Classical Intractable Problems ................................................................................................................................ 422
9.2 Problem Complexity Classes ....................................................................................................................................... 425
9.3 NP-Complete and NP-Hard Problems........................................................................................................................ 426
9.3.1 Examples of NP-Completeness Proofs ................................................................................................................... 430
9.3.2 Reduction Techniques and Pitfalls.......................................................................................................................... 434
9.4 Techniques for Dealing with Intractable Problems................................................................................................... 437
9.4.1 Parameterized Complexity ...................................................................................................................................... 438
9.5 Approximation Algorithms and Bounds .................................................................................................................... 441
9.5.1 Approximation for the Traveling Salesman problem.............................................................................................. 441
9.5.2 Approximation for Vertex Cover ............................................................................................................................ 442
9.5.3 Approximation for Set Cover.................................................................................................................................. 445
9.5.4 Approximation for Graph Coloring......................................................................................................................... 446
10. ITERATIVE LOCAL SEARCH ...................................................................................... 454

10.1 Overview ..................................................................................................................................................................... 454
10.2 Progressive Search ..................................................................................................................................................... 455
10.2.1 A Priority Queue for All Seasons.......................................................................................................................... 456
10.2.2 A Framework for Using Progressive Search......................................................................................................... 457
10.3 Sample Problems ........................................................................................................................................................ 459
10.3.1 Magic Squares....................................................................................................................................................... 459
10.3.2 Sudoku .................................................................................................................................................................. 463
10.3.3 Graph Coloring ..................................................................................................................................................... 465
11. PROGRAMMING FUNDAMENTALS AND TECHNIQUES........................................... 470

11.1 Data Types .................................................................................................................................................................. 471
11.1.1 Value Types Versus Reference Types .................................................................................................................. 472
11.1.2 Reference Types.................................................................................................................................................... 479
11.2 Classes and Methods .................................................................................................................................................. 484
11.2.1 Modifiers for Classes & Class Members............................................................................................................... 485
11.2.2 Parameter Passing ................................................................................................................................................. 488
11.3 Inheritance and Polymorphisms ............................................................................................................................... 489
11.3.1 Class Inheritance ................................................................................................................................................... 489
11.3.2 Polymorphism ....................................................................................................................................................... 492
11.4 Interfaces..................................................................................................................................................................... 494
11.5 Dynamic Data-Structures .......................................................................................................................................... 506
11.5.1 Linked Lists .......................................................................................................................................................... 506
11.5.2 Binary Search Trees .............................................................................................................................................. 509
11.5.3 Array Doubling: A Priority-Queue Implementation ............................................................................................. 512
11.6 Exceptions Handling .................................................................................................................................................. 515
11.7 Text Files I/O .............................................................................................................................................................. 519
11.8 Generics....................................................................................................................................................................... 523
11.9 Delegates and Events.................................................................................................................................................. 528
11.10 Multithreading.......................................................................................................................................................... 536
11.10.1 Creating and Starting a Thread ........................................................................................................................... 538
11.10.2 Threads in Windows Applications ...................................................................................................................... 543
11.10.3 Thread Synchronization ...................................................................................................................................... 546
11.10.4 Multithreaded Access to Collections................................................................................................................... 559
11.10.5 Applications: Dining Philosophers, Multithreaded Sorting ................................................................................ 560
12. PROGRAMMING EXCURSIONS.................................................................................. 569

12.1 Recursive Drawings.................................................................................................................................................... 570
12.2 Large Number Arithmetic ......................................................................................................................................... 575
12.2.1 Using Large Number Library in a Web-setting..................................................................................................... 576
12.2.2 Implementation of LargeNumber class ................................................................................................................. 577
12.3 Public Key Encryption............................................................................................................................................... 582
12.3.1 RSA in a Nutshell ................................................................................................................................................. 583
12.3.2 A Class for RSA.................................................................................................................................................... 584
12.4 Crossword Generation ............................................................................................................................................... 588
12.4.1 Crossword Generation using Progressive Search.................................................................................................. 590
12.4.2 Filling Strategies ................................................................................................................................................... 598
BIBLIOGRAPHY ................................................................................................................. 603

INDEX ................................................................................................................................. 613
Preface
Why this book?
This book is intended to provide a thorough treatment of fundamental algorithmic concepts, while balancing
coverage between theory and implementation. The standard text books on algorithms limit the presentation of
many algorithms to high-level pseudocode description and omit details needed for algorithm implementation.
On the other hand, this book advocates an approach that emphasizes programming algorithms and their
utilization in some interesting applications.
From my own teaching experience, I have observed that most students find it difficult to comprehend the given
algorithms to the degree that permit them to quickly translate them into correct programs. Although students
come into the algorithm course having taken courses on programming and data structures; still, many students,
have difficulty with some programming fundamentals, such as distinguishing between parameters passed by
reference versus parameters passed by value. In addition, many students have bad code-writing habits, such as
ignoring proper indentation, which makes their programs unreadable and obscures the underlying logic. Such
students end up spending too much time (at the expense of learning the fundamental algorithmic concepts) on
any modest programming task. For these reasons, in teaching the algorithm course, I, and undoubtedly many
other instructors, often avoid giving programming assignments altogether; preferring, instead, to concentrate on
the process of algorithm development. In reality, a computer science professional (or a software engineer) needs
to master both skills.
The author believes that students will enhance their understanding of algorithms and become more motivated if
they are also taught how to become better programmers, say by presenting well-coded complete programs. To
achieve this end, the book and companion website provide complete programs for many of the algorithms and
solved exercises covered in the book. Furthermore, the book includes two programming-oriented chapters: a
chapter on fundamental programming concepts and techniques and another Programming Excursions
chapter presenting several interesting complete applications demonstrating useful data structures, algorithms and
programming techniques. These include the following topics.
Using recursion to draw Sierpinskis curves and display the output by a web browser.
Building a library for doing arithmetic with large numbers. The library is then used for encryption using
RSA public key encryption.
Utilizing priority queues and hash tables to build a web-based crossword generator.
In writing this book, the author has set forth the following goals, which are to be the features of this book.
Organize algorithmic topics in a logical order from basic to advanced.
Present the concepts in a direct clear language.
Give worked-out examples for many of the algorithms presented.
Give different algorithms for the same problem and highlight any relationship among specific problems.
Consider some recent interesting problems and algorithms.
Give plenty of solved and unsolved exercises.
A successful programmer today must be well versed in a mainstream object-oriented programming language
and its associated development environment. The choice today is between Microsofts CSharp/.NET Framework
and JavaSofts Java/Java 2 Enterprise Edition (J2EE). Each technology has its fans and detractors, and each has
a substantial installed base today. These competing technologies are strikingly similar. Both technologies
support the same kinds of applications (Desktop GUI-based and Web-based), and both provide a comprehensive
library to help build such applications. Java virtual machine plays a role similar to the .NET Frameworks CLR
(Common Language Runtime) and even the semantics of the dominant languages (Microsofts CSharp vs. Java)
are quite similar. One key distinction between the two environments is the fact that .NET Framework is more
tightly integrated with Windows, while the Java environment runs on diverse operating systems.
The programs presented in this book are written in Microsofts CSharp (the CSharp language is also named C#).
CSharp (a registered trademark to Microsoft) is a modern object-oriented programming language created by
Microsoft. It was introduced as part of Microsoft Visual Studio .NET (2002). The language has undergone many
enhancements since its first release, with C# 4.0 being the most recent version..
CSharp and Java have quite similar syntax and semantics. Therefore, readers who feel more comfortable with
Java can still make use of the programs and code-snippets presented in this book. A comparison between
CSharp and Java can be found at http://en.csharp-online.net/CSharp_FAQ.
Targeted Audience
This book can serve as a primary textbook for the standard undergraduate course on algorithms. It can also serve
as a secondary reference for an MS-level course, especially for those students who need to strengthen their
understanding of the subject. Lastly, the book can be used for self-study by computer science professionals,
since it discusses technical issues in algorithm design as well as programming aspects.
Using the Book for a Course on Algorithms
The structure and sequence of topics have been designed to fit the standard academic course on algorithms.
However, the book includes too many topics to be covered in one semester course. The author suggests the
following outline for topics to be covered within a 15-week semester (the equivalent of 45 one-hour lectures).
Ch.1 (skip 1.5.6)
7 lectures
Ch.2 (skip 2.2.2, 2.4.4, 2.4.5, 2.4.6, 2.5)
5 lectures
Ch.3 (skip 3.10)
7 lectures
Ch. 4 (skip 4.7)
4 lectures
Ch. 5 (skip 5.9.3, 5.11)
5 lectures
Ch. 6 (skip 6.3.2, 6.4.3, 6.4.4)
5 lectures
Ch. 7 (skip 7.5, 7.6)
4 lectures
Ch. 8 (skip 8.6, 8.7)
4 lectures
Ch. 9 (skip 9.4, 9.5)
4 lectures
The preceding outline suggests the topics that students must have a fair knowledge about by the time they finish
with the course. Admittedly not all of the topics listed above can be covered by class lectures. The idea is that
some of these topics are too basic (for example, 1.5.5, 2.1, 3.1.2, 3.1.3, 4.1.3, 5.5, 6.3.2, 6.4.2) to be covered in
a class, and thus should be covered as reading assignments. Finally, to enhance their programming skills, the
students should be encouraged to gradually read parts of Chapters 11 and 12, as they progress through the
course.
Introducing Algorithms
1. Introducing Algorithms
1.1 Introduction
The field of algorithms, as a branch of computer science and modern mathematics, focuses on developing and
formalizing problem-solving techniques. In such context, the word algorithm corresponds to a step-by-step
procedure for solving a problem by computer. The term algorithm is derived from the name of the Persian
mathematician Al-Khwarizmi (780-850) Al-Khwarizmis book Hisab al-jabr wal-muqabala is considered to
be the first known book on algebra.
The concept of an algorithm originated as a means of recording procedures for solving mathematical problems
such as finding the common divisors of two integers or multiplying two numbers. One of the oldest, yet
interesting, algorithms is Euclids gcd algorithm for finding the greatest common divisor of two integers. The
study and development of algorithms became a scientific endeavor with the invention of digital computers in the
late 1940s and early 1950s, because such devices can be programmed to process digital data and carry out
logical and arithmetic calculations at high speeds. Todays computers are designed to compile and execute
programs written in some high-level programming language such as C or Java. However, because algorithms are
intended to be read and understood by humans, algorithms are better expressed in pseudocode, which uses a mix
of English, abstract mathematical notation and high-level programming language constructs (such as variable
assignment and flow-control statements). Describing an algorithm in pseudocode frees the algorithm designer
from worrying about the specific syntax of a particular programming language. Also, it is easier to analyze the
algorithm and reason about its correctness when it is expressed using as simple notation as possible.
Algorithm development is not an end of itself. It is just a means to efficiently solve a problem by computer. The
study of algorithms evolves around two interrelated tasks: design and analysis. To design an algorithm for
a given problem simply means to develop a procedure (expressed in pseudocode) that manipulates the problems
input to obtain the desired output. This is followed up by the algorithm analysis task whose goal is to quantify
the algorithm running time (or memory space) as a function of input size. These tasks are interrelated because
the results of the analysis often lead the algorithm designer to rethink and modify his algorithm. The final form
of an algorithm is its embodiment in a computer program, which gives a materialistic assurance of its utility,
efficiency, and correctness.
Recognition of the importance of algorithms has led Donald Knuth1 to state, Computer Science is the study of
algorithms [Knu74]. This is not an exaggeration; in recent years, we have witnessed that algorithms have
proliferated into various research and study areas of computer science.
This book focuses on the fundamental concepts related to algorithm design, analysis and programming.
Donald E. Knuth is a professor (emeritus) at Stanford. He is most famous for inventing the TeX and Metafont typesetting systems, and
for writing The Art of Computer Programming, a multi-volume compendium on algorithms. He won the Turing award, the ACMs
highest honor, in 1974.
1.1.1 Describing Algorithms

The process of developing an algorithm for some given problem begins with characterizing the problem input
and output, which represent a level of abstraction (approximate model) of reality. An algorithm is then a
procedure (and eventually a computer program) that manipulates the problems input in order to produce the
desired output. The following various means are used to express an algorithm and convey it to its intended
readers:
Textual and pictorial description

Pseudocode
Mathematical equations
Program flowcharts
Textual and Pictorial Description

Since the beginning of human civilization, spoken (or written) language has been an essential means to
communicate (convey and share) information. In a scientific setting, the description of scientific concepts
exhibits two features. First, the textual description uses technical language; that is, it makes heavy use of
terminology (jargon) with predefined semantics. Second, the textual description is often coupled with some kind
of pictorial or graphical representation. In this regard, the field of algorithms is no exception. A glaring example
of the use of graphics is the graphical representation of graphs and trees. For many of the algorithms presented
in this book, we start with a textual description, which is then evolved into pseudocode.
Pseudocode
Pseudocode has become a standard means of expressing algorithms. Pseudocode uses a mix of English, abstract
mathematical notation and high-level programming language constructs (such as variable assignment and
flow-control statements). Program statements are expressed using Pascal-like (or Java-like) syntax but with
relaxed syntactic rules. Because pseudocode is intended to be read by humans and not processed by machines,
strict adherence to programming language syntax rules is not mandatory. For example, we can use line breaks
instead of semicolons to separate program statements, and it is normal not to explicitly declare the data-type of a
variable when writing pseudocode. Also, It is acceptable for pseudocode to include high-level operations (like
the construct A B, to mean copy array B to array A) that have no corresponding constructs in typical
programming languages. Table 1.1 shows some examples of CSharp program statements and the alternative
forms used in pseudocode.
CSharp Program Statement
Alternative form used in Pseudocode
int a = 10;
a 10
if (a==b) S1; else S2;
if a=b then S1 else S2

for i = 1 to n
S1
S2
end for
while (a b) and (c d)
S1
S2
end while
for(int i=1; i <= n; i++)

{ S1; S2; }
while ((a != b) && (c <= d) )
{ S1; S2; }
Table 1.1 Examples of CSharp program statements and the alternative forms used in pseudocode.
Expressing an Algorithm via Program Code: Procedural versus Object-Oriented

The fundamental algorithmic concepts were developed in the 1960s and 1970s when
procedural programming languages were thriving. Traditionally, algorithms in those days were
expressed using Algol-like or Pascal-like programming constructs. In the early 1980s, the
concepts of object-oriented programming came into existence and were advocated by C++ in
the mid 1980s and later by Java in the 1990s.
In procedural programming, we do not use the concept of an object acting on itself; rather,
any method we write, is passed explicit parameters for any data structures that need to be
manipulated by the method. Object-oriented programming, on the other hand, calls for the
methods that manipulate data-structures to be encapsulated within the definition of the datastructures themselves.
Since the late 1970s, several algorithm-books authors encouraged using object-oriented
programming concepts for coding data structures, and promoted the concept of ADTs
(Abstract Data Types). These calls for the separation of interface from implementation and,
while making the interface public, implementation details should be hidden and not accessible
from outside of the ADT.
In recent years, with the proliferation of a number of functional (declarative) programming
languages such as Scheme and Haskell, some institutions, notably MIT, opted at some stage
for teaching algorithms using functional approach. However, the mainstream view remains
that algorithmic concepts are better presented using a procedural style, and that an algorithm
should be expressed as a self-contained procedure. Such procedures readily map into CSharp
or Java program methods that are declared as static. This is the style followed in this book
with few exceptions.
Note: For most of the algorithms presented in this book, in order to economize on space, we
confine ourselves to giving either pseudocode or actual program code but not both. Also, more
often than not, a program method is assumed static, even if it is not explicitly stated.
Mathematical Equations
Mathematical equations or functions are handy in defining recursive algorithms a definition is recursive if it
has a self-reference; for instance, a nationality law might define eligibility for American nationality as follows:
A person is an American if his father (or mother) is an American. Any such definition (formulation) normally
utilizes two equations: recursive and nonrecursive. For example, consider Euclids algorithm for computing the
greatest common divisor (gcd) of two positive integers a, b where a b. The algorithm is specified by the
following Equations 1.1 and 1.2 Note that we normally specify the nonrecursive (base) equation first.
gcd(a,b) = b
gcd(a,b) = gcd(b, a mod b)
if a mod b = 0
if a mod b > 0
1.1
1.2
In the preceding formulation the expression a mod b is the remainder of division of a by b. Using this algorithm,
we can compute the gcd of 82 and 12 as follows:
gcd(82,12)=gcd(12, 82 mod 12)=gcd(12,10)=gcd(10, 12 mod 10)=gcd(10,2)=2.
Normally, the base equation deals with special (or small) inputs which the solution can be given directly. In such
cases, to guarantee termination, the recursive equation must decrease the values of its input parameters in order
to eventually reach the parameter-values that are handled by the base equation. For the preceding gcd-algorithm,
we observe that the recursive equation specifies a function with a and b as input parameters that is expressed as
the same function with b and (a mod b) as input parameters. Thus, the first parameter is decreased from a to b,
because we have already assumed that a b, and the second parameter is decreased from b to (a mod b),
because division of a by b always leaves a remainder < b.
Translating an algorithm specified by recursive equations into pseudocode is straightforward and purely
mechanical; we merely rewrite the equations using a different syntax. For example, based on the previous
equations, we can express the preceding gcd algorithm as the following CSharp program method:
int gcd(int a, int b)
{ // returns the greatest common divisor of the positive integers a and b
if (a % b == 0) return b; // The base equation
else return gcd(b, a % b); // The recursive equation
}
Program Flowchart
As suggested by its name, the program (or algorithm) flowchart is a diagram showing the execution paths
through the various program statements from start to finish. Program flowcharts were a popular method to
document programs written in FORTRAN a programming language that was widely used during the 1960s
and 1970s. In those days, because FORTRAN lacked structured statements (i.e., a while-loop and other
compound statements), FORTRAN programs were coded with lots of goto statements and were difficult to read,
unless accompanied with flowcharts. Figure 1.1 shows a flowchart for some simple program.
Start
x = 2
False
x < 100
True
Print x
True
x prime
False
x = x+1
Done
Figure 1.1 A program flowchart for a program that prints primes less than 100.
1.1.2 Algorithm-Design Guidelines

We summarize certain guidelines (or blueprints) that govern the process of algorithm development. These
guidelines translate into desirable characteristics that should be exhibited by the algorithm pseudocodedescription. These characteristics, together, ensure that the algorithm can be properly analyzed, make it easier to
prove the algorithm correct, ease the manual process of translating the algorithm into a computer program, and,
finally, ensure that the algorithm, timewise (and spacewise), is practical.
1. Modularity
Modularity is an important principle in system design, where a large complex structure is constructed from
smaller (simpler) building blocks. Using this principle in algorithm design, an algorithm is evolved by a process
of stepwise refinement. This means that the initial description is based on a macroview and is composed of highlevel steps. Then each of these high-level steps is rewritten into simpler, more refined steps. The process is
repeated until the description has enough details that it can be translated into a computer program in a
nonambiguous way using moderate effort. The process is generally known as the top-down design principle.
Modularity calls for dividing a large monolithic pseudocode-description into a number of smaller-sized
procedures, where each procedure embodies some specific functionality. In general, a procedure (i.e., a program
method) is an abstraction that assigns a name to a program code block, along with named parameters
representing input and output. Thus, a procedure has a heading (interface) and a body (implementation). A
procedure is executed only when it is invoked (called) via its interface.
Note: In the context of high-level programming languages, the term module is used as a synonym for a class,
which defines some type of object and its behavior as a set of program methods.
2. Readability
It is essential that the algorithm pseudocode-description be readable. Readability is achieved by using
meaningful variable names, using indentation to indicate nesting of compound statements such as nested-loop
structures and using comments to annotate (and explain the actions of) various parts of the algorithm.
3. Correctness
It is essential that an algorithm for a given problem produce the correct output for all possible inputs (problem
instances). Often it is not feasible to ensure the algorithm correctness by testing the algorithm on an instance-byinstance basis, as there could be a large number of problem instances; rather, a mathematical proof of
correctness must be given. As we will see later, algorithms developed by induction (a well-known
algorithm-design technique) embody their own proof of correctness. An induction-related proof technique
employs the concept of a loop-invariant, which is an assertion about a for-loop (or a while-loop) block that
remains valid in every iteration of the loop.
4. Time and space efficiency
Computer processor time (i.e., time spent by the CPU executing a program) and computer memory are
considered precious and scarce resources that ought to be used efficiently. This is because in reality a computer
is always shared among many competing tasks and is rarely dedicated to executing a single task. Often,
computer time is viewed as a more precious resource than computer memory because computer time is human
time, as there is always a person waiting for a running program to finish. Therefore, for solving a given problem,
we always seek a fast algorithm that uses a reasonable amount of computer memory.
Even with the most efficient algorithm, further reduction in the running time is possible through code
optimization. Normally, code optimization (code tweaking) is done on the actual program-code, and it involves
removing redundant code, relocating loop-invariants (variables or expressions that remain unchanged in all
iterations of a loop) to outside of the loop, and, if applicable, replacing floating-point program variables (and
expressions) by integers, etc.
One form of optimization is caching. Caching is a technique where frequently accessed data is moved from a
slow storage to a faster storage. Here is an example. Suppose you have a program that returns a random quote
from a list of quotes that is stored in a text file. The program is to be running (in the background) continuously
and to return a random quote every time it is accessed. To return a random quote, the program can open the file,
skip a random number of lines, and then return the quote found at the current line. But this is slow. A faster
solution would be to read the file once (when the program is started) and store (cache) all quotes in an array.
Any time a quote is needed, the program looks up a random position in the array.
Caching can take on other forms such as using a lookup in lieu of repeated, complex computations and, more
generally, can be seen as setting aside some space (memory) to save on time. This kind of time-space trade-off
is frequently encountered during algorithm design, when it is possible to reduce the running time of an algorithm
by allocating more memory space.
Exercise 1.1 Consider a function f with two input parameters a and b (a and b are nonnegative integers) defined
by the following equations.
f(0, b) = 0.
f(a, b) = b+f(a1, b) if a > 0.
Give an equivalent nonrecursive definition of the preceding function.
Exercise 1.2 Consider a function f with two input parameters a and b defined by the following equations.
f(0, b) = b
f(a, b) = a+f(a1, b)
f(a, b) = f(b, a)
if a b
if a < b
Compute f(5,2). Does the given function always terminate on input a and b that are nonnegative integers?
Justify your answer.
Exercise 1.3 Let BinaryRep(n) be a function that returns (as a string of 1s and 0s) the binary representation of
a nonnegative integer n. Write base and recursive equations for BinaryRep(n). Hint: Consider the relationship
between BinaryRep(n) and BinaryRep(n/2).
1.1.3 Proof Techniques

Proof techniques are essential tools to prove the correctness of algorithms or claims about their efficiency. In
this section, we review these techniques but do not give any examples.
In logic, a proposition P is a statement (claim or assertion) that is either true or false. Two propositions P and Q
are logically equivalent (or equivalent, for short) if whenever P is true, Q is true, and vices-versa.
Propositions can be combined using logical connectives (and, or, not) to produce compound propositions. For
example, the proposition not P is the negation of P (not P is true if and only if P is false). The proposition P
and Q is true if and only if both P and Q are true. The proposition P or Q is true if and only if P is true or Q
is true.
Most of the time, the statement we are to prove takes the form of an implication, expressed using the English
statement: if P then Q (or P implies Q, which we also write as P Q). An implication is just one more example
of a proposition as argued in the next paragraph, it is equivalent to: (not P) or Q. It consists of two
independent propositions: P is the premise (precondition) and Q is the conclusion. The implication asserts that
we can conclude Q if the premise P is satisfied.
If the implication P Q is true, it guarantees that Q will be true if P is true. In other words, we cannot have P
true and Q false [Note: The implication P Q is considered true (valid) in the case where P is false and Q is
true]. Therefore, the implication P Q is equivalent to: not (P and (not Q)). The latter expression is, by De
Morgans law, equivalent to: (not P) or Q.
A proof is a sequence of derivation steps to establish the validity of a given proposition. A step in a proof often
takes the form of an implication that is based on a definition, an algebraic or a logical identity (such as De
Morgans law), a logical-inference rule, or a previously-established theorem.
Direct Proof
A direct proof is used to prove an implication P Q, by establishing a sequence of implications where the first
starts with P and the last ends with Q, as in the following:
PP1, P1 P2, , Pk-1 Pk, Pk Q.
Indirect Proof
An indirect proof of an implication P Q makes use of the fact that this implication is logically equivalent to
not Q not P; this is so because if P Q is true, we cannot have P and (not Q). Thus, for an indirect proof of
P Q, we start with not Q and then produce a sequence of derivation steps terminating with not P. [Note: The
implication not Q not P is known as the contrapositive of the implication P Q.]
Proof by Contradiction
In a proof by contradiction of P, we assume P is false and then try to derive a contradiction (something that is
false). Assuming that the derivation steps in the proof are valid, it must be the falsity of the starting assumption
that has lead to the contradiction. Thus, we conclude that P is true.
A proof by contradiction can be used to prove stand-alone propositions or implications. To prove a stand-alone
proposition P using a proof by contradiction, we assume not P and then produce a sequence of derivation steps
that leads to a contradiction.
To prove an implication P Q using a proof by contradiction, we would start with the negation of the
implication and then try to derive a contradiction. Thus, the proof starts as follows: Assume P and (not Q).
A contradiction is reached if we derive not P (or Q), because this contradicts part of the starting assumption.
Proof by Induction
Induction is an extremely powerful method of proving results in many areas of mathematics. It is based on the
following principle.
The Induction Principle: Let P(n) be a statement that involves a natural number n (i.e., n=1, 2, ), then P(n) is
true for all n 1 if the following are true:
a. P(1) is true, and
b. For all natural numbers k, P(k) P(k+1).
Induction is unique in comparison with other proof techniques, in that an induction proof is easily translated into
an algorithm. As an algorithm-design technique, induction is very much related to recursion: Expressing the
solution to a problem P of size (input size) n in terms of the solution to the same problem P of size smaller than
n. In this respect, other design techniques (e.g., divide-and-conquer, dynamic programming) can be seen as
variants of induction. Udi Manber2 was so fond of induction that he devoted an entire book to algorithm design
by induction [Man88, Man89]. We cover induction and its relevance to algorithm design in Ch. 3. However,
induction is a recurring theme throughout this book.
Udi Manber is currently a vice president of engineering at Google. He also worked as a senior VP at Amazon. His work on search
algorithms started with the invention of Suffix Arrays (with Gene Myers) in 1989 while he was a professor at the University of Arizona.
He participated in the development of several searching tools, including WebGlimpse and Harvest.
1.2 Problem-Solving Techniques

Problem-solving is a generic term that is normally associated with the thought processes that have governed
human survival through lifes hardships, and to which we attribute the advancement of science. In the context of
information representation and manipulation, problem-solving techniques (strategies) is a synonym for a set of
techniques (i.e., for our purpose, algorithm-design techniques) for representation and manipulation of
computerized information. Each technique reflects a particular pattern of attack that has been found to be
applicable to a broad class of problems. Most books on algorithms, including this one, organize the subject
around a taxonomy of algorithm-design techniques.
At this early stage, it is appropriate to introduce two fundamental, yet powerful, techniques: brute-force search
and problem reduction.
1.2.1 Brute-Force Search
Brute-force search is known by several other names, including exhaustive search, generate-and-test. Mostly we
find the brute-force search strategy applicable to problems that are of the form: among a collection of objects,
find an object that satisfies some given constraints.
To employ brute-force search, we first define a proper data structure to represent (model) the type of object
defined by the problem, and then carry out a process of enumeration (object instantiation) that is capable of
generating all object instances. Instances are generated one instance at a time. An instance is then checked
against the given constraints, and, if found acceptable, it is returned as a solution to the problem. The search
process is terminated when a solution is found, or, in general, when all objects are generated and examined. The
outline of a brute-force search can be expressed with the aid of two helper functions (shown in bold) as follows:
Brute-force-search()
{
Define a data structure to represent an object obj
Let obj be the "first" instance
while (obj null))
{ if Acceptable(obj) return obj; // A solution is found
obj = NextObject(obj)
}
return null; // No solution is found
}
An important part of the preceding process is the generation of object instances. It is necessary that this process
be done by a systematic (organized) search such that all potential instances are generated and no instance is
generated more than once. Generally, this can be done by imposing an ordering relation on the set of objects and
then the objects are generated in that order. This is the essence of the NextObject() function and why it is
dependent on the current object obj; if there are no more objects, the function should return a special object such
as null.
Slot Filling
In cases where an object can be represented as a sequence (vector) of n elements, we can use a process of slot
filling (Note: Slot filling is normally used to implement backtracking, which is a powerful algorithmic technique
that we consider later in Chapter 8). The sequence is represented as an array S[1..n]. Each array component S[i]
represents a slot that can be filled with a value that was not used previously. Let us consider an example.
Example 1.1 Given a positive integer n, print all binary sequences of length n.
A binary sequence of length n can be represented by the 0/1-array S[1..n]. The following is a novice and
erroneous solution.
for i= 1 to n // The index i ranges over positions (slots)
for j = 0 to 1 // Try the 0/1 values for filling slot i
{ S[i] = j; // Fill slot i
if i=n then Print(S);
}
There is a simple reason why the preceding solution is incorrect. There are 2n binary sequences of length n but
the preceding code executes Print at most 2n times (a conclusion based on noting the range of values for the
nested loop) actually, Print is executed only 2 times (Why?). The problem with the preceding code is it fills
slot i with 0 and, before continuing-on to the next slot (i.e., advance i), it immediately overwrites the slot with 1.
There is another key detail missing. There is a need for backward movement that allows us to return to an earlier
position to fill with the next value. The notion of nextness is certainly dependent on the current sequence. For
example, assuming n=4 and given the sequence 1011, what is the next sequence? Well, move backward from the
last position until finding a position where you can go forward; that is, (position underlined) 1011. So, we will
use the next (untried) value at position 2 and go forward from there. This gives the sequence is 1100. For
another example, suppose a sequence is to use the digits {1,2,3,4}, then the sequence following 1234 is 1241.
In short, slot filling can be implemented as follows.
{ int[] S = new int[n+1]; // The slot at position 0 is not used
int i =1; // Start at position 1
S[i] =-1; // Initialize
while (i > 0) // i ranges over positions (slots)
{ while (S[i] < 1 ) // Try the 0/1 values for filling slot i
{ S[i] = S[i]+1; // Next value for a slot is dependent on current value
if (i==n) Print(S);
if (i < n) {i++; S[i] = -1; } // Advance to next slot and initialize
}
i--; // move back
}
}
Note that in the preceding code, we are setting S[i] = 1 at two places to provide the proper initialization for the
forward movement. For backward movement, every time we move backward one position, we immediately try
to go forward, because we are reentering the inner while-loop. At that time, we test if we have exhausted all
possible values at the position we are now at, and if so, we move back one more step. If we move back all the
way to an invalid position (i=0), we exit the outer while-loop.
Example 1.2 Compute the number of binary strings of length n that do not contain consecutive 1s.
We present three different solutions. The first two solutions use brute-force search to enumerate all of the 2n
binary vectors of length n. Therefore, they are usable for small values of n. For the third solution, we derive a
recurrence relation (equation) R for the number of binary strings of length n satisfying the conditions of the
problem, which we then express as a recursive program function.
A binary string of length n can be represented by the 0/1-array S[1..n]. Using brute-force search, we generate all
binary strings of length n, one at a time, and we test each string for acceptability using the following function:
bool Acceptable(int[] S, int n)
{ // returns false if S[1..n] contains consecutive 1s
for(int i = 2; i <=n; i++)
if ((S[i]==S[i-1]) && (S[i]==1)) return false;
return true;
}
To generate binary strings of a specified length, we can use slot filling (Solution 1) or some other simpler
technique (Solution 2).
Solution 1
int Count_V1(int n)
{ // returns the number of binary strings of length n
// that do not contain consecutive 1s
int[] S = new int[n+1];
int count=0;
int i =1;
S[i] =-1; // i ranges over positions (slots)
while (i > 0)
{ while (S[i] < 1) // Try the 0/1 values for filling slot i
{ S[i] = S[i]+1;
if ((i==n) && Acceptable(S,n)) count++;
if (i < n) { i++; S[i] = -1; } // Advance to next slot & initialize
}
i--; // move back
}
return count;
}
Solution 2
We recognize that binary strings of length n are in one-to-one-correspondence with the integers in [0,2n 1].
Thus, we can write the following method as a solution to our problem this works for n 63 (we can left-shift
a 1 up to 63 positions because the ulong type in C# uses 64 bits):
int Count_V2(int n)
{ // returns the number of binary strings of length n that
// do not contain consecutive 1s
int count = 0;
ulong MaxVal = ((ulong)1 << n) - 1; // (2^n)-1 with n as large as 63
for(ulong i=0; i < MaxVal; i++)
{ int[] S = ToBinary(i,n); // Convert i to an n-bit number S[1..n]
if (Acceptable(S, n)) count++;
}
return count;
}
To get the binary representation (as an n-bit number) of an integer x, we can rely on repeated application of
modulo-2 and division-by-2 operations to extract the successive bits of x as in the following:
int[] ToBinary(ulong x, int n)
{ // returns the binary representation of x as an n-bit number S[1..n];
// the least significant bit is S[1]
int[] S = new int[n+1];
int i = 1;
while (x > 0)
{ S[i] = (int) (x % 2);
x = x / 2; // integer division by 2
i++;
}
while (i <= n) { S[i] = 0; i++; }
return S;
}
Note: In .NET, the Convert class provides a static ToString() method that can be used to get the bit
representation of a given integer. One of the methods overloads accepts two integer parameters and returns a
string containing the specified integer in the specified base. For example, the expression
Convert.ToString(11,2) returns the string "1011".
Solution 3
Slot filling can be used to derive a recurrence equation for the number of binary strings of length n that do not
contain consecutive 1s.
(a)
Binary strings of length n1 with no consecutive 1s
(b)
Binary strings of length n2 with no consecutive 1s
0
0
Figure 1.2 Binary strings of length n with no consecutive 1s are (a) those ending with 0 and (b) those ending with 1.
Let Rn denote the number of binary strings of length n that do not contain consecutive 1s. Let us try to relate Rn
to Rn-1 (or, in general, to Rk where k < n). Consider the two cases of filling the n-th slot, as illustrated by Figure
1.2. If this slot is filled with 0, the first n1 slots must not contain consecutive 1s; by definition, the number of
such strings is Rn-1. On the other hand, if the n-th slot is filled with 1, the (n1)-th slot can only be filled with 0,
and the first n2 slots must not contain consecutive 1s. Thus, the count in this case is given by
Rn-2. Therefore, we conclude that Rn = Rn-1+Rn-2 recall the sum rule, |AB| = |A|+|B||AB|, where in this case
AB = . We need two base cases (Why?). R1=2, because both binary strings of length 1 are acceptable. Also,
R2 = 3, because we are to count the strings 00, 01 and 10. Based on the preceding analysis, we can write the
following recursive method to compute Rn:
int R(int n)
{ if (n == 1) return 2;
else if (n == 2) return 3;
else return R(n-1)+R(n-2);
}
Caution: The preceding method will take a long time (days) to complete for n 30. This is caused by the
presence of the two recursive calls R(n1) and R(n2), which leads to an excessive number of procedure calls;
certain calls with the same input arguments will be encountered more than once (for example, in computing
R(4), R(2) is called twice). The problem can be resolved by converting recursion into iteration (See Problem 1
at the end of this chapter solved exercises).
Recursive Slot Filling
If you think deeply about Solution 3, you might be pondering the following thought: Cant we use the
argument used to derive a recurrence equation for the number of solutions to generate the solutions
themselves? This is indeed the case, and we state it as a principle.
The Enumeration Principle: A recurrence equation for the number of solutions can be used to
generate the solutions themselves.
Moreover, this approach, because it generates the solutions that satisfy the constraints and no other solutions, is
expected to be more efficient than the approach that generates a larger set of solutions and filters. For the
program code, we simply encode slot filling as a recursive method mimicking the process illustrated in Figure
1.2. We will assume that the solution vector is given by a global (static) integer array S[1..n]. Thus, we can do
with a single input parameter n that denotes the position of the current slot to be filled. To generate binary
strings of length n, we simply execute the call SlotFill(n). Filling starts from the n-th slot downward until we
reach the first slot. At that time we have a completed solution vector, which we print.
// Initial call is SlotFill(n), where n is the length of binary strings
// Uses a global (static) array S[1..n]
void SlotFill(int n)
{ if (n == 1) { S[1]=0; Print(S); S[1]=1; Print(S); }
else if (n == 2)
{ S[1]=0; S[2]=0; Print(S);
S[1]=0; S[2]=1; Print(S);
S[1]=1; S[2]=0; Print(S);
}
else
{ S[n]=0; SlotFill(n-1);
S[n]=1; S[n-1]=0; SlotFill(n-2);
}
}
Note: For another example of using the previous technique, consider the problem of generating subsets of size r
chosen from the set {1, 2, , n}. It is given as a solved problem (Problem 2) at the end of Ch. 5.
Exercise 1.4 Rewrite Count_V2() to get rid of the call to ToBinary() and use, instead, a modified version of
Acceptable(). The modified version is passed the integer i and should test for violation by testing the bits as they
are generated.
Exercise 1.5 Let C(n,k) denote the count of n-digit (using decimal digits 0-9) numbers whose sum of digits = k.
a. Write an algorithm that utilizes the technique of slot filling to compute C(n,k).
b. Derive recurrence equations for C(n,k), and use it to verify the solution found in part a.
1.2.2 Problem Reduction
Problem reduction is a problem-solving technique that tries to relate one problem to another. Suppose we are
given some problem A to solve. Rather than solving A directly, we invoke (call) an algorithm that solves another
problem B (B-algorithm). However, because the input (and output) specifications for these two problems are
likely to differ, we have to undertake some kind of input (and output) conversion. Thus, we essentially get an
algorithm for A (A-algorithm) that is like the following:
A-algorithm(Ain, Aout)
{ Bin Ain; // Transform the input of A (Ain) into an input for B (Bin)
B-algorithm(Bin, Bout); // Call B-algorithm with input Bin and get output Bout
Aout Bout; // Transform the output Bout into an output Aout
}
The process of reducing problem A to problem B is denoted by AB and is depicted graphically in Figure 1.3.
In this process, B-algorithm is a helper algorithm (we also use the phrases: helper routine, helper function) and
used as a black box (locked box); that is, we do not care how it works; we only care about its input-output
specification (what it does).
Ain
Bin
B-algorithm
Bout
Aout
Figure 1.3 Solving a problem A by problem reduction; AB: Problem A is reduced to Problem B.
Example 1.3 Suppose we have an algorithm SplitEvenOdd(A) that takes as input an array A of positive integers
and arranges the elements of A such that all even values appear before the odd values. Show how to use this
algorithm to develop another algorithm, which we call SplitNegPos(A), that takes an array A of integers (positive
and negative) and arranges the elements of A such that all negative elements appear before positive elements.
We should somehow map the elements of the array A that is input to SplitNegPos() as follows: Map negative
elements to even positive numbers and map positive elements to odd positive numbers. Then call
SplitEvenOdd(), and finally, do the inverse mapping (even to negative and odd to positive). We need a simple
reversible (invertible) mapping.
Multiply-by-two is a sure way to get even numbers and multiply-by-two-add-one is a sure way to get odd
numbers, but in both cases, we need the result to be positive. So our first attempt for a mapping is:
f(x) = 2 * abs(x)
f(x) = 2x + 1
if x < 0
if x > 0
// map a negative integer to even positive

// map a positive integer to odd positive
Is this function invertible? A function f is invertible if there does not exist two distinct elements a and b such that
f(a) = f(b).
The given function maps any two distinct negative numbers into distinct even numbers, maps any two distinct
positive numbers into distinct odd numbers. In addition, no two numbers, one negative and one positive, map to
the same value, because negative numbers map to even and positive numbers map to odd.
Using this mapping, we can write the following SplitNegPos() algorithm:
SplitNegPos(int[] A)
{ // Transform A into proper input for SplitEvenOdd
for i = 1 to n
if A[i] < 0 then A[i] = 2*Abs(A[i]);
else A[i] = 2*A[i]+1;
SplitEvenOdd(A); // Invoke SplitEvenOdd()
// Do inverse transformation
for i = 1 to n
if A[i] is even then A[i] = - A[i]/2;
else A[i] = (A[i]-1)/2;
}
In general, a reduction AB is not limited to calling B-algorithm once. In fact, we can enlist the help of several
algorithms and call them in any combination. Thus, problem reduction can be seen as an application of the
top-down design principle with one difference; problem reduction, because its purpose is to relate one problem
to another, does not bother to specify the implementation details of helper algorithms.
Question: What do you call (one-word answer) a reduction from a problem to itself?
Example 1.4 Suppose we have an algorithm CountLess(A,i) that takes as input an integer array A and an integer i
and returns a count of the elements in A that are less than i. Show how to use this algorithm to compute for any
given integer array A and two integers a and b, the number of elements that lie between a and b inclusive.
Assume a < b. The count of elements that lie between a and b inclusive can be found by counting the elements
less-than a and counting the elements less-than-or-equal-to b and taking the difference. Thus, we can express the
CountBetween() algorithm in terms of CountLess(), as follows:
CountBetween(A,a,b) = CountLess(A,b+1) CountLess(A,a).
Why Problem Reduction Is Important?
Problem reduction exhibits several important properties. First, the technique becomes more valuable as ones
knowledge about problems and algorithms grows, because this gives more choices for helper algorithms.
Second, problem reduction is useful in determining problem complexity (or, loosely speaking, difficulty level).
In general, when a problem A is reducible to another problem B, we can claim something about the difficulty of
problem A (assuming the input-output conversion is easy): A is not any more difficult than B (equivalently, B is
at least as hard as A). So, we might as well concentrate our effort in solving problem B. If we ever find an
algorithm for B, the AB reduction becomes a real (usable) algorithm for A. Finally, problem reduction is
transitive; that is, given the reductions AB and BC then, by composition of these reductions, we can obtain
an AC reduction. Problem reduction is an important tool for the theory of NP-completeness and will be
revisited in Chapter 9.
Exercise 1.6 Show how the multiplication of two arbitrary-size integers x and y can be expressed in terms of
(x+y)2 and (xy)2. Based on this, can you draw any conclusion regarding the relative difficulty of multiplication
to squaring.
1.3 Running-Time and Space Complexity Analysis

How do you measure the running time of an algorithm for the purpose of comparing the running times of
different algorithms? Using the actual CPU time is not a convenient measure because it is dependent on many
factors. Some factors have to do with hardware such as processor architecture, processor clock speed, etc. Other
factors, that have to do with software, include the programming language and compiler used to program the
algorithm. For these reasons, it is better to seek a technology-independent measure of the running time. The
measure is simply to count the number of elementary operations.
An elementary operation is an operation whose execution time is constant independent of the size (in bits) of the
data manipulated by the operation. Roughly, an operation that maps to a machine instruction is considered an
elementary operation, because normally a machine instruction takes a fixed number of CPU clock cycles. Also,
for the purpose of counting elementary operations, any fixed number of elementary operations executed together
as a group can be considered as a single elementary operation.
Examples of Elementary Operations:
Arithmetic and logical operations on primitive data types.

Comparisons of two integer or floating-point numbers.
Program flow-control operations (e.g., if, goto).
Examples of Nonelementary Operations:
Computing the maximum value in an array of numbers.

Arithmetic operations on numbers of varying bit size (i.e., numbers are allowed to be arbitrarily large).
Concatenation of two strings, because it usually involves creating a new string containing a copy of the
characters of the original strings.
To illustrate the process of analyzing the running time of an algorithm, let us consider an example.
The Fixed-Sum Pair Problem. Given a sequence of n nonnegative integers a1, a2, , an and a positive integer K,
determine if there are two elements whose sum is K.

We give an algorithm based on brute-force search.
Algorithm Idea: Systematically enumerate element pairs and test the sum of each pair.
Listing 1.1 gives the corresponding algorithm. It generates all (ai, aj) pairs where 1 i < n and i < j n.
Input: An integer array A[1..n] and a positive integer K

Output: True if there are two elements that sum to K; False, otherwise.
bool FixedSum(int[] A, int n, int K)
{ for i = 1 to n-1
for j = i+1 to n
{ if (A[i] + A[j] == K) return true; }
return false;
}
Listing 1.1 An algorithm for the fixed-fum pair problem.
A rough, yet quite acceptable as explained later, running-time analysis of the preceding algorithm might count
the number of element pairs tested. It is possible that the first pair tested sums to K, and, thus, the procedure
returns immediately. Thus for this algorithm, the best-case (smallest) running time would be a constant value
independent of the number of elements in the input sequence. However, the worst-case (largest) running time
happens when we have an input that forces testing all possible element pairs (e.g., when there is no pair that
n
sums to K). In this case, the algorithm will have to test = n(n1)/2 = n2/2 n/2 (orderwise approximation)
2
n2 pairs.
Note: A polynomial expression in n like aknk +ak-1nk-1+...+a1n+a0, where ak is positive, can be orderwise
approximated as nk, because the highest-degree term dominates other terms as n grows large. See Theorem 1.1.
Let T(n) denote the running time as a function of the number of input elements n. Based on the number of pairs
generated, we conclude that the running times for best and worst cases can be stated as follows (where the
constant C is some positive real number, and can be different for the different equations):
Best-Case Running Time:
Worst-Case Running Time:
T(n) = C
T(n) = Cn2
1.3
1.4
Next, we turn our attention to a detailed worst-case analysis of the running time based on counting all
elementary operations, as detailed in Table 1.2.
Step
Program Statement
Elementary Operation
for i = 1 to n-1
for j = i+1 to n
Operation Cost
Total Cost of all Operations
Set (or increment) i and test i n1
C1
(n1)* C1
Set (or increment) j and test j n
C2
Sum A[i], A[j] and test if sum = 0
C3
C3
Return true
C4
0 or C4
Return false
C5
0 or C5
n 1
i =1
j = i +1
n 1
{ if (A[i]+A[j] = K)
4
5
return true }
return false
i =1 j = i +1
Table 1.2 A detailed worst-case analysis of running time.

* Note: The loop is entered n1 times but the loop index test is executed n times.
The for-statement involves setting (or incrementing) the loop index and testing it against the exit condition (for
simplicity, we view this combined operation of setting and testing as a single elementary operation). Such an
operation is executed as many times as there are loop iterations.
n1
The double-sum
C2
can be rewritten as (the inner sum
i =1 j =i +1
j =i +1
n 1
C2
n i = (n1+n2++1) C
= [n(n1)/2] C2.
i =1
Thus, the total number of operations is as follows:
= C2 1 = C2(ni) ):
j =i +1
[n1]C1 + [n(n1)/2]C2 + [n(n1)/2]C3+0+ C5.

Using the dominating terms (i.e., the ones involving n(n1)) as an approximation, we get (by letting C=C2+C3)
the following:
(C2+C3) [n(n1)/2] = C [n(n1)/2] = C [n2/2 n/2] Cn2
Thus, based on counting all elementary operations, we conclude the following:
Worst-Case Running Time: T(n) = Cn2
1.5
This is exactly the result obtained earlier in Equation 1.4. Is this a mere coincidence? Actually, it is not. We note
that in the program code, the operation in step 2 (or step 3) is executed more than (or as often as) any other
elementary operation. The rough analysis that we did earlier in essence counted the operation corresponding to
step 3 (without bothering about the actual code). This leads to the following important observation.
Running-time analysis by counting dominant elementary operations
To analyze the running time of an algorithm, instead of considering and counting all elementary
operations, it suffices to identify one dominant elementary operation and count it. A dominant elementary
operation for an algorithm (or a program-code block) is an elementary operation that is executed more
than (or as often as) any other elementary operation in the algorithm (code block). Also, the count is
expressed in terms of n where n is the size of the input.
In summary, analyzing the running time of an algorithm by counting all elementary operations is quite
cumbersome. Instead, the analysis of the running time is normally done by executing the following steps, in
sequence:
1. Analyze the algorithm in blocks (normally, a block can be a loop, a doubly-nested loop, a procedure
call, etc.). For each code block, identify a dominant elementary operation. For example, as we will see
later, for searching and sorting algorithms, a comparison involving input data elements is a dominant
elementary operation.
2. For each block, derive a formula (algebraic expression) f(n) for the number of times the dominant
operation is executed as a function of the input size n. The input size is the amount of memory space (in
bits) used for the algorithms input parameterized by a proper measure n. For example, for an algorithm
whose input is an integer sequence (i.e. an array of integers), the input size n can be taken to be the
length of the sequence.
3. Find the overall running time T(n) by adding f(n)s. Simplify and express T(n) using O-notation
(covered in the next section).
For running-time analysis, we distinguish among three cases. Let RT(x) denote the running time of an algorithm
on input x, expressed as a count of elementary or dominant operations.
Worst-Case Time Complexity
The worst-case running time of an algorithm is a function T mapping the input size n to the largest running time
for any input x of size n. Thus, the worst-case running time is, T(n) = maximum {RT(x), |x| = n}.
Best-Case Time Complexity
The best-case running time of an algorithm is a function T mapping the input size n to the smallest running time
for any input x of size n. Thus, T(n) = minimum {RT(x), |x| = n}.
Average-Case Time Complexity
The average-case (expected) running time is the running time averaged over all possible inputs x of size n. Thus,
T(n) = average {RT(x), |x| = n}. The average is computed by varying the input from best case to worst case and
cases in between, summing and averaging the work of all cases. This assumes all cases are equally likely.
However, the average is more accurate if statistical average is used, where each case is weighted by its
probability of occurrence. Therefore, the average-case time complexity is computed in accordance with
Equation 1.6.
Average-Case Running Time =
(probability of case
i occuring) * (number of operations for case i )
1.6
all input cases i of size n
1.3.1 Asymptotic Notations
Given a function f(n) that represents the running time of an algorithm normally, f(n) specifies an expression
for the number of times some dominant elementary operation is executed in terms of input size n we would
like to compare and classify different f(n)s. For instance, how does f1(n)=n2 compare to f2(n)=2n2+10n? In
general, for any given f(n), we would like to come up with another function g(n), where g(n) is both a simple
expression in n, and an approximate upper bound for f(n). The function g(n) represents the order of growth for
f(n). If f(n) represents the running time, then g(n) represents the order of running time. This is what the Onotation is all about.
Important Note: For any function f(n) that we consider in the context of running-time (or space) analysis, we
assume that n (the domain of f(n)) represents the input size (n is an integer 0). Furthermore, f(n) is
a monotonically increasing function (that is, for any a, b in the domain of f, f(a) f(b) if a b). Informally, this
says that the running time (or space) increases as the input size increases.
Definition. O-Notation, O(g(n)): A function f(n) is O(g(n)) if and only if f(n) Cg(n) for some positive real
constant C and all integers n n0 (n0 is some fixed positive integer).
Note: O(g(n)) is pronounced as Oh gee of n. The O-notation as defined here is also known as Big-O.
Because the inequality in the preceding definition needs only to be satisfied for large n, we say that g(n) is an
asymptotic bound. As a general rule, when we do the running-time (or space) analysis for some algorithm, we
are primarily interested in the asymptotic growth of the running time (space); that is, the running time (space)
where the input size can grow unbounded.
It is important to note that the upper bound g(n) is not a strict upper bound but can be smaller (in magnitude)
than f(n). All we care about is that g(n) is within a multiplicative constant factor of f(n). For example, g(n)=n2 is
an upper bound for f(n)=2n2.
Example 1.5 f(n) = 2n2 is O(n2) because f(n) = 2n2 2n2 Cn2, for C=2 and for all n 1.
Example 1.6 Let f(n) = 2n2 +3n. Is f(n) O(n2)?
Solution: Yes, because f(n) = 2n2 +3n 2n2 +3n2 5n2 Cn2, for C=5 and for all n 1.
Theorem 1.1 The function f(n)=aknk+ak-1nk-1+ ...+a1n+a0 (assume that the ais are real numbers and ak > 0) is
O(nk).
Proof: We have to show that f(n) Cnk for some positive constant C.
If we let aMax = maximum of {ak, ak-1, ..., a0}, then

f(n) aMax nk + aMax nk-1 + ... + aMax n + aMax
aMax nk + aMax nk + ... + aMax nk + aMax nk
= aMax (k+1) nk.
Now, if we let C = aMax (k+1), then f(n) Cnk for all n 1.
Example 1.7 Let f(n) = 2n2. Show why f(n) is not O(n).
If f(n) is O(n), then we must have, f(n) Cn for some constant C and all n n0. Thus, 2n2 Cn 2n C
C 2n, which is impossible because C is a constant but n can grow unbounded. We conclude that f(n) is not
O(n).
The Bad Side and Good Side of the O-notation
The O-notation suffers from one problem. It allows for loose upper bounds. For example, f(n) = 2n2 is O(n2)
and is also O(n3), O(n4), and so on. However, as we saw from the previous example, the O-notation does not
allow for the wrong upper bound. There is a way to force the selection of a tight upper bound, which we will
consider after we introduce the -notation.
Definition. Omega-Notation, (g(n)): A function f(n) is (g(n)) if and only if f(n) Cg(n) for some positive real
constant C and all integers n n0 (n0 is some fixed positive integer).
Note: (g(n)) is pronounced as Omega gee of n.
Example 1.8 Let f(n) = 2n2. Then f(n) is (n2) because f(n) = 2n2 n2 Cn2 for C=1 and for all n 1.
Example 1.9 Let f(n) = 2n2. Is f(n) (n)?
Solution: Yes, because f(n) = 2n2 n Cn for C=1 and for all n 1.
Example 1.10 Let f(n) = 2n2. Is f(n) (n3)?
Solution: If f(n) is (n3), then f(n) = 2n2 Cn3 C 2/n. This means that the larger the n, the smaller the
constant C, but C is supposed to be a constant independent of n. Thus, we concluded that f(n) is not (n3).
The Bad Side and Good Side of the -notation
As can be seen from the previous examples, the -notation suffers from one problem. It allows for a loose lower
bound. However, the -notation does not allow for the wrong lower bound.
Because, for a given f(n), our goal is to have g(n) a tight bound for f(n), we should choose g(n) such that f(n) is
both O(g(n)) and (g(n)). This leads to the following tight bound notation.
Definition. Theta-Notation, (g(n)): A function f(n) is (g(n)) if and only if f(n) is O(g(n)) and f(n) is (g(n)).
Note: (g(n)) is pronounced as Theta gee of n.
It is easy to see that if f(n) is (g(n)) then c1 g(n) f(n) c2 g(n) for some positive real constants c1 and c2 and
for all n > some positive integer n0. Thus g(n) is a tight bound for f(n).
Example 1.11 Let f(n) = n2 10n. Show that f(n) is (n2).
Solution: We need to show that f(n) is O(n2) and f(n) is (n2). First, f(n)=n2 10n n2 Cn2 for C=1 and for all n
1. Thus, f(n) is O(n2). Second, we must show that f(n)=n2 10n Cn2 for some positive constant C. This is
only possible if C < 1. Thus, we try to find n0 the value of n for which n2 10n (1/2) n2. The latter
inequality implies n 10 (1/2) n n /2 5 n 10. We conclude that n2 10n Cn2 for C=1/2 and for all n
10. Thus, f(n) is (n2).
Example 1.12 Show that if f(n)=af'(n), where a is a positive constant, then f(n) is (f'(n)).
Solution: We show that f(n) is O(f'(n)) and f(n) is (f'(n)). First, f(n) = af'(n) Cf'(n) for C=a and for all n 1.
Thus, f(n) is O(f'(n)). Second, f(n) = af'(n) Cf'(n) for C=a and for all n 1. Thus, f(n) is ( f'(n)).
The preceding example shows that, as a first step in finding the order for f(n) using O-notation (and its
derivatives), we can drop any constant factor of the (whole) expression specified by f(n). For example, f1(n) =
106 n log n and f2(n) = 10-2 n log n are both (n log n).
Note: Throughout the text, we use log n for log2 n. Also, with equations/inequalities, we use the contructions
LHS [RHS] for left-hand side [right-hand side] (in relation to the equality/inequality symbol).
Example 1.13 Show that log n! is (n log n).

Solution:
log n! = log (n(n1) 1)

= log n + log (n1) + + log 1
log n + log n + + log n = n log n. Thus, log n! is O(n log n).
On the other hand, to show that log n! = (n log n),
log n! = log (n(n1) 1) = log n + log (n1) + + log 1.
Now, for the RHS, taking the first n/2 terms only and replacing each term by log n/2 would give a value smaller
than RHS. Thus, the preceding equation can be rewritten as the inequality:
log n! (n/2) log n/2 = (n/2) log n (n/2) log 2 = n/2 log n n/2
(Because (n/4) log n > n/2 for all n > 16, we can replace n/2 in the previous expression by (n/4) log n and get a
smaller expression)
(n/2) log n (n/4) log n = (n/4) log n.
Using Integration for Finding the Order of Summations
Let f(x) be a continuous function that is monotonically decreasing or increasing, and we are to compute the
n
summation
f (i). We can get a lower and upper bounds by approximating the summation using integration as
i =1
follows.
If f(x) is decreasing, then we have
n +1
f ( x) dx
f (i )
i =m
f ( x) dx.
m 1
If f(x) is increasing, then we have

n
f ( x) dx
n +1
f (i )
i =m
m 1
f ( x) dx.
Example 1.14 In this example we derive lower and upper bounds for the Harmonic series
1 1
1
H n = + + ... + .
n
1 2
y=1/x
y=1/x
n+1
(a)
(b)
Figure 1.4 Approximation of the Harmonic series H n =
1
is decreasing, we conclude that
x
n
n +1 dx
n dx
1
.
0 x
1
x
i =1 i
Because f ( x) =
i
i =1
The inequality to the left is clearly seen from Figure 1.4(a). Thus,
n
i
i =1
n +1
dx
= ln (n + 1).
x
The inequality to the right gives an upper bound of infinity, but it can be manipulated to give a tighter bound as
follows. From Figure 1.4(b), we can see that
n
1
=1+
i =1 i
i 1+
i =2
dx
= 1 + ln n.
x
From the lower and upper bounds computed previously, it follows that
n
1
ln (n + 1)
1 + ln n.
i =1 i
Hence, by definition of the notation, we have Hn = (ln n) = (log n) (because ln n =
log n
).
log e
Which Notation (O, , ) is the Appropriate to Use?
Generally, you can think of as a synonym for at least, O as a synonym for at most, and as a sysnom for
equal (orderwise, of course). For example, the running time of an algorithm is always at least as the size of its
output (Why?). Consequently, for an output of size n, the running time is (n). Big-O is normally used for
expressions that are overestimation. For example, suppose you are analyzing the running time of an algorithm
by counting its elementary (or dominant) operations, and it occurs to you that it would be easier to get a closed
formula if you added extra terms or enlarged others, then the derived formula is valid as an O-expression. On
the other hand, if your formula was an underestimation because you dropped some terms (or ignored essential
parts of the algorithm), then use your formula as an -expression. Finally, if the O-expression you have is a
tight one (either because you have counted all operations that affect the order or only ignored parts that are of a
lower order than what you have), then you might as well use your formula as a -expression.
Caution
Whenever f(n) is O(g(n)), we often write this as f(n) = O(g(n)). However, equality in expressions involving Onotations (and its derivatives) must not be treated as having transitivity property like equality on numbers. For
example, f(n)=2n is O(2n-1) because 2n C2n-1 (for C=2). Similarly, 2n-1 is O(2n-2). Continuing this way and
using transitivity, we will conclude that f(n)=O(1), which is absurd.
1.3.2 Classification of Algorithms Based on Order of Running Time
Usually, an algorithm is classified based on its worst-case running time using the O-notation (using an upper
bound that is as tight as possible). For example, sequential search is O(n), binary search is O(log n). For sorting,
we find that selection sort, insertion sort and bubble sort are all O(n2), but Mergesort and Heapsort are O(n log
n). There are algorithms whose running times grow exponentially like O(2n) or O(n!). Such algorithms will run
in a reasonable time only if the input size is small. An example of a problem in this class is the generation of all
subsets of an n-element set. In this case, there are 2n subsets. For n=50, 2n 1015. Thus, even with a computer
that is capable of generating one subset per nanosecond (10-9 second), generating all subsets would take 106
seconds = 278 hours = 11.6 days.
Figure 1.5 contrasts the growth for different orders of complexity. Table 1.3 contrasts the running times (and
problem sizes) that can be solved for the different orders of running time. Example 1.15 shows the calculation
for some entries but it is left to the reader to verify the rest.
f(n)
f(n) = 2n
f(n) = n2
f(n) = n log n
f(n) = n
f(n) = log n
n
Figure 1.5 Contrasting function growth for different orders of complexity.
Order of
Running
Time
Time to solve a problem of size n
Largest problem-size n solved in
n = 1000
n = 1 Million
n = 1 Billion
1 Hour
1 Day
O(log n)
0.01 millisec
0.02 millisec
0.03 millisec
23600*1000000
224*3600*1000000
O(n)
1 millisec
1 sec
103 sec
3600*106 3.6 billion
24*3.6 86.4 billion
O(n log n)
10 millisec
20 sec
30000 sec 8.3 hrs 108
O(n2)
1 sec
106 sec
278 hrs
1012 sec 317 yrs
sqrt(3600*106)
60,000
sqrt(24*3600*106)
294,000
O(2n)
very huge
very huge
very huge
log (3600*06) 32
log (24*3600*106) 37
5*108
Table 1.3 Comparison of different orders of running time; an elementary operation is assumed to take 1 microsecond.
Example 1.15 Compute the approximate running time in seconds taken by an O(n log n)-algorithm on an input of
size n=1 million when the algorithm is run on a computer capable of executing 106 operations per second. What
is the maximum input size that can be handled by this algorithm in one hour?
Solution: For an input of size n, an O(n log n)-algorithm executes on the order of n log n operations the
running time in seconds = n log n operations (operation time in seconds). In this case, the running time = 106
log 106(10-6) = log 106 = 20 seconds. For the maximum input-size n that can be handled in one hour, 1 hour =
3600 seconds = (n log n)(10-6) 36108 = n log n. Try to view 36108 as n log n (if 36 is close to log 108,
then n=108). Because log 108 = log 102 + log 106 7+20 36. Thus, n=108. Note that 109 will be an
overestimation of n because 109log 109 >> 36108.
Example 1.16 Fill in the blanks.
(a) With a (log n) algorithm, the running time doubles when the input size increases from n to ___________.
(b) With a (n2) algorithm, the running time doubles when the input size increases from n to _____________.
Solution: (a) n2 (b) 2 n.
Let T(n) denote the running time for an input of size n. We are to find n1 (in terms of n) where T(n1) = 2T(n).
For (a), T(n) = c log n. T(n1) = 2T(n) = 2 c log n = c log n2, but T(n1) = c log n1; thus, log n1 = log n2 n1= n2.
For (b), T(n) = cn2. T(n1) = 2T(n) = 2cn2, but T(n1) = cn12; thus, n12 = 2n2 n1 = 2 n.
An Ordering Relation on Big-O
We can define a total-order relation () on O-expressions, as follows.

O(f1(n)) O(f2(n)) if f1(n) is O(f2(n)).
This definition is consistent with the data of Table 1.3 in that the larger the order, the larger the running time.
Using our ordering relation, here is an ordering of commonly occurring orders:
O(1) O(log n) O( n ) O(n) O(n log n) O(n2) O(n3) O(2n) O(n!).
Table 1.4 describes the general characteristics for algorithms that exhibit certain orders of running time.
Note: The order of a function f(n) is very much determined by the dominating term as n grows large. Two
functions will have different orders if their dominating terms differ by a factor that is dependent on n. For
example, f(n) = n log n and g(n) = n log n log n have different orders. In this case, f(n) has a lower order and f(n)
cannot be an upper bound for g(n). This is easily seen, because if g(n) is O(f(n)) then there exists some positive
constant C such that (n log n log n) C(n log n) log n C. Thus, C must grow with n, which is a
contradiction.
Order of
Running
Time
Where It Occurs
O(1)
O(1) is a constant running time. This occurs whenever the algorithm executes a fixed number of
operations independent of the size of input such as returning the first (or last) element in an array.
O(log n)
O(log n) is a logarithmic running time. This occurs in an algorithm that performs a fixed number
of elementary operations and discards half of the input. Then the process is repeated recursively.
This is the case for binary search for some element in a sorted sequence of n elements. Note that
log n grows only by a constant when n doubles, and does not double until n increases to n2.
O(n)
O(n log n)
O(n) is a linear running time. This normally happens when it is the case that each input element is
processed (using a fixed number of elementary operations) once. Whenever n doubles, the running
time doubles. If a problem calls for processing n input items or producing n output items then an
O(n)-algorithm is optimal (under the assumption that items can only be processed one at a time).
O(n log n) running time occurs when an algorithm solves a problem by breaking it up into two
subproblems, solving them independently, and then combining the solutions, as in the Mergesort
algorithm. Whenever n doubles, the running time increases to slightly more than doubles.
O(n2) is a quadratic running time. This typically occurs in algorithms that process all pairs of data
items (e.g., in a doubly nested loop). Whenever n doubles, the running time increases fourfold. A
quadratic algorithm is considered impractical for large-size input.
O(n )
O(n3) is a cubic running time. This arises for an algorithm that processes triples of data items (e.g.,
in a triply nested loop). Whenever n doubles, the running time increases eightfold. Such an
algorithm is practical only for small-size problems.
O(2n)
A good example of this is generating all possible subsets of n elements.
O(n!)
A good example of this is generating all possible permutations of n elements.
O(n )
Table 1.4 Orders of running time and corresponding algorithms.
Problem Complexity: Easy Problems versus Difficult Problems
For the generation of subsets, it is quite expected that no polynomial-time algorithm is foreseeable, because the
amount of output that has to be produced is exponential in n. However, there are problems that call for a tiny
amount of output (such as a Yes/No-answer), and yet we do not know of any polynomial-time algorithms for
them. From an algorithmic perspective, a problem can be classified as either easy (tractable) or difficult
(intractable). Any problem that is solvable using a polynomial-time algorithm is classified as an easy problem.
On the other hand, any problem for which there is no known polynomial-time algorithm, is normally classified
as a difficult problem. Note that the use of the word difficult here does not mean the problem is hard to solve;
rather, it means that finding a solution takes a long time even for moderate-size input. Many of the difficult
problems come from graph theory such as determining the existence of a Hamiltonian circuit (a path that starts
at some vertex, passes through each vertex once and returns to the start vertex) in a given directed graph. A
common pattern to many of the difficult problems is their combinatorial nature. To find the right solution,
somehow we are forced to examine all combinations (possibilities).
1.3.3 Space Complexity Analysis
The space complexity of an algorithm indicates the amount of memory storage (memory space) used by the
algorithm. By convention, the space complexity does not count the storage for the input or output. This is so
because the space complexity is used to compare different algorithms for the same problem (i.e., both the input
and output are fixed for a given problem). In addition, the space complexity does not count the storage required
for the program code because it is independent of the size of the input.
Like time complexity, space complexity is most often specified for the worst case and given as an asymptotic
expression in the size of the input. Thus, an O(1)-space algorithm requires a constant amount of memory space
independent of the size of the input. An O(n)-space algorithm uses an amount of memory space linear to the size
of the input n.
In practice, the running time of an algorithm is always emphasized over its memory requirement. This can be
seen from two angles. First, the running time is more or less human time, whereas computer memory is mostly
affordable. Second, if we let T(n) and S(n), respectively, denote the time and space complexity of an algorithm
in terms of its input size n, then T(n) S(n) let us refer to this as the time-space inequality. The time-space
inequality implies that any reduction in time is likely to yield a reduction in space. The justification for the given
inequality comes from the fact that an algorithm that allocates memory would normally spend time to access the
allocated memory at least once [Question: Is the first time a memory cell is accessed a read or a write?]. We
expect that only a nonuseful program would violate the time-space inequality. For example, the program
consisting of the single instruction "int[] A = new int[n]" is O(n) space but O(1) time.
Exercise 1.7 Show that if f(n) is O(g(n)) then g(n) is (f(n)).
Exercise 1.8 Let f(n) = n3 100n2. Show that f(n) is (n3).
Exercise 1.9 An O(n2)-algorithm takes 1 minute on some given input, how much it would take on the same input
if the processor clock speed is doubled?

Exercise 1.10 Order the following functions in increasing asymptotic growth order.
f1(n) = (n2 log n) 100, f2(n) = (n log3 n) + 12n, f3(n) = (5n/6)2.
1.4 Searching
Searching and sorting are fundamental and routine processes in most computer applications. No wonder, sorting
and searching algorithms have received considerable study since the early 1960s when the notion of algorithm
complexity was formalized.
The primary use of a computer system is to store and retrieve data. In a simple situation, input data can be
merely a sequence of numbers such as students scores on an exam. Once entered into the computer, we are
interested in further manipulation such as determining the average score, score distribution in relation to the
average or printing the scores in a sorted order. In a more plausible situation, the input data is an aggregate data
consisting of student IDs and their scores. In this case, it is appropriate to store the input as a collection (or an
array) of records where a record consists of the ID and score for a particular student. In such a situation, we are
often interested in looking up the students score given the students ID. In this case, the ID acts as the search
key. To locate a particular record, the search key is compared against the ID-component of the data records until
a matching record is found. The details of this process are specified by the search algorithm and are dependent
on how the data (and keys) are stored. Generally, the searching problem calls for maintaining a collection of
items with keys so that the search for any of the items by key is done efficiently.
The Collection-Search Problem. Maintain a collection of items with keys so that the search for any of the items
by key is done efficiently.

There are various techniques to store a collection in memory; however, at this stage, we limit our view to the
case where a collection is specified as a sequence. A sequence, as opposed to a set, exhibits two attributes: (1)
the order (position within the sequence) of elements is important and (2) an element may appear more than once.
A sequence readily maps into an array structure, and we normally assume that the starting index is 1 (rather than
0). Thus, a sequence A with n elements is normally given as an array A[1..n].
The collection-search problem can be specialized to sequence search as follows.
The Sequence-Search Problem. Given a sequence A of n elements (keys) and an element (a search key value) x,
determine the position of an occurrence of x in A or indicate that x is not found.
In this section, we introduce two classical sequence-search algorithms: sequential search and binary search. We
will assume that the input sequence is an array of integers. We will code a search algorithm as a program
function that takes an integer array and the search key as input; the function returns the array index (position)
where the search key is found. In the case that the search key is not found, the function simply returns a special
value that cannot be a valid array index, e.g., 1.
1.4.1 Sequential Search
Sequential search, as suggested by its name, scans the input elements (keys) in sequence from beginning to end,
comparing each element in turn with the search key x. The algorithm is given in Listing 1.2, where a for-loop is
used to scan over the array elements. The loop (and the enclosing method) terminates as soon as the search key
is found. Alternatively, the loop can be expressed as a while-loop, but we prefer to use a for-loop, because the
latter is more compact. Clearly, the running time for sequential search is O(1) in the best case (which happens if
A[1]=x), and is O(n) in the worst case, which happens if x occurs near the end of the sequence or does not occur
at all. Because its worst-case running time is linearly proportional to input size n, sequential search is also
known as linear search.
Input: An integer array A[1..n] and a search key x.

Output: return the array position of first occurrence of x or -1 if x is not found.
int SequentialSearch(int[] A, int n, int x)
{ for (i = 1; i <= n; i++)
{ if (A[i] = x) return i; }
return -1;
}
Listing 1.2 Sequential Search algorithm.
1.4.2 Binary Search
Binary search is a much more efficient searching algorithm than sequential search but is only applicable to the
case where the input sequence is sorted; that is, for an input sequence A[1..n] we have: A[1] A[2] A[n].
Thus, binary search is an algorithm that solves the following problem.
The Sorted-Sequence Search Problem. Given a sorted sequence A of n elements (keys) and an element (a search
key value) x, determine the position of an occurrence of x in A or indicate that x is not found.
Although most of the time we use binary search on a sorted sequence of integers, binary search is applicable to
any collection of elements that is sorted based on a total-order relation among its elements. A total-order
relation on a set A, normally denoted by (less-than-or-equal) symbol, states that for any two elements a, b in
A, either a b or b a. The expression a < b (less-than) means that a b but a (not equal) b.
Binary search uses a divide-and-conquer approach, where the sequence A[lo..hi] is viewed as having three parts:
A[lo..mid1] (lower half), A[mid] and A[mid+1..hi] (upper half), where mid=(lo+hi)/2 (assume integer division).
When searching for x in A[lo..hi], x is compared with A[mid]. The result of such comparison is one of three
cases: x = A[mid], x < A[mid], or x > A[mid]. In the first case, x is found. In the second [third] case, there is no
point of searching for x among the elements in the upper [lower] half; thus, it suffices to repeat the search
restricted to the lower [upper] half. Such a repetitive process can be implemented recursively, as given in
Listing 1.3, or iteratively, as given in Listing 1.4. In either case, the process terminates when either x is found or
the sequence becoming empty (lo exceeds hi).
Input: A sorted integer array A[lo..hi] and a search key x.
Output: Return an array position where x is found or -1 if x is not found.
int BinarySearch(int[] A, int lo, int hi, int x)
{ if (lo <= hi)
{ int mid = (lo+hi)/2; // integer division
if (x == A[mid]) return mid;
else if (x < A[mid])
return BinarySearch(A,lo,mid-1,x);
else // x > A[mid]
return BinarySearch(A,mid+1,hi,x);
}
else return -1;
}
Listing 1.3 Recursive binary search algorithm.
Input: A sorted integer array A[1..n] and a search key x.

Output: Return an array position where x is found or -1 if x is not found.
int BinarySearch(int[] A, int n, int x)
{ int lo =1; int hi = n;
while (lo <= hi)
{ int mid = (lo+hi)/2; // integer division
if (x == A[mid]) return mid;
else if (x < A[mid])
hi = mid-1;
else // x > A[mid]
lo = mid+1;
}
return -1;
}
Listing 1.4 Iterative binary search algorithm.

Example 1.17 Table 1.5 gives a worked-out example for binary search for the input: x=41 and some input array
A[1..18]. We will follow the iterative version of the algorithm as given in Listing 1.4 (the trace for the recursive
version is same). The algorithm starts by initializing lo to 1 and hi to 18 and then enters the main loop. The first
mid value computed is (lo+hi)/2= (1+18)/2 = 9. Hence, x is compared to A[9]. In this case, x < A[9]; thus, hi is
reset to 8 (mid1) and a new iteration of the loop is started. The second mid value computed is (1+8)/2 =4;
hence, x is compared to A[4]. Since x > A[4], lo is reset to 5 (mid+1). The process continues until x is found
(which is the case for this example) or lo exceeds hi, which indicates that x is not found.
Iteration
lo
hi
mid
test
Comment
18
41 ? 58
41 < 58; hi is reset to mid-1
41 ? 37
41 > 37; lo is reset to mid+1
41 ? 49
41 < 49; hi is reset to mid-1
41 ? 41
41 is found; its position (5) is returned
Table 1.5 A worked-out example for binary search for 41 on the input array
A[1..18]= [17, 29, 33, 37, 41, 49, 52, 55, 58, 60, 63, 68, 71, 73, 75, 80, 82, 86].
Running-Time Analysis for Binary Search
To derive an expression for the number of comparisons executed by binary search on n elements, we make use
of the decision tree for binary search. This is essentially a binary search tree (i.e., a binary tree where for any
node x we have: keys in left subtree of x key(x) < keys of right subtree of x) that traces the behavior of binary
search. Figure 1.6 depicts the decision tree for a 13-element sorted sequence A[1..13]. The root of the tree
corresponds to the first element that is compared with the search key (i.e., the middle element of the sorted
sequence), which is in this case is A[7]. The elements A[1..6] belong to the left subtree and the elements A[8..13]
belongs to the right subtree. The roots of these trees are identified similarly as the middle elements of their
sequences. For example, the root of the right subtree is the element at the index (8+13)/2 = 10. We should note
that the structure (shape) of the decision tree depends only on the number of elements n and not on the elements
themselves.
Based on this tree, it is easy to see that the number of comparisons needed to locate an element that appears at
level j is j+1. For example, to search for the element A[12] (which happens to appear at level 2), we do a
comparison with A[7], then A[10], and then A[12], for a total of three comparisons. Note that, to simplify the
analysis, we are counting the two comparisons, = A[mid] and < A[mid] as a single comparison. In general
doing this sort of simplification affects the count of comparisons by a constant factor, which from
complexity-order viewpoint does not matter.
level 0
A[7]
A[8]
A[5]
A[1]
A[2]
level 1
A[10]
A[3]
A[4]
A[6]
level 2
A[12]
A[9]
A[11]
A[13]
level 3
Figure 1.6 The decision tree for a 13-element sorted sequence A[1..13]; the structure of the decision tree is dependent on
the number of elements rather than their actual values.
The worst-case number of comparisons happens when we are searching for an element that appears as one of the
bottommost elements in the decision tree. Therefore, the worst-case number of comparisons for successful
search is equal to the number of levels in the decision tree (i.e., 1+ tree height, where the height of the tree is
maximum level number, assuming the root is at level 0). Since the decision tree is height-balanced (see solved
problem #2 at the end of this chapter), its height is log n. Therefore, we conclude the following theorem.
Theorem 1.2 The worst case number of comparisons for binary search on n elements for successful search is
log n +1. The order of running time for binary search in the worst case is (log n).
Average-Case Analysis for Binary Search
In this section, we will derive an expression for the average-case number of comparisons performed by binary
search on n elements. However, it is easier to work out the analysis for some small value of n and then
generalize from there.
Example 1.18 Compute the average number of comparisons for successful search made by binary search on
a 13-element sequence assuming that the element searched for is equally likely to be any of the elements in the
input sequence.
Solution: We make use of the decision tree for a 13-element sequence shown in Figure 1.6, and the formula for
the statistical average,
(probability of case
all input cases i of size n
i occuring) * (number of comparisons for case i ) .
Because the element searched for is equally likely to be any of the 13 elements, the probability (x=A[i]) is equal
to 1/13, for i=1, 2, , 13. If the element searched for appears at level i (root is at level i=0), binary search does
i+1 comparisons. Thus, the average number of comparisons = (1/13)(1+22+34+46)=41/13=3.15
comparisons.
Next, by generalizing the preceding example, we derive a formula for the average-case number of comparisons
performed by binary search on n elements for successful search.
Assume n=2k1 for some integer k, which leads to a complete binary decision tree with k levels (a tree is
complete if every level has the maximum number of nodes). Let x denote the element to be searched for. We
will compute the average-case number of comparisons for successful search, assuming x is equally likely to be
any of the n elements A[1], A[2], ..., A[n]. Thus, the probability of x being equal to any particular element is 1/n.
For each node (element) at level i, binary search does i+1 comparisons. Summing over all nodes at level i gives
(i +1)2i comparisons. Thus,
Average number of comparisons = (1/n) (1 + 22 + 322 + 423 + ... + k2k-1)
1.7
We try to find a simple formula for S,

S = 1 + 22 + 322 + 423 + ... + k2k-1
1.8
Equation 1.8 can be rewritten in terms of x (where x=2) as,

S = 1 + 2x + 3x2 + 4x3 + ... + kxk-1
1.9
A useful trick to simplify 1.9 is to multiply both sides by (1x). This gives,
(1x) S = (1 + 2x + 3x2+ 4x3 + + k xk-1) (x+2x2 + 3x3+ + k xk)
= (1 + x+ x2 + + xk-1) k xk
Utilizing the formula for the geometric progression,
xi = ( xn+1 1) /( x 1) , we get,
i =0
(1x) S = [(xk 1) /(x1)] k xk

Finally, we substitute 2 for x to get,
S = k 2k (2k 1) n log n n
Average number of comparisons = S/n = log n 1 = (log n).
1.5 Sorting
Sorting is a fundamental and widely-studied problem with many applications. In this section, we introduce four
sorting algorithms: Selection Sort, Insertion Sort, Mergesort and Quicksort and analyze their running time and
space complexities. Both selection sort and insertion sort use the simple idea of growing the sorted sequence one
element at a time, whereas Mergesort and Quicksort utilize a divide-and-conquer approach.
The Sorting Problem. Given a sequence A of n elements and a total-order relation among the elements, arrange
the elements of A such that A[1] A[2] A[n].

Thus, we are to sort the sequence A in an increasing (ascending) order. However, because the sequence may
contain repeated values, we sometimes describe the sorted sequence as nondecreasing, since it may not be
(strictly) increasing.
Note: In presenting sorting algorithms, we normally assume the input elements are integers. However, in actual
program code for sorting elements of an arbitrary data-type, the comparison operations (, <, =) must be mapped
to the appropriate operations for the input data-type.
1.5.1 Selection Sort

Algorithm Idea: As shown by Figure 1.7, picture the input sequence as two parts: left (sorted) part and right
(unsorted) part. Grow the sorted part one element at a time by repeatedly finding (or selecting, hence the name
selection sort) the smallest element in the unsorted part and swapping it with the leftmost element in the
unsorted part.
Find smallest then swap with leftmost element
sorted elements (smallest, next smallest, )
unsorted elements
input sequence
Figure 1.7 Graphical illustration of the selection sort algorithm.
If we assume that the input is given by the array A[1..n], then using this algorithm, we make n1 passes
(iterations). In the first pass, we select the smallest element in A[1..n] and swap it with A[1]. In the second pass,
we select the smallest element in A[2..n] and swap it with A[2]; and so on. Thus, in the i-th iteration, we select
the smallest element in A[i..n] and swap it with A[i]. We need to do iterations for i = 1 to n1.
Listing 1.5 gives the program code for the selection sort algorithm; Table 1.6 gives a worked-out example.
Input: An integer array A[1..n]

Output: The array A sorted in ascending (nondecreasing) order
void SelectionSort(int[] A, int n)
{ for (i=1; i < n; i++) // suffices to do n-1 iterations
{ // find the smallest element (min) among A[i],A[i+1], ...,A[n]
min = A[i]; minpos = i;
for (j = i+1; j<= n; j++)
if (A[j] < min) { min=A[j]; minpos = j; }
// swap smallest with the i-th element
t = A[i]; A[i] = min; A[minpos] = t;
}
}
Listing 1.5 The selection sort algorithm.
Iteration
Array content at the start of iteration;

selected element is underlined.
Number of comparisons
per iteration
20, 33, 17, 61, 91, 26, 78, 11, 41
11, 33, 17, 61, 91, 26, 78, 20, 41
11, 17, 33, 61, 91, 26, 78, 20, 41
11, 17, 20, 61, 91, 26, 78, 33, 41
11, 17, 20, 26, 91, 61, 78, 33, 41
11, 17, 20, 26, 33, 61, 78, 91, 41
11, 17, 20, 26, 33, 41, 78, 91, 61
11, 17, 20, 26, 33, 41, 61, 91, 78
Final Array
11, 17, 20, 26, 33, 41, 61, 78, 91
Total = 36
Table 1.6 A worked-out example for the selection sort algorithm.
Analysis of Running Time for Selection Sort
Let us count the number of element comparisons that is A[j] < min because it is a dominant operation.
We note that in the first iteration of the outer loop the test is performed n1 times, and, in general, for a given
value i for the outer-loop index, the test is performed ni times. Thus, we get the following:
Total number of comparisons = (n1) + (n2) + + 1 = n(n1)/2 = n2/2 n/2 = (n2).
Note that the number of comparisons is the same regardless of the original order of the input. Thus, the
algorithm is (n2) for all cases (worst case, best case and average case).
1.5.2 Insertion Sort
The Insertion Sort algorithm mimics the process used by a person to order a deck of cards, while playing a card
game. The player maintains a growing collection of sorted cards by repeatedly lifting a card and inserting it at its
proper position within the sorted collection. The algorithm is of the form:
for i = 2 to n
insert A[i] in its proper place among the (sorted) elements A[1], A[2], ..., A[i-1].
To find the proper place for A[i], we compare A[i] with A[i1], A[i2], ... until we find some element A[k] such
that A[i] A[k]. Then A[i] should be placed at position k+1 and each of the elements A[k+1], A[k+2], ... A[i1]
should be advanced (moved) one position to the right. Equivalently, we can advance an element every time we
do a comparison.
Listing 1.6 gives the program code for the insertion sort algorithm; Table 1.7 gives a worked-out example.
Void InsertionSort(int[] A, int n)
{ for (i = 2; i<= n; i++)
{ // find the place for A[i]
item = A[i]; j = i-1;
while ((j>0) && (item < A[j]))
{ A[j+1] = A[j]; // advance A[j] one position to the right
j = j-1;
}
A[j+1] = item;
}
}
Listing 1.6 The insertion sort algorithm.
We should note that insertion sort will, in general, perform fewer element comparisons than selection sort. For
example, if the input array is already sorted, then, in every iteration, the number of comparisons will be exactly
one. This gives a total of n1 comparisons. However, if the input is sorted in reverse order, then the number of
comparisons is 1+2+ + n1 = n(n1)/2 = n2/2 n/2 (n2). Thus, in the worst case, insertion sort is no
better than selection sort.
Iteration
Array content at the start of iteration;

element to be inserted is underlined.
Number of comparisons
per iteration
20, 33, 17, 61, 91, 26, 78, 11, 41
20, 33, 17, 61, 91, 26, 78, 11, 41
17, 20, 33, 61, 91, 26, 78, 11, 41
17, 20, 33, 61, 91, 26, 78, 11, 41
17, 20, 33, 61, 91, 26, 78, 11, 41
17, 20, 26, 33, 61, 91, 78, 11, 41
17, 20, 26, 33, 61, 78, 91, 11, 41
11, 17, 20, 26, 33, 61,78, 91, 41
Final Array
11, 17, 20, 26, 33, 41, 61, 78, 91
Total = 22
Table 1.7 A worked-out example for the insertion sort algorithm.
Exercise 1.11 Compute as a function of the input size n, the number of array element assignments performed by
selection sort and insertion sort in the best and worst cases.
Exercise 1.12 Show that the average-case number of comparisons performed by insertion sort on n elements is
n2/4. Hint: When inserting the i-th element it is equally likely that its final position be 1, 2, , i. Based on this,
determine the average-case order of running time.
1.5.3 Mergesort
Mergesort is a good example for illustrating the divide-and-conquer strategy for algorithm design. Using this
strategy leads to an efficient sorting algorithm whose worst-case running time is (n log n). In other words, the
number of comparisons performed by Mergesort in the worst case is proportional to n log n, instead of n2 for
selection sort or insertion sort. Although Mergesort is an efficient (running-time-wise) sorting algorithm, it is
not used in practice because it is space inefficient. Besides the input array, Mergesort uses a working array as
large as the input array; therefore, the space complexity of Mergesort is (n).
Algorithm Idea: Divide the input array into two halves; sort each half and then merge the two sorted halves.
How do we sort each half? Simple, we use the same algorithm again (i.e., the algorithm calls itself on new input
arguments). To avoid infinite recursion, we need a stopping condition (base case). How about a single element
input case? A single element by itself is sorted; therefore, we do nothing. Any call to the algorithm is working
on a specified contiguous section of the initial input array and thus, as input, we use lo and hi parameters to
indicate the section boundary: lo [hi] is the index (position) of first [last] element. As shown in Listing 1.7, the
algorithm is coded to handle the general case of two or more elements, which happens if lo < hi.

// Initial call from Main is Mersgesort(A,1,n)
void Mergesort(int[] A, int lo, int hi)
{ if (lo < hi)
{ int mid =(lo+hi)/2; // integer division
Mergesort(A,lo,mid);
Mergesort(A,mid+1,hi);
Merge(A,lo,mid,hi);
}
}
void Merge(int[] A, int p, int q, int r)
{ // Input: An array section A[p..r] consists of
//
sorted subsections: A[p..q] and A[q+1..r]
// Output: A[p..r] sorted in nondecreasing order
// Use a global working array B[1..n]
s = p; t = q+1; k = p;
while (s q and t r)
{ if (A[s] A[t]
{ B[k] = A[s]; s=s+1; k=k+1; }
else
{ B[k] = A[t]; t=t+1; k=k+1; }
}
// Test which section is nonempty and copy its elements to B
if (s q) B[k..r] A[s..q];
else B[k..r] A[t..r];
// Copy B back to A
A[p..r] B[p..r]
}
Listing 1.7 The recursive mergesort algorithm.
The next logical question is how to do Merge?

Merging two adjacent sorted sections, a left section A[p..q] and a right section A[q+1..r], into a combined sorted
section A[p..r] is done with the aid of an extra work array B[p..r]. The algorithm is given as the Merge() method
in Listing 1.7.
The process of merging is illustrated in Figure 1.8. It uses three moving pointers: s over the left section of A, t
over the right section of A, and k over B. These pointers are initialized, respectively, to: p, q+1, p. The main
operation of the while-loop is to copy the smallest of A[s] and A[t] to B[k], increment two pointers and keep the
third pointer unchanged until either the left or right section is exhausted. Following the while-loop, the elements
of the unfinished section are copied to B. The final step is to copy back the array B into A.
s moves over left section
t moves over right section
1
A[1..11]:
17
11
6
29
37
41
99
23
31
49
55
83
97
smallest value of A[s] and A[t] is copied to B

B[1..11]:
17
23
29
Figure 1.8 Illustration of Merge algorithm executing Merge(A,1,5,11).
Based on the Merge algorithm given above, we can state the following lemma.
Lemma 1.1 The number of element comparisons performed by the Merge algorithm to merge two sections of
sizes n1 and n2, into a sorted section of size n=n1+n2 is between minimum(n1,n2) and n1. The number of element
assignments is 2n.
A Trace of Mergesort
Let us examine how Mergesort works on sorting the sequence A[1..8]=[8,7, 1]. The proper way to trace
through a recursive algorithm such as Mergesort is to consider the tree of recursive calls (the top tree of Figure
1.9; in the figure, we use the simpler notation MS(lo,hi) to stand for MergeSort(A,lo,hi) ). The recursive calls on
a sequence having a single element return immediately (because lo < hi is false) with the respective array
element unchanged. Other calls return after executing a call to Merge(). The bottom tree shows the result as the
recursive calls return. Each node in this tree (other than leaf nodes) corresponds to a call to the Merge() method.
After the two calls MS(1,1) and MS(2,2) return, we execute the call Merge(A,1,1,2) (i.e., we are to merge A[1]
and A[2]). This results in A[1..2]=[7,8]. The next call to Merge() is Merge(A,3,3,4) which returns with
A[3..4]=[5,6]. The process continues until reaching the final call to Merge, Merge(A,1,4,8) to merge A[1..4] and
A[5..8].
MS(1,8)
Divide and Recurse
MS(1,4)
MS(1,2)
MS(3,4)
MS(2,2)
MS(1,1)
MS(5,8)
MS(3,3)
MS(7,8)
MS(5,6)
MS(4,4)
MS(5,5)
MS(6,6)
MS(7,7)
MS(8,8)
1, 2, 3, 4, 5, 6, 7, 8
(7)
5, 6, 7, 8
1, 2, 3, 4
(3)
(6)
7, 8
(1)
8
5, 6
3, 4
1, 2
(2)
(4)
(5)
Merge
1
Figure 1.9 Recursive Mergesort sorting A[1..8]=[8, 7, 1]; parenthesized numbers indicate calls to Merge() and their
order.
Iterative Mergesort (Bottom-Up Mergesort)
The trace of recursive Mergesort reveals that Mergesort is simply issuing a sequence of calls to Merge() in some
particular order dictated by recursion. To implement this process iteratively, let the iterative Mergesort call
Merge() in the order of tree levels from bottom to top (bottom-up). Thus, in reference to Figure 1.9, the iterative
Mergesort will issue the calls 1, 2, 4, 5 corresponding to the bottommost level, where each call merges two
sections of size one element each then the calls 3, 6, where each call merges two sections of size two elements
each, etc. In general, this process can be implemented as a nested loop; the outer loop varies the section size
from 1, doubling the section size with every iteration; and an inner loop that varies the starting index of the first
of the two sections to be merged, from 1 then incrementing by twice the section-size with every iteration. The
resulting algorithm is given Listing 1.8. While the size of the first section is always a power 2, the algorithm has
to account for the possibility that this may not be the case for the second section. Such situation is illustrated in
Figure 1.10. Note that in Listing 1.8, the expression 2*sec_size, which appears twice in the body of the inner
loop, is a loop-invariant (i.e., it does not change from one iteration to the next) and, in an optimized code, should
be computed once prior to entering the inner loop.
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
3, 4, 5, 6, 9, 10, 11
5, 6, 9, 10
6, 10
10
5, 9
1, 2, 7
3, 4, 8, 11
1, 2
4, 8
3, 11
11
Figure 1.10 Iterative (bottom-up) Mergesort to sort a sequence whose size is not a power of 2.

Output: The array A sorted in nondecreasing order
void MergeSort(int[] A, int n)
{ int sec_size = 1;
while (sec_size < n)
{ int sec1_start = 1;
while (sec1_start < n)
{ int sec2_end = sec1_start + 2 * sec_size-1;
if (sec2_end > n) sec2_end = n;
Merge(A,sec1_start,sec1_start+sec_size-1,sec2_end);
sec1_start = sec1_start+ 2*sec_size;
}
sec_size = 2*sec_size;
}
}
Listing 1.8 Iterative (bottom-up) Mergesort algorithm.
Analysis of Mergesort
Let us count the total number of element comparisons performed by Mergesort. The total number of
comparisons can be computed using the following expression:
number of calls to Merge() number of comparisons per call
Total Comparisons =
levels of the tree
To simplify things, we will assume that the number of elements n=2k for some positive integer k. In this case,
the calls to Merge() will operate on the same data (size-wise) for both recursive and iterative versions of
Mergesort. Recall from Lemma 1.1 that when merging two sections of sizes n1 and n2, the number of
comparisons lies between minimum(n1,n2) and n1+n21. Thus, the count of comparisons for best and worst cases
will follow the pattern shown in Table 1.8.
Tree Level
Calls to Merge
Count of comparisons
Best Case
Worst case
1 (lowest)
n/2 calls, each merging two sections of size 1
n/2 1
n/2 1
n/4 2
n/4 3
n/8 4
n/8 7
k (root)
1 call merging two sections of size n/2
1 n/2
1 (n1)
Table 1.8 Comparison count for Mergesort.
Thus, we get,
Best-Case Total Comparisons = n/2 1 + n/4 2 + + 1 n/2 = k (n/2) = (n/2) log n.
[Note that the given expression consists of k terms by noting that the first factor varies from n/2=2k-1 to 1=20.]
Worst-Case Total Comparisons, S = n/2 1 + n/4 3 + + 1 (n1)
Rewrite the preceding expression in terms of k to get,
S = n/2 (21) + n/4 (41) + + n/2i (2i 1) + + (n/2k) (2k 1)
= [n/2 2 + n/4 4 + + (n/2k) (2k)] (n/2+ n/4 + + n/2k)
= kn n (1/2+ 1/4 + + 1/2k).
Because n = 2k, we get,
= kn (2k-1 + 2k-2 + + 1) = kn (2k 1) = n log n n +1.
Theorem 1.3 Let m be the number of comparisons performed by Mergesort on n elements, where n=2k for some
positive integer k, then (n log n) m (n log n)n+1. For all positive integers n, the running time of
Mergesort is (n log n) in both the best case and worst case.
Proof: The first statement is restating the results established previously. This also implies that for n=2k,
Mergesort is (n log n) in both the best case and worst case. If n is not a power of 2, then n lies between n1 and
n2, where n1=2i and n2=2i+1 for some positive integer i. Thus, the order of running time lies between (n1 log n1)
and (n2 log n2). Because n1 > n/2 (Why?) and n2 < 2n, the running time lies between (n/2 log n/2) and (2n
log 2n), but, orderwise, these two expressions reduce to (n log n).
Space Complexity Analysis of Mergesort
The iterative Mergesort on n elements is clearly having a complexity of (n) because of the extra array of size n
used by Merge. On the other hand, the recursive Mergesort uses, in addition to the working array used by
Merge, stack space to store call parameters and return addresses. The amount of stack space used is proportional
to the maximum number of stacked procedure calls, which is equal to the height of the tree of recursive calls
(i.e., log n). Thus, the space complexity of recursive Mergesort is (n+log n)=(n). Therefore, although the
recursive Mergesort uses more memory space, both algorithms have the same order of space complexity.
1.5.4 Quicksort
Quicksort is a well-known and practical sorting algorithm attributed to C. Hoare [Hoa62]. It has an average-case
time complexity of (n log n). However, in the worst-case (which happens rarely) Quicksort is (n2). In
practice, Quicksort is much faster than other (n log n) algorithms including Heapsort and Mergesort. In an
actual experiment run on a Pentium IV 2.6 MHz running Windows XP [Ald05], in sorting 100 million elements,
Quicksort took 26 seconds versus 219 seconds for Heapsort.
The outline of the Quicksort algorithm resembles that of Mergesort see Figure 1.11 and Listing 1.9. It uses a
divide-and-conquer strategy, dividing the input sequence into two parts (sections) and sorting each part
recursively but it does away with the Merge step. There will be no need to call Merge() if the elements in the
first part are all smaller than the elements in the second part; therefore, once each part is sorted, we are done. To
achieve this, Quicksort does in-place partition (in-place means without using an additional working array) of its
input: it picks an element p and arranges the elements so that elements p are moved to one side (left side) of
the sequence and the elements > p are moved to the other side (right side). The element p used for partitioning is
referred to as the pivot element. The pivot can be any of the elements in the input sequence to be partitioned;
however, quite often, the first (leftmost) element is chosen as the pivot. In summary, Mergesort sorts two
sections and then does postwork (i.e., merge), whereas Quicksort does prework (i.e., partition) and then sorts
two sections.
Pivot
Partition
Sort recursively by Quicksort
>p
Sort recursively by Quicksort
Figure 1.11 Graphical illustration of Quicksort.

Example 1.19 Indicate the two calls to Quicksort that result when executing the call Quicksort(A,1,10), where
A[1..10] = [43, 22, 81, 69, 63, 71, 48, 19, 51, 26].
Solution: The call Quicksort(A,1,10) gives rise to the call Partition(A,1,10). In turn, Partition() uses 43 as the
pivot and arranges the elements such that {22,19,26} (because they are pivot) belong to the left section, with
the pivot ending at location 4. Thus, the call to Quicksort(A,1,10) results in the calls: Quicksort(A,1,3) and
Quicksort(A,5,10).
One simple algorithm for partitioning a sequence is given as the Partition() method in Listing 1.9. The
algorithm utilizes two pointers (left and right) that are, respectively, set to the start and end of the input sequence
to be partitioned. The left pointer moves toward the right, skipping elements pivot, whereas the right pointer
moves toward the left, skipping elements > pivot, but where the left and right pointers stop, the corresponding
elements are swapped. The process ends when the left and right pointers meet or pass each other. However, a
final swap involving the pivot element and the element pointed to by the right pointer is executed to put the
pivot into its proper sorted location.
Example 1.20 Table 1.9 illustrates the actions of the call Partition(A,lo,hi) with lo=1 and hi=11 for some given
input array A[1..11]. This is the first call to Partition() resulting from the call Quicksort(A,1,11). A detailed trace
of Quicksort(A,1,11) is shown in Table 1.10. Each row in this table corresponds to a call to Quicksort hence,
a call to Partition() and it suffices to show the calls for which lo < hi. Note that the call to Quicksort(A,1,11)
results in the calls: Quicksort(A,1,5) and Quicksort(A,7,11). However, the actual execution of the latter call does
not begin until the call to Quicksort(A,1,5) returns.

Output: The array A sorted in nondecreasing order
void Quicksort(int[] A, int lo, int hi)
{ if (lo < hi)
{ int mid = Partition(A,lo,hi);
Quicksort(A,lo,mid-1);
Quicksort(A,mid+1,hi);
}
}
int
{ //
//
//
//
//
Partition(int[] A, int lo, int hi)

Input: An integer array A[lo..hi]
Output: The elements arranged as: elements pivot, pivot, elements > pivot;
Return the final position of pivot
Assume pivot is A[lo] but it can be any of the elements A[lo], .. A[hi];
if pivot is other than A[lo] then (at the start) swap it with A[lo]
int pivot = A[lo];

int left = lo+1; int right = hi;
while (left <= right)
{ while ((left <= right) && (A[left] <= pivot)) left++;
while ((left <= right) && (A[right] > pivot)) right--;
if (left < right)
{ int t = A[left]; A[left] = A[right]; A[right] = t; left++; right--; }
}
//Note: When exiting the loop A[right] pivot A[right] belong to left part
//Final swap: swap pivot with A[right] and return right as the final location of pivot
A[lo] = A[right]; A[right] = pivot;
return right;
}
Listing 1.9 Quicksort algorithm.
lo
1
hi
11
Content of array A; pivot element is A[lo].

1
2
3
4
5
6
7
8
9
10
11
33
17
29
51
83
23
67
11
43
19
71
33
17
29
19
83
23
67
11
43
51
71
33
17
29
19
11
23
67
83
43
51
71
23
17
29
19
11
33
67
83
43
51
71
Comments
mid
left stops at A[4] and right stops at A[10];

swap A[4] with A[10]
swap A[5] with A[8]
swap A[lo] with A[right]; return right as mid
Table 1.9 Actions of the call Partition(A,lo,hi) with lo =1 and hi =11.
lo
hi
Content of array A at start of iteration; pivot element is A[lo].

1
2
3
4
5
6
7
8
9
10 11
mid
11
33
17
29
51
83
23
67
11
43
19
71
23
17
29
19
11
33
67
83
43
51
71
19
17
11
23
29
33
67
83
43
51
71
11
17
19
23
29
33
67
83
43
51
71
11
11
17
19
23
29
33
67
83
43
51
71
11
17
19
23
29
33
43
51
67
83
71
10
11
11
17
19
23
29
33
43
51
67
83
71
11
11
17
19
23
29
33
43
51
67
71
83
Final array
Table 1.10 A worked-out example for Quicksort algorithm.
QS(1,11)
QS(7,11)
QS(1,5)
QS(1,3)
QS(1,2) QS(4,3)
QS(1, 0)
QS(5,5)
QS(7,8)
QS(7,6)
QS(10,11)
QS(8,8) QS(10,10) QS(12,11)
QS(2,2)
Figure 1.12 The tree of recursive calls for Quicksort corresponding to the example shown in Table 1.10.
Running-Time Analysis of Quicksort
Examining the Partition() algorithm given in Listing 1.9, we note that it does one comparison (either A[left]
pivot or A[right] > pivot) and move by one position until the left and right pointers meet. Thus, the number of
comparisons is hilo (the number of positions from lo+1 to hi inclusive). In general, Partition() does n1
element comparisons when called on an input of size n.
The worst-case running time for Quicksort happens when the input A[1..n] is sorted. In this case, assuming that
the pivot is chosen as the leftmost element, the first call to Partition() does n1 comparisons and returns with
mid=1 and two subproblems: Quicksort empty input and Quicksort A[2..n] (i.e., n1 elements). The latter call to
Quicksort results in a call to Partition(A[2..n]) which does n2 comparisons and returns with mid=2. This pattern
continues; thus, the total number of comparisons is given as,
Total number of comparisons = (n1) + (n2) + + 1 = n(n1)/2 = n2/2 n/2 = (n2).
Ideally, for Quicksort to finish quickly, every call to Partition() should divide its input in two equal (or nearly
equal) parts. In this case, the number of comparisons for Quicksort, C(n), is given by the recurrence C(n)=(n
1)+2C(n/2). Later, we will discuss such recurrence and prove that C(n)=(n log n). This implies that the bestcase order of running time for Quicksort is (n log n). Furthermore, the analysis of the average-case running
time (see Section 4.1.5) shows that Quicksort is (n log n).
Practical Considerations for Using Quicksort
Quicksort works well if the pivot used by Partition() falls near the middle of the segment to be partitioned.
Choosing the leftmost element as the pivot causes Quicksort to do poorly when the array is sorted or nearly
sorted. To help in these situations, it has been suggested that a better pivot selection strategy is to select the pivot
randomly (i.e., let r be a random integer between lo and hi, then choose A[r] as the pivot). Another strategy is to
use the median (the element having the middle value) of A[lo], A[hi] and A[(lo+hi)/2].
Quicksort is known to suffer from two other problems that cause performance degradation. The first problem is
that the algorithm (i.e., repeated partitioning) is slow (in comparison with other sorting algorithms) when
applied to small size data, especially for data that is nearly sorted or containing many duplicate values. Altering
the pivot selection strategy does not help. Rather, it has been suggested that Quicksort does not call itself when
the input falls below a certain size such as 25 elements; instead, it calls another sorting algorithm such as
insertion sort. A variation of this solution is to exit Quicksort when the input size falls below 25, and then when
the initial (top-level) call to Quicksort returns, call insertion sort once on the whole data.
The second problem has to do with the stack space (associated with the recursive calls) incurred by Quicksort.
In the worst case, the recursion depth (and associated stack space) can grow as bad as (n). For example, if the
input is sorted and the pivot is the leftmost element, then this results in a sequence of procedure calls (all
awaiting completion): Quicksort(A,1,n), Quicksort(A,2,n), , Quicksort(A,n1,n). A solution to this problem is
to rewrite Quicksort utilizing tail-recursion elimination to limit the depth of recursion. Tail-recursion
elimination means replacing a recursive call located at the end (tail) of a procedures body by a jump back to the
first statement in the procedure. The trick to limit recursion depth to O(log n) is to eliminate the recursive call
that has more than half of the elements. This way, every recursive call handles at most half of the elements
handled by its parent, which guarantees a maximum of log n pending calls. The modified algorithm is given in
Listing 1.10.
void Quicksort_TRO(int[] A, int lo, int hi)

{ while (lo < hi)
{ int mid = Partition(A,lo,hi);
if ((mid-lo) > (hi-mid))
{ Quicksort_TRO(A,mid+1,hi);
hi = mid-1; }
else
{ Quicksort_TRO(A,lo, mid-1);
lo = mid+1; }
}
}
Listing 1.10 Quicksort _TRO(): A tail-recursion optimized Quicksort.
Exercise 1.13 Draw the trees of recursive calls for both Quicksort and Quicksort_TRO on the input A[1..10] =
[43, 22, 81, 69, 63, 71, 48, 19, 51, 26]. Assume Partition() uses the leftmost element as pivot.
1.5.5 Sorting on Multiple Fields
In many situations, the need arises for sorting on multiple fields. For example, it may be desirable to print
student records sorted by GPA and that for every group of records having the same GPA value, the records must
be sorted by Student ID. A similar situation occurs when it is required to sort student records by DateOfBirth. In
this case, we are to sort data on a composite (multicomponent) field. It means that records are to be output
sorted by year and for every group of records that have the same year value, they must appear sorted by month,
and, finally, for every group having a particular year-month value, records must be sorted by day.
In general, whenever we sort by the fields, Fld1 then Fld2 (or, simply, the field sequence, Fld1, Fld2), Fld1
[Fld2] is said to be the most [least] significant field for justification of this, see the side note.
Before we discuss the different solutions to sorting on multiple fields, let us introduce the concept of stable
sorting.
Definition. A Stable Sorting Algorithm: A sorting algorithm is stable if it does not change the order of equal-key
elements (think of the elements as records with keys) from that given in the input.
Example 1.21 Sort by GPA the following input:
(99310, 3.0), (99511, 2.0), (99615, 2.5), (99617, 3.1), (99726, 2.5), (99735, 2.0).
Note:The input happens (or on purpose!) to be sorted by ID.
The following two outputs are both correct because each output is sorted by GPA.
Output-A: (99511, 2.0), (99735, 2.0), (99615, 2.5), (99726, 2.5), (99310, 3.0), (99617, 3.1).
Output-B: (99735, 2.0), (99511, 2.0), (99615, 2.5), (99726, 2.5), (99310, 3.0), (99617, 3.1).
Question: Which output do we prefer and which output is likely to be produced by a stable sorting algorithm?
Answer: Output-A for both questions. We prefer Output-A, because the data is sorted by GPA and then by ID.
Also, Output-A, unlike Output-B, is consistent with the behavior of a stable sorting algorithm.
We observe that a stable sorting algorithm cannot produce Output-B because (compare the input to Output-B)
the two records having the same sort key (GPA) value of 2.0 have their order changed from that appearing in the
input. This observation tells us that to sort student records by GPA-then-ID, we can do the following: first, sort
the records by ID, and then use a stable sorting algorithm to sort the whole data by GPA.
In general, we claim that given an input that has already been sorted on some field and then sorting the whole
data on another field using a stable sorting algorithm, does not undo the sorting work done previously. This is
the basis for the first solution to sorting on multiple fields.
Next, we give three different solutions to the problem of sorting on multiple fields.
Output Layout for Sorting on Multiple Fields
An organized way to output a set of records sorted by two fields Fld1 then Fld2, is to have the
records grouped by Fld1, which is output to the left. For the example given, where records are to
be sorted by GPA and then ID, the records will be grouped by GPA, as shown next. Such
organization clearly shows that Fld1 is the most significant field.
GPA
2.0
2.5
3.0
3.1
ID
99511
99735
99615
99726
99310
99617
The preceding organization helps in visualizing (and reasoning about) the structure of the data.
For example, with the grouping by in mind, does it make sense to sort student records by ID and
then GPA?
Solution 1:
Sort the input data (as a whole) by the least significant field then by the next-to-least significant field, until the
most significant field. This solution requires a stable sorting algorithm. For example, to sort A[1..n] by GPA and
then by ID, we execute the following in order:
Sort(A,1,n, "ID"); Sort(A,1,n, "GPA").
As another example, to sort A[1..n] on a Date field, we execute the following in order:
Sort(A,1,n, "day"); Sort(A,1,n, "month"); Sort(A,1,n, "year").
Solution 2:
First, sort the input on the most significant field Fld1 and remember the boundaries (start and end positions) of
every occurring value (group). Suppose there are m groups and that P1, P2, ..., Pm represent the starting positions
of these groups (the end position is one less than the starting position of the next higher group). Next, sort each
group on the next-to-most significant field as in the following:
Sort(A,P1,P21, Fld2), Sort(A,P2,P31, Fld2), etc.
The process is continued in similar fashion using Fld3, Fld4, and so on. This solution does not require a stable
sorting algorithm.
Solution 3:
Using this solution, the sorting algorithm is called once. In this solution, we modify the comparison operator so
as to examine the fields in a particular order from most significant field to least significant field but only as
necessary. For example, to sort on two fields: Fld1 and then Fld2, the lessthan (<) comparison of elements x and
y becomes:
x[Fld1, Fld2] < y[Fld1, Fld2] is computed as: (x[Fld1] < y[Fld1]) or (x[Fld1] = y[Fld1] and x[Fld2] < y[Fld2]).
In object-oriented programming, this solution is best implemented by making use of interfaces. The sorting
algorithm is coded so that its input array parameter is declared to be of type IComparable interface rather than
a specific data-type such as integer or float. Within the sorting algorithm, the comparison operation involving
elements x and y is replaced by the expression x.CompareTo(y). In turn, the CompareTo operation is defined in
accordance with the preceding specification. For further discussion of this solution, see Section 11.4.
Note: The first two solutions presented are similar to the process of radix sorting (Section 3.9.2). Solution 1 is
akin to LSD radix sort, whereas Solution 2 is akin to MSD radix sort.
Exercise 1.14 Out of the known sorting algorithms, identify the ones that are stable and the ones that are
unstable.
Exercise 1.15 Write a program method to implement Solution 2 along with the required modifications to the
Sort algorithm called by your method. Assume we are to sort on two fields: Fld1 and Fld2.
Hint: A simple approach is to sort the data on Fld1 and then scan the output from first item to last item noting
that any two adjacent items with different values for Fld1 would indicate the start of a new group.
1.5.6 A Lower Bound for Sorting by Comparison
The number of comparisons executed by insertion sort and Quicksort in the worst case is O(n2). Mergesort
improves the worst case to O(n log n). Can this be improved further? In this section, we derive a lower bound
for the number of comparisons executed in the worst case by any algorithm that sorts by comparison of
elements. These results tell us when we should stop looking for a better algorithm. To derive the lower bound,
we assume that the elements in the sequence to be sorted are distinct.
The technique used to establish a lower bound on sorting by comparisons utilizes decision trees.
Figure 1.13 shows the decision tree for insertion sort for three elements. A decision tree for a sorting algorithm
is a binary tree whose nodes (other than leaf nodes) correspond to comparisons executed by the algorithm. The
root of the tree corresponds to the first comparison. The algorithm learns more information about the final order
with every comparison, and, thus, it can be thought of as following either the left or right branch depending on
the (true/false) result of comparison until the algorithm terminates with the sorted order.
Note that regardless of the sorting algorithm, the decision tree for an input of size n will have n! leaves because
there are n! different orderings of n distinct elements. The number of comparisons performed by the algorithm
on a specific input is the number of internal nodes on a path from the root to a leaf node (the root is included but
the leaf node is excluded; viewed from the leaf node this number is the depth of the leaf). Thus, the maximum
number of comparisons is the height of the tree. For the example shown in Figure 1.13, we easily see that the
height of the tree (and, therefore, the maximum number of comparisons) is 3.
Lemma 1.2 In a binary tree with L leaves (terminal nodes) and height h, L 2h.
Proof (using strong induction on h):
Base Step: A tree of height 0 consists of a single node, which itself is a terminal node and the given inequality is
satisfied for L=1 and h =0.
Induction Step: Assume that any binary tree of height h1 has 2h-1 leaves. Now consider a tree T of height h.
We must show that T has 2h leaves. The root of T has two child trees (subtrees) (T1 and T2), where each
subtree T1 and T2 is of height h1. The leaves in T consist of the leaves in T1 plus the leaves in T2. By the
induction hypothesis T1 (and T2) has 2h-1 leaves. Thus, L (number of leaves in T) 2h-1 + 2h-1 = 2h.
a2 < a1
a 1 , a2 , a3
F
a3 < a2
T
a 3 < a1
a 1 , a3 , a2
a3 < a1
a 1 , a2 , a3
a 2 , a1 , a3
F
a 1 , a3 , a2
a 2 , a3 , a1
T
a 3 < a2
a 2 , a1 , a3
T
a 3 , a1 , a2
F
a 2 , a3 , a1
T
a 3 , a2 , a1
Figure 1.13 The decision tree for the insertion sort algorithm on three elements.
Theorem 1.4 Any algorithm to sort n elements by pairwise comparison of elements executes (n log n)
comparisons in the worst case.
Proof [Cor01]: The length of the longest path in the decision tree from the root to any of its leaves represents the
worst-case number of comparisons performed by the sorting algorithm. Consequently, the worst-case number of
comparisons corresponds to the height of the decision tree. A lower bound on the height of the tree is, therefore,
a lower bound on the running time. The decision tree has n! leaves because there are n! permutations of n
elements. By Lemma 1.2, the height of a binary tree h with L leaves must satisfy h log L. Thus, the height of
the decision tree is such that h log n! (n/2) log (n/2) h = (n log n).
Theorem 1.5 Any algorithm to sort n elements by pairwise comparison of elements executes (n log n)
comparisons in the average case.
Proof [Blu06]: Consider the decision tree again. Because the depth of a leaf represents the number of
comparisons to an input (in some particular order) of n elements, the average number of comparisons is simply
the average depth = sum of the depths of leaves divided by the number of leaves. If the tree is completely
balanced (for the purpose of this proof, a tree is completely balanced if the difference between the smallest
depth and the largest depth of any of its leaves is at most 1), then each leaf is at depth log n! or log n! , and
we are done. To prove the theorem, we just need to show that out of all binary trees on a given number of
leaves, the one that minimizes their average depth is a completely balanced tree. This is not too hard to see:
Given some unbalanced tree, we take two sibling leaves at largest depth and move them to be children of the
leaf of the smallest depth. Because the difference between the largest depth and the smallest depth is at least 2
(otherwise the tree would be balanced), this operation reduces the average depth of the leaves. Specifically, if
the smaller depth is d and the larger depth is D, we have removed two leaves of depth D and one of depth d, and
we have added two leaves of depth d+1 and one of depth D1. Because any unbalanced tree can be modified to
have a smaller average depth, such a tree cannot be one that minimizes average depth, and therefore the tree of
smallest average depth must, in fact, be balanced.
Exercise 1.16 Does there exist an algorithm that makes at most four comparisons to sort four elements? Justify
your answer.
Exercise 1.17 What is the smallest possible depth of a leaf in a decision tree for a comparison sort? Argue that
your answer is correct.
Exercise 1.18 Precise calculation of log n! using integrals shows that log n! n log n 1.443 n. Compare this
number with the worst-case number of comparisons needed by Mergesort on 16 elements.
1. Solved Exercises
1. Convert the recursive procedure for counting binary strings of length n that do not contain consecutive 1s,
which is given as Solution 3 for Example 1.22, into an equivalent iterative procedure.
To convert recursion into iteration, we can use a known technique to evaluate the recurrence forward starting
with the base equations. Using this approach, we store the solutions for Rn for different values of n as elements
in an array r. This way, the recursive call R(i) is changed into an array-lookup r[i]. The iterative procedure is as
follows:
int R(int n)
{ int[] r = new int[n+1];
r[1] = 2;
r[2] = 3;
for i = 3 to n
{ r[i] = r[i-1]+r[i-2]; }
return r[n];
}
The preceding procedure takes little time even for large values of n because it basically does about n additions in
total. We can further reduce the space used by using a 3-element array as follows:
int R(int n)
{ int[] r = new int[3]; // We use r[0], r[1], r[2]
r[1] = 2;
r[2] = 3;
for i = 3 to n
{ r[i mod 3] = r[(i-1) mod 3]+r[(i-2) mod 3]; }
return r[n mod 3];
}
2. Prove that the decisions tree for binary search on k elements is of height at most log k.
Solution (Proof): (Note: The preceding statement uses the floor function because the height must be an integer.)
The proof uses the fact that the decision tree has the property that the left and right subtrees of the root either
have the same number of elements or differ by one element. This follows from the fact that binary search
divides A[1..n] into A[1..mid1] going to the left subtree and A[mid+1..n] going to the right subtree, where
mid=(1+n)/2. The proof is by strong induction on k.
The base step is for k=1. Because a single-node tree is of height zero, the statement is true in this case. For the
induction step, assume that for all integers p where 1 p < k, the height of a decision tree with p nodes is log
p. Let T be a decision tree with k nodes where k > 1. Then T consists of a root and two child trees: T1 with m
nodes and T2 with m or m1 nodes. This implies that m k/2 (because either m+m=k1 or m+(m1)=k1). Let
h1 and h2 denote the heights of T1 and T2, respectively. By the induction hypothesis, the height of T1 (and T2) is
log m. The height of T is as follows:
1+ maximum { h1, h2 } 1+ log m 1+log k/2 = 1+ log k log 2 = 1+log k 1 = log k.
This completes the proof.
3. If an algorithm has O(n2) order of running time for n being a power of 2, can we conclude that the algorithm
is O(n2) even if n is not a power of 2? Justify your answer.
Solution: Yes. If n is not a power of 2, then n lies between two successive powers of 2. Thus, we can assume that
there exists some positive integer k such that 2k < n < 2k+1. Now, the running time T(n) is such that
T(n) T(2k+1)
C (2k+1)2
= C (22k) 2
< C (2n)2
= 4C n2.
because n < 2k+1 and we assume that the running time increases as
input size increases
because T(n) = O(n2) if n is a power of 2; C is a positive real constant
because 2k < n
Thus, T(n) 4Cn2 T(n) is O(n2).

4. Work out the time complexity of the following program code.
for(i=1; i < n; i *= 2)
for(j = n; j > 0; j /= 2)
for(k = j; k < n; k += 2)
{ sum += i+j*k; }
Solution: The running time of the outer loop is proportional to log n because i starts at 1 and is doubled each
time until reaching n. Similarly the running time of the middle loop is proportional to log n. The running time
for the inner loop is proportional to nj for j = n, n/2, , 1. Thus, the running time for the inner and middle
loops combined is proportional to 0+n/2+n/4++n = O(n log n). Thus, the overall time complexity is O(n(log
n)2).
5. The following algorithm generates a random permutation (stored in array A) of the integers from 0 to n1.
Determine the order of its running time as a function of n. Assume that the function call Random(n) returns
in O(1) time a random integer in [0,n1].
int[] A = new int[n];
int[] values = new int[n];
for(int i=0; i < n; i++) values[i]=i;
int pos = 0;
while (pos < n)
{ int r = Random(n); // r in [0,n-1] and is used as an index into values array
if (values[r] 0)
{ A[pos] = values[r];
values[r] = -1; // Mark the value as no longer available
pos++;
}
}
Solution: When filling the last position of the array A the probability of r corresponding to an unused value (i.e.,
values[r] 0) is 1/n because n1 values (out of the n different values) have already been used. Thus, we expect
in the worst case to invoke Random() n times. By a similar argument, to fill the next-to-last position of the array
A, we expect in the worst case to invoke Random() n1 times. Thus, the number of times the while-loop is
executed is n+(n1)+ +1 = n(n1)/2. Thus, the given algorithm is O(n2).
6. Modify the preceding algorithm to achieve O(n) running time.
Solution: To do away with the test values[r] 0, we move the used values toward the end of the values array
(i.e., swap values[r] with values[n1pos] but there is no need to save values[r]). This way, we can replace the
call Random(n) by Random(npos). The resulting algorithm is shown next and is clearly O(n).
int[] values = new int[n];
for(int i=0; i < n; i++) values[i]=i;
int pos = 0;
while (pos < n)
{ int r = Random(n-pos); // r is used as a position into values array
A[pos] = values[r];
values[r] = values[n-1-pos];
pos++;
}
7. Modify the preceding algorithm to achieve O(1) space.

Solution: To achieve O(1) space, we must get rid of the values array and somehow use the array A to simulate
the role of the values array. Simply, think of the values that we need to move toward the end of the array as
generating the elements of the permutation from the last (rightmost) position to the first position.
for(int i=0; i < n; i++) A[i]=i;
int pos = 0;
while (pos < n)
{ int r = Random(n-pos);
int t = A[n-1-pos];
A[n-1-pos] = A[r];
A[r] = t;
pos++;
}
8. Given a sorted array A of n distinct integers (positive or negative), devise an algorithm to find an index i such
that A[i] = i if such an index exists.
Solution: We adopt the binary search principle. Consider an/2 and compare it with n/2. We can consider three
cases. If an/2 = n/2, then we are done. If an/2 < n/2, then, because all elements are distinct, an/2-1 will be less than
n/21, and so on. Thus, no element in the first half of the sequence can satisfy the property, and we can continue
searching the second half. A similar argument holds if an/2 > n/2 that leads to excluding the second half and
searching the first half.
9. Consider the problem of merging k sorted lists, each having n/k elements.
(a) First, consider the following algorithm. We take the first list and merge it with the second list using a lineartime algorithm for merging two sorted lists, such as the merging algorithm used in Mergesort. Then, we merge
the resulting list of 2n/k elements with the third list, and so forth, until we end up with a single sorted list of all
the n elements. Analyze the worst-case running time of this algorithm in terms of n and k.
(b) Next, suppose we use an algorithm that merges lists of equal size in pairs; i.e., first consider lists of size n/k
and merge lists 1 and 2, lists 3 and 4, and so on. Then consider merging pairs of lists of size 2n/k, and so on.
Solution:
(a) Merging the first two lists, each of n/k elements, takes 2n/k time. Merging the resulting elements with the
third list of n/k elements takes 3n/k time, and so on. Thus, for a total of k lists, we have:
Time = 2n/k + 3n/k + + kn/k =n/k (2+ 3 + + k) = n/k [k(k+1)/21]= n/k [(k2)(k+1)/2]= (nk).
(b) This algorithm can be seen as the merge-phase of Mergesort, but where the leaves in the tree are groups of
n/k elements. Such a tree is of height log k. For any tree level (each tree level corresponds to a specific list size),
the cost of all Merge-calls is (n). Thus, the algorithm runs in (nk) time.
10. Trace Quicksort for the input A[1..9] = [15, 3, 22, 66, 7, 81, 25, 17, 33]. Indicate the values of lo, hi and mid
as passed to (or returned from) Partition() as well as the modified array. List only the rows for which lo < hi,
and use the leftmost element as the pivot.
Solution:
Content of array A at start of iteration; pivot element is A[lo].
1
2
3
4
5
6
7
8
9
lo
hi
15
22
66
81
25
17
33
15
66
22
81
25
17
33
15
66
22
81
25
17
33
15
17
22
33
25
66
81
15
17
22
33
25
66
81
15
17
22
33
25
66
81
15
17
22
25
33
66
81
Final array
mid
11. Recall the definition of a stable sorting algorithm. How can we use an unstable sorting algorithm
(comparison-based), such as Quicksort or Heapsort, to build a stable sorting algorithm with the same time
complexity?
Solution 1: Let A be the unstable sort algorithm. Add a new field, index, to each element in the input array S.
This new field holds the original index of that element in the input. This step is O(n). The remaining steps:
1. Execute A on S to sort the elements by their key T(A)

2. Execute A on each set of equal-key elements by the index field T(A)
Time complexity: The time complexity is T(A+n) = T(A). Assume we have m different keys (1 m n):
T(n) = O(n log n) + O(n1 log n1 + n2 log n2 + ... + nm log nm)
O(n log n) + O(n1 log n + n2 log n + ... + nm log n)
= O(n log n ) + log n O(n1 + n2 + ... + nm)
= O(n log n) + log n O(n)
= O(n log n) + O(n log n) = O(n log n)
Solution 2: Add a new field index to each element. This new field holds the original index of that element.
Change the comparison operator so that:
[key1, index1] < [key2, index2] (key1 < key2) or (key1 = key2 and index1 < index2)
[key1, index1] = [key2, index2] (key1 = key2) and (index1 = index2)
Execute the sort algorithm with the new comparison operation.
1. Exercises
1.
Utilize the slot-filling algorithm given in Section 1.2.1 to write algorithms for the following problems.
a. Print all strings of length n that use the letters af. Hint: Think of S[i] as an index to lookup a letter.
b. Print all strings of length n that use the letters af with no repeated adjacent letters.
2.
Utilize slot filling to write algorithms to print all permutations of n objects. Assume the objects are denoted
by the digits 1n.
3.
A group of n men and m women are to stand in a row for a photo such that no woman is squeezed between
two men. Develop (and argue) a recurrence equation for the number of ways in which this can be done.
Hint: If we think of men (women) as represented by 1s (0s), then we are to count binary strings with n 1s
and m 0s that do not contain the substring "101". Now, let cn,m be the number of such strings. Write (and
argue) base and recursive equations for cn,m.
4.
Given two algorithms whose running times (as a count of elementary operations) are given by
T1(n) = n log n and T2(n) = n2, respectively. If the first algorithm takes 1 second on an input of size n = 1
Million, how much time we expect the second algorithm to take on the same input? Hint: Express T2(n) in
terms of T1(n).
5.
For each pair f, g of functions, indicate whether f(n) = O(g(n)) and/or f(n) = (g(n)).
f(n)
g(n)
f(n) = O(g(n))
f(n) = (g(n))
Justification
2n + log n
n + (log n)
n + log n
n + log n2
n!
nn
3n
3n/2
log n!
log (n+1)!
(log n)log n
6.
Show that, in general, f(n) is not necessarily O(f(n/2)). This shows that, in general, we cannot replace the
factors that go with n by 1 and expect the order to be unchanged.
7.
Use the integration method to find upper and lower bounds (that differ by at most 0.05) for the sum
1
. Hint: Try adding the first few terms explicitly and then use integrals to bound the sum of the
3
i =1 i
remaining terms.
8.
Compute (in terms of n) the total number of times statement S1 is executed. Assume n is even.
for i= 0 to n step 2
for j = i+1 to n
{ S1; }
9.
Work out the time complexity of the following program code (assume n=2m for some positive integer m):
for(int i = n; i > 0; i--)
for(int j = 1; j < n; j *= 2)
for(int k = 0; k < j; k++)
{ // a constant number C of operations }
10. Finding a duplicate element. Given an array of n integers, where each integer from 1 to n1 occurs once
and one integer occurs twice, design an O(n) time and O(1) space algorithm to find the duplicated integer.
11. Finding a missing element. Given an array of n integers from 1 to n, where one integer occurring twice and
one missing, design an O(n) time and O(1) space algorithm to find the missing integer.
12. Design an O(n log n) algorithm for the fixed-sum pair problem. Hint: Utilize searching and sorting.
13. Zero-sum pair problem. Given a sequence of n integers (positive and negative), determine whether there
are two elements whose sum is zero. Give an algorithm for this problem based on problem reduction to the
fixed-sum pair problem. In other words, write a procedure ZeroSum(A,n) that calls the procedure
FixedSum(B,n,K), where B is a sequence derived from A but only contains nonnegative integers.
14. Given two sets S1 and S2 and a real number x, find whether there exist an element in S1 and an element in S2
whose sum is exactly x. Your algorithm should run in O(n log n), where n is the total number of elements in
both sets.
15. Name the run-time error that will result when we try to execute the following code for Mergesort algorithm.
Explain the reason for the error.
void MergeSort(int[] A, int lo, int hi)
{ if (lo hi)
{ int mid =(lo+hi)/2; // integer division
MergeSort(A,lo,mid);
MergeSort(A,mid+1,hi);
Merge(A,lo,mid,hi);
}
}
16. Compute the total number of element comparisons executed by recursive Mergesort on A[1..13]=[13,12, ...,
1]. Hint: Draw and utilize the tree of recursive calls.
17. Write an efficient algorithm to search for a value x in an mn matrix A, where every row and column in A is
sorted. Hint: Use a method MatrixSearch(A, row_lo, row_hi, col_lo, col_hi, x) with the initial call as
MatrixSearch(A, 1, m, 1, n). Argue the correctness of your algorithm, as well as its worst-case time order.
Hint: Consider a binary-search-like procedure.
18. A sequence x1, x2, , xn is said to be cyclically sorted if the smallest element in the sequence is xi for some
unknown i, and the sequence xi, xi+1, , xn , x1, , xi-1 is sorted in increasing order. Given a cyclically
sorted sequence, find the position of the smallest element (assume the smallest element is unique).
Hint: Utilize the idea of binary search by comparing the element at the middle of the sequence with the
rightmost element.
19. Write an algorithm based on binary search for the following problem: Given a sorted integer array and an
integer value x, return the largest element x.
20. Let f be a monotonically increasing function (that is, for any a, b in the domain of f, f(a) f(b) if a b)
with f(0) < 0 and f(n) > 0. Design an algorithm to find the smallest integer i such that f(i) > 0. The algorithm
is constrained to make O(log n) calls to f. Hint: Assuming we know n, maintain an interval [lo, hi] such that
f(lo) < 0 and f (hi) > 0 and apply binary search. If we do not know n, repeatedly compute f(1), f(2), f(4),
until finding a value of n such that f(n) > 0.
21. Given a sorted array of n elements, possibly with duplicates, find the index of the first and last occurrence
of element k in O(log n) time. Hint: Modify binary search.
22. Given a sorted array of n elements, possibly with duplicates, find the number of occurrences of element k in
O(log n) time. Hint: See the previous exercise.
23. Let be A be a finite set of n positive integers. Design an O(n log n) algorithm to verify that:
S A,
x|S |
x S
In other words, if there is a subset S of A such that the sum of the elements in S is less than |S|3, your
algorithm should output no. Otherwise, it should output yes. Argue (informally) that your algorithm is
correct and analyze its running time. Assume that a basic arithmetic operation (addition, multiplication,
comparison) can be done in one unit of time.
24. Let A[1..n] be an array of n distinct numbers. If i < j and A[i] > A[j], then the pair (i, j) is called an inversion
of A. Describe a comparison-based algorithm that sorts A in O(n+k) time where k is the number of
inversions in A. Hint: Think of an in-place version of insertion-sort that, after linear-time preprocessing,
only swaps elements that are inverted.
25. Finding common elements. Given two arrays of integers, design an algorithm to print out all elements that
appear in both lists. The output should be in sorted order. Your algorithm should run in O(n log n) time.
Hint: Utilize Mergesort.
26. [Programming contrasting (n2) versus (n log n)] Develop a program that includes program methods
for Insertion Sort, Mergesort and Quicksort in order to compare their running times on random data with
varying degrees of redundancy, and summarize the results in a table similar to the fllowing one. What
conclusions can you draw from the experiment results?
x: (U/x) Dist.
Input Size in
Millions
Insertion Sort
Execution Time (msec)

Mergesort
Quicksort
10
1
20
40
10
20
M
M
To generate data with varying degrees of redundancy, use the following FillArray() method. This method
fills an array of size n with integers in the range [0,max] using uniform random distribution, where
max=n/x. This scheme is simply referred to as U/x, where x can be thought of as a repetition (redundancy)
factor.
void FillArray(int[] A, int n, int repfactor)

{ Random r = new Random();
int max = n/repfactor;
for(int i = 1; i <= n; i++)
A[i] = r.Next(max);
}

Introduction To Algorithms - Design Techniques and Analysis M. Alsuwaiyel - Chapter - 1

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Introduction To Algorithms - Design Techniques and Analysis M. Alsuwaiyel - Chapter - 1

Diunggah oleh

Hak Cipta:

Format Tersedia

Table of Contents

1. INTRODUCING ALGORITHMS ........................................................................................ 11

2. FUNDAMENTAL DATA STRUCTURES........................................................................... 74

3.1 Structure of a Proof by Induction............................................................................................................................... 147

4. DIVIDE AND CONQUER................................................................................................. 197

5. DYNAMIC PROGRAMMING........................................................................................... 234

6. GREEDY TECHNIQUE ................................................................................................... 280

7. GRAPH ALGORITHMS................................................................................................... 327

7.6.2 Relating Bipartite Matching to Vertex Cover ......................................................................................................... 359

8. BACKTRACKING ........................................................................................................... 378

9. THEORY OF NP.............................................................................................................. 415

10. ITERATIVE LOCAL SEARCH ...................................................................................... 454

10.3.3 Graph Coloring ..................................................................................................................................................... 465

11. PROGRAMMING FUNDAMENTALS AND TECHNIQUES........................................... 470

12. PROGRAMMING EXCURSIONS.................................................................................. 569

BIBLIOGRAPHY ................................................................................................................. 603

Ch.2 (skip 2.2.2, 2.4.4, 2.4.5, 2.4.6, 2.5)

Ch.3 (skip 3.10)

Ch. 4 (skip 4.7)

Ch. 5 (skip 5.9.3, 5.11)

Ch. 6 (skip 6.3.2, 6.4.3, 6.4.4)

Ch. 7 (skip 7.5, 7.6)

Ch. 8 (skip 8.6, 8.7)

Ch. 9 (skip 9.4, 9.5)

1.1.1 Describing Algorithms

Textual and pictorial description

Textual and Pictorial Description

CSharp Program Statement

Alternative form used in Pseudocode

if (a==b) S1; else S2;

if a=b then S1 else S2

for(int i=1; i <= n; i++)

Expressing an Algorithm via Program Code: Procedural versus Object-Oriented

1.1.2 Algorithm-Design Guidelines

1.1.3 Proof Techniques

1.2 Problem-Solving Techniques

Binary strings of length n1 with no consecutive 1s

Binary strings of length n2 with no consecutive 1s

// map a negative integer to even positive

1.3 Running-Time and Space Complexity Analysis

Arithmetic and logical operations on primitive data types.

Examples of Nonelementary Operations:

Computing the maximum value in an array of numbers.

determine if there are two elements whose sum is K.

Input: An integer array A[1..n] and a positive integer K

Listing 1.1 An algorithm for the fixed-fum pair problem.

Total Cost of all Operations

Set (or increment) i and test i n1

Set (or increment) j and test j n

Sum A[i], A[j] and test if sum = 0

Table 1.2 A detailed worst-case analysis of running time.

can be rewritten as (the inner sum

Thus, the total number of operations is as follows:

[n1]C1 + [n(n1)/2]C2 + [n(n1)/2]C3+0+ C5.

Best-Case Time Complexity

i occuring) * (number of operations for case i )

all input cases i of size n

1.3.1 Asymptotic Notations

If we let aMax = maximum of {ak, ak-1, ..., a0}, then

The Bad Side and Good Side of the -notation

Example 1.13 Show that log n! is (n log n).

log n! = log (n(n1) 1)

If f(x) is increasing, then we have