Python: Journey from

1ovice to Expert

Learn core concepts of Python and unleash its power

to script highest quality Python programs.

A course in three modules

BIRMINGHAM - MUMBAI
Python: Journey from 1ovice to (xpert

Copyright 2016 Packt Publishing

All rights reserved. No part of this course may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.

Every effort has been made in the preparation of this course to ensure the accuracy
of the information presented. However, the information contained in this course
is sold without warranty, either express or implied. Neither the authors, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this course.

Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this course by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.

Published on: August 2016

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78712-076-1
www.packtpub.com
 Credits

Authors Content Development Editor

Fabrizio Romano Onkar :ani
Dusty Phillips
Rick van Hattem Graphics
Abhinash Sahu

Reviewers
Simone Burol Production Coordinator
Melwyn Dsa
Julio Vicente Trigo Guijarro
Veit Heller
AMahdy AbdElAziz
Grigoriy Beziuk
Krishna Bharadwaj
Justin Cano
Anthony Petitbois
Claudio Rodriguez
Randall Degges
Dave de Fijter
I. de Hoogt
 Preface
Python is a dynamic programming language. It is known for its high readability
DQGKHQFHLWLVRIWHQWKHUVWODQJXDJHOHDUQHGE\QHZSURJUDPPHUV3\WKRQEHLQJ
multi-paradigm, it can be used to achieve the same thing in different ways and it is
compatible across different platforms. Coding in Python minimizes development
time and increases productivity in comparison to other languages. Clean,
maintainable code is easy to both read and write using Python's clear, concise syntax.

What this learning path covers

Module 1, Learning Python, This module begins by exploring the essentials of
programming, data structures and teaches you how to manipulate them. It then
PRYHVRQWRFRQWUROOLQJWKHRZRIDSURJUDPDQGZULWLQJUHXVDEOHDQGHUURUSURRI
code. You will then explore different programming paradigms that will allow you to
QGWKHEHVWDSSURDFKWRDQ\VLWXDWLRQDQGDOVROHDUQKRZWRSHUIRUPSHUIRUPDQFH
optimization as well as effective debugging. Throughout, the module steers you
through the various types of applications, and it concludes with a complete mini
website built upon all the concepts that you learned.

Module 2, Python 3 Object-Oriented Programming, Second Edition, You will learn

how to use the Python programming language to clearly grasp key concepts from the
object-oriented paradigm. This modules fully explains classes, data encapsulation,
inheritance, polymorphism, abstraction, and exceptions with an emphasis on when
you can use each principle to develop a well-designed software. You'll get an in-
depth analysis of many common object-oriented design patterns that are more
suitable to Python's unique style. This module will not just teach Python syntax,
EXWZLOODOVREXLOG\RXUFRQGHQFHLQKRZWRSURJUDPDQGFUHDWHPDLQWDLQDEOH
applications with higher level design patterns.

[i]
Preface

Module 3, Mastering Python, This module is an authoritative guide that will help
you learn new advanced methods in a clear and contextualized way. It starts off
E\FUHDWLQJDSURMHFWVSHFLFHQYLURQPHQWXVLQJYHQYLQWURGXFLQJ\RXWRGLIIHUHQW
Pythonic syntax and common pitfalls before moving on to cover the functional
features in Python. It covers how to create different decorators, generators, and
metaclasses. It also introduces you to functools.wraps and coroutines and how
they work. Later on you will learn to use asyncio module for asynchronous clients
and servers. You will also get familiar with different testing systems such as
py.test, doctest, and unittest, and debugging tools such as Python debugger and
faulthandler. You will learn to optimize application performance so that it works
HIFLHQWO\DFURVVPXOWLSOHPDFKLQHVDQG3\WKRQYHUVLRQV)LQDOO\LWZLOOWHDFK\RX
how to access C functions with a simple Python call. By the end of the module, you
will be able to write more advanced scripts and take on bigger challenges.

What you need for this learning path

Module 1:

You are encouraged to follow the examples in this module. In order to do so, you
will need a computer, an Internet connection, and a browser. The module is written
in Python 3.4, but it should also work with any Python 3.* version. It has written
instructions on how to install Python on the three main operating systems used
today: Windows, Mac, and Linux. This module also explained how to install all
the extra libraries used in the various examples and provided suggestions if the
UHDGHUQGVDQ\LVVXHVGXULQJWKHLQVWDOODWLRQRIDQ\RIWKHP1RSDUWLFXODUHGLWRULV
required to type the code; however, module suggest that those who are interested in
following the examples should consider adopting a proper coding environment.

Module 2:

All the examples in this module rely on the Python 3 interpreter. Make sure you
are not using Python 2.7 or earlier. At the time of writing, Python 3.4 was the latest
release of Python. Most examples will work on earlier revisions of Python 3, but you
are encouraged to use the latest version to minimize frustration. All of the examples
should run on any operating system supported by Python.

If this is not the case, please report it as a bug. Some of the examples need a working
Internet connection. You'll probably want to have one of these for extracurricular
research and debugging anyway! In addition, some of the examples in this module
rely on third-party libraries that do not ship with Python. These are introduced
within the module at the time they are used, so you do not need to install them in
advance. However, for completeness, here is a list:

[ ii ]
 Preface

 pip
 requests
 pillow
 bitarray

Module 3:

The only hard requirement for this module is a Python interpreter. A Python 3.5 or
newer interpreter is recommended, but many of the code examples will function
in older Python versions, such as 2.7, with a simple from __future__ import print_
VWDWHPHQWDGGHGDWWKHWRSRIWKHOH$GGLWLRQDOO\&KDSWHU([WHQVLRQVLQ&
&6\VWHP&DOOVDQG&&/LEUDULHVUHTXLUHVD&&FRPSLOHUVXFKDV*&&
9LVXDO6WXGLRRU;&RGH$/LQX[PDFKLQHLVE\IDUWKHHDVLHVWWRH[HFXWHWKH&&
examples, but these should function on Windows and OS X machines without too
much effort as well.

Who this learning path is for

This course is meant for programmes who wants learn Python programming from a
basic to an expert level. The course is mostly self-contained and introduces Python
Programming to a new reader and can help him become an expert in this trade.
Intended for students and practitioners from novice to experts.

Reader feedback
)HHGEDFNIURPRXUUHDGHUVLVDOZD\VZHOFRPH/HWXVNQRZZKDW\RXWKLQNDERXW
this coursewhat you liked or disliked. Reader feedback is important for us as it
helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail feedback@packtpub.com, and mention

the course's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing
or contributing to a course, see our author guide at www.packtpub.com/authors.

Customer support
Now that you are the proud owner of a Packt course, we have a number of things to
help you to get the most from your purchase.

[ iii ]
Preface

Downloading the example code

<RXFDQGRZQORDGWKHH[DPSOHFRGHOHVIRUWKLVFRXUVHIURP\RXUDFFRXQWDW
http://www.packtpub.com. If you purchased this course elsewhere, you can visit
http://www.packtpub.com/supportDQGUHJLVWHUWRKDYHWKHOHVHPDLOHGGLUHFWO\
to you.

<RXFDQGRZQORDGWKHFRGHOHVE\IROORZLQJWKHVHVWHSV

1. Log in or register to our website using your e-mail address and password.
2. Hover the mouse pointer on the SUPPORT tab at the top.
3. Click on Code Downloads & Errata.
4. Enter the name of the course in the Search box.
5. 6HOHFWWKHFRXUVHIRUZKLFK\RX
UHORRNLQJWRGRZQORDGWKHFRGHOHV
6. Choose from the drop-down menu where you purchased this course from.
7. Click on Code Download.

<RXFDQDOVRGRZQORDGWKHFRGHOHVE\FOLFNLQJRQWKHCode Files button on the

course's webpage at the Packt Publishing website. This page can be accessed by
entering the course's name in the Search box. Please note that you need to be logged
in to your Packt account.

2QFHWKHOHLVGRZQORDGHGSOHDVHPDNHVXUHWKDW\RXXQ]LSRUH[WUDFWWKHIROGHU
using the latest version of:

 :LQ5$5=LSIRU:LQGRZV
 =LSHJL=LS8Q5DU;IRU0DF
 =LS3HD=LSIRU/LQX[

The code bundle for the course is also hosted on GitHub at https://github.com/
PacktPublishing/Python-Journey-from-Novice-to-Expert. We also have
other code bundles from our rich catalog of books and videos available at https://
github.com/PacktPublishing/. Check them out!

[ iv ]
 Preface

Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
GRKDSSHQ,I\RXQGDPLVWDNHLQRQHRIRXUFRXUVHVPD\EHDPLVWDNHLQWKHWH[W
or the codewe would be grateful if you could report this to us. By doing so, you
can save other readers from frustration and help us improve subsequent versions
RIWKLVFRXUVH,I\RXQGDQ\HUUDWDSOHDVHUHSRUWWKHPE\YLVLWLQJhttp://www.
packtpub.com/submit-errata, selecting your course, clicking on the Errata
Submission Form link, and entering the details of your errata. Once your errata are
YHULHG\RXUVXEPLVVLRQZLOOEHDFFHSWHGDQGWKHHUUDWDZLOOEHXSORDGHGWRRXU
website or added to any list of existing errata under the Errata section of that title.

To view the previously submitted errata, go to https://www.packtpub.com/books/

content/supportDQGHQWHUWKHQDPHRIWKHFRXUVHLQWKHVHDUFKHOG7KHUHTXLUHG
information will appear under the Errata section.

Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all
media. At Packt, we take the protection of our copyright and licenses very seriously.
If you come across any illegal copies of our works in any form on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.

Please contact us at copyright@packtpub.com with a link to the suspected pirated

material.

We appreciate your help in protecting our authors and our ability to bring you
valuable content.

Questions
If you have a problem with any aspect of this course, you can contact us at
questions@packtpub.com, and we will do our best to address the problem.

[v]
Module 1: Learning Python

Chapter 1: Introduction and First Steps Take a Deep Breath 3

A proper introduction 4
Enter the Python 6
About Python 7
What are the drawbacks? 9
Who is using Python today? 10
Setting up the environment 10
Installing Python 11
How you can run a Python program 19
How is Python code organized 23
Python's execution model 27
Guidelines on how to write good code 35
The Python culture 36
A note on the IDEs 37
Summary 38
Chapter 2: Built-in Data Types 39
Everything is an object 39
Mutable or immutable? That is the question 40
Numbers 42
Immutable sequences 48
Mutable sequences 52
Set types 57
Mapping types dictionaries 59
The collections module 64
Final considerations 68
Summary 72

i
Table of Contents

Chapter 3: Iterating and Making Decisions 75

Conditional programming 76
Looping 80
Putting this all together 93
A quick peek at the itertools module 99
Summary 102
Chapter 4: Functions, the Building Blocks of Code 103
Why use functions? 104
Scopes and name resolution 109
Input parameters 112
Return values 123
A few useful tips 126
Recursive functions 127
Anonymous functions 128
Function attributes 129
Built-in functions 130
2QHQDOH[DPSOH 
Documenting your code 132
Importing objects 133
Summary 136
Chapter 5: Saving Time and Memory 137
PDS]LSDQGOWHU 
Comprehensions 144
Generators 150
Some performance considerations 161
Don't overdo comprehensions and generators 164
Name localization 169
Generation behavior in built-ins 170
One last example 171
Summary 173
Chapter 6: Advanced Concepts OOP, Decorators, and Iterators 175
Decorators 175
Object-oriented programming 184
Writing a custom iterator 212
Summary 213

ii
 Table of Contents

Chapter 7: 7HVWLQJ3UROLQJDQG'HDOLQJZLWK([FHSWLRQV 

Testing your application 216
Test-driven development 235
Exceptions 237
3UROLQJ3\WKRQ 
Summary 247
Chapter 8: The Edges GUIs and Scripts 249
First approach scripting 252
Second approach a GUI application 260
Where do we go from here? 275
Summary 278
Chapter 9: Data Science 279
IPython and Jupyter notebook 280
Dealing with data 283
Where do we go from here? 309
Summary 310
Chapter 10: Web Development Done Right 311
What is the Web? 311
How does the Web work? 312
The Django web framework 313
A regex website 316
The future of web development 338
Summary 342
Chapter 11: Debugging and Troubleshooting 343
Debugging techniques 344
Troubleshooting guidelines 357
Summary 358
Chapter 12: Summing Up A Complete Example 359
The challenge 359
Our implementation 360
Implementing the Django interface 360
Implementing the Falcon API 387
Where do you go from here? 404
Summary 405
A word of farewell 406

iii
Table of Contents

Module 2: Python 3 Object-Oriented Programming

Chapter 1: Object-oriented Design 409

Introducing object-oriented 409
Objects and classes 411
Specifying attributes and behaviors 413
Hiding details and creating the
public interface 417
Composition 419
Inheritance 422
Case study 426
Exercises 433
Summary 434
Chapter 2: Objects in Python 435
Creating Python classes 435
Modules and packages 445
Organizing module contents 451
Who can access my data? 454
Third-party libraries 456
Case study 457
Exercises 466
Summary 466
Chapter 3: When Objects Are Alike 467
Basic inheritance 467
Multiple inheritance 473
Polymorphism 483
Abstract base classes 486
Case study 490
Exercises 503
Summary 504
Chapter 4: Expecting the Unexpected 505
Raising exceptions 506
Case study 522
Exercises 531
Summary 532
Chapter 5: When to Use Object-oriented Programming 533
Treat objects as objects 533
Adding behavior to class data with properties 537
Manager objects 546

iv
 Table of Contents
Case study 553
Exercises 561
Summary 562
Chapter 6: Python Data Structures 563
Empty objects 563
Tuples and named tuples 565
Dictionaries 568
Lists 575
Sets 581
Extending built-ins 585
Queues 590
Case study 596
Exercises 602
Summary 603
Chapter 7: Python Object-oriented Shortcuts 605
Python built-in functions 605
An alternative to method overloading 613
Functions are objects too 621
Case study 627
Exercises 634
Summary 635
Chapter 8: Strings and Serialization 637
Strings 637
Regular expressions 652
Serializing objects 660
Case study 668
Exercises 673
Summary 675
Chapter 9: The Iterator Pattern 677
Design patterns in brief 677
Iterators 678
Comprehensions 681
Generators 687
Coroutines 692
Case study 699
Exercises 706
Summary 707

v
Table of Contents

Chapter 10: Python Design Patterns I 709

The decorator pattern 709
The observer pattern 715
The strategy pattern 718
The state pattern 721
The singleton pattern 728
The template pattern 733
Exercises 737
Summary 737
Chapter 11: Python Design Patterns II 739
The adapter pattern 739
The facade pattern 743
7KH\ZHLJKWSDWWHUQ 
The command pattern 749
The abstract factory pattern 754
The composite pattern 759
Exercises 763
Summary 764
Chapter 12: Testing Object-oriented Programs 765
Why test? 765
Unit testing 768
Testing with py.test 776
Imitating expensive objects 786
How much testing is enough? 790
Case study 793
Exercises 799
Summary 800
Chapter 13: Concurrency 801
Threads 802
Multiprocessing 807
Futures 814
AsyncIO 817
Case study 826
Exercises 833
Summary 834

vi
 Table of Contents

Module 3: Mastering Python

Chapter 1: Getting Started One Environment per Project 837

Creating a virtual Python environment using venv 838
Bootstrapping pip using ensurepip 843
Installing C/C++ packages 844
Summary 847
Chapter 2: Pythonic Syntax, Common Pitfalls, and Style Guide 849
Code style or what is Pythonic code? 850
Common pitfalls 871
Summary 884
Chapter 3: Containers and Collections Storing Data the Right Way 885
Time complexity the big O notation 886
Core collections 887
Advanced collections 898
Summary 915
Chapter 4: Functional Programming Readability Versus Brevity 917
Functional programming 918
list comprehensions 918
dict comprehensions 921
set comprehensions 922
lambda functions 922
functools 925
itertools 931
Summary 938
Chapter 5: Decorators Enabling Code Reuse by Decorating 939
Decorating functions 940
Decorating class functions 952
Decorating classes 961
Useful decorators 966
Summary 976
Chapter 6: *HQHUDWRUVDQG&RURXWLQHV,QQLW\2QH6WHSDWD7LPH 
What are generators? 978
Coroutines 990
Summary 1002

vii
Table of Contents

Chapter 7: Async IO Multithreading without Threads 1003

Introducing the asyncio library 1004
Summary 1024
Chapter 8: Metaclasses Making Classes (Not Instances) Smarter 1025
Dynamically creating classes 1026
Abstract classes using collections.abc 1030
Automatically registering a plugin system 1037
Order of operations when instantiating classes 1043
6WRULQJFODVVDWWULEXWHVLQGHQLWLRQRUGHU 
Summary 1051
Chapter 9: Documentation How to Use Sphinx and reStructuredText 1053
The reStructuredText syntax 1054
The Sphinx documentation generator 1069
Documenting code 1085
Summary 1091
Chapter 10: Testing and Logging Preparing for Bugs 1093
Using examples as tests with doctest 1094
Testing with py.test 1110
Mock objects 1138
Logging 1141
Summary 1153
Chapter 11: Debugging Solving the Bugs 1155
Non-interactive debugging 1156
Interactive debugging 1168
Summary 1180
Chapter 12: Performance Tracking and Reducing Your Memory and CPU
Usage 1181
What is performance? 1182
Timeit comparing code snippet performance 1183
F3UROHQGLQJWKHVORZHVWFRPSRQHQWV 
/LQHSUROHU 
Improving performance 1199
Memory usage 1205
Performance monitoring 1218
Summary 1219

viii
 Table of Contents

Chapter 13: Multiprocessing When a Single CPU Core Is Not Enough1221

Multithreading versus multiprocessing 1221
Hyper-threading versus physical CPU cores 1224
Creating a pool of workers 1226
Sharing data between processes 1228
Remote processes 1229
Summary 1238
Chapter 14: Extensions in C/C++, System Calls, and C/C++ Libraries 1239
Introduction 1239
Calling C/C++ with ctypes 1242
CFFI 1249
Native C/C++ extensions 1252
Summary 1263
Chapter 15: Packaging Creating Your Own Libraries or Applications 1265
Installing packages 1265
Setup parameters 1266
Packages 1270
Entry points 1270
Package data 1274
Testing packages 1275
C/C++ extensions 1279
Wheels the new eggs 1282
Summary 1285
Bibliography 1287

ix