Anda di halaman 1dari 196

A comprehensive masterclass to

becoming an instant expert

Welcome to

Linux&
OpenSource
Despite heavy competition from other operating systems, Linux has not only
managed to survive but it has thrived. The recent success of the Raspberry
Pi has provided an added impetus for coders, developers and everyday
users to take advantage of the many customisable services and utilities that
the platform offers. Linux & Open Source Genius Guide Volume 5 is
guaranteed to keep any ardent Linux user busy for weeks with the number of
useful tutorials, masterclasses and guides on offer. Weve covered a wide
range of fun and useful tips & tricks from creaing your own Windows 8.1 VM
to taking pictures and video with the Raspberry Pi Camera. Youll nd tons of
useful guides that include dual-booting Linux on any device and an ultimate
Qt masterclass. Weve even featured the essential Linux software for your
everyday needs. So, re up your machine, turn the page, and follow our
experts as they guide you through all areas of using and developing Linux.

Linux&
OpenSource
Imagine Publishing Ltd
Richmond House
33 Richmond Hill
Bournemouth
Dorset BH2 6EZ
 +44 (0) 1202 586200
Website: www.imagine-publishing.co.uk
Twitter: @Books_Imagine
Facebook: www.facebook.com/ImagineBookazines

Publishing Director
Aaron Asadi
Head of Design
Ross Andrews
Production Editor
Hannah Kelly
Senior Art Editor
Greg Whitaker
Designer
Abbi Denney
Photographer
James Sheppard
Printed by
William Gibbons, 26 Planetary Road, Willenhall, West Midlands, WV13 3XT
Distributed in the UK, Eire & the Rest of the World by:
Marketforce, Blue Fin Building, 110 Southwark Street, London, SE1 0SU
Tel 0203 148 3300 www.marketforce.co.uk
Distributed in Australia by:
Network Services (a division of Bauer Media Group), Level 21 Civic Tower, 66-68 Goulburn Street,
Sydney, New South Wales 2000, Australia Tel +61 2 8667 5288
Disclaimer
The publisher cannot accept responsibility for any unsolicited material lost or damaged in the
post. All text and layout is the copyright of Imagine Publishing Ltd. Nothing in this bookazine may
be reproduced in whole or part without the written permission of the publisher. All copyrights are
recognised and used specifically for the purpose of criticism and review. Although the bookazine has
endeavoured to ensure all information is correct at time of print, prices and availability may change.
This bookazine is fully independent and not affiliated in any way with the companies mentioned herein.
Linux & Open Source Genius Guide Volume 5 2014 Imagine Publishing Ltd

Part of the

bookazine series

Contents

Your guide to whats inside

Tips & Tricks


18 Host your own webmail server

60 SSH from a web browser

22 Network scanning using Nmap

64 Master version control with Git

26 Create the perfect Windows 8.1


virtual machine

68 Take pictures and video with the


Raspberry Pi Camera

30 Understand regular expressions

70 Record slow-motion videos with


the Raspberry Pi

34 Use TeX and LaTex to create


beautiful documents

72 Browse privately with Onion Pi

38 Set up a wireless access point


with a Raspberry Pi
40 Configure your PC for remote
maintenance
44 Build your own private cloud with
ownCloud
48 Monitor network traffic
52 Access your network anywhere
with a VPN
56 Use gnuplot for drawing

Page 8r

Build you
own distro

6 Linux & Open Source Genius Guide

Masterclass
78 Dual-boot Linux on any device
88 Get started with Android
Studio
96 Get to grips with Go Program!
100 Qt masterclass
112 Program with Erlang

Linux & Open Source

Genius Guide

Developer guide

Essential software

120 Write once, use anywhere


with Qt

160 Presentation software

124 Answer real-world questions


with Gt

168 Cloud storage services

164 Music players

128 Handle Twitters three-legged


oAuth process

172 Integrated development

132 Create dynamic templates


with Flask, Jinja2 and Twitter

180 System recovery

176 Ultimate IRC client

136 Build a blog with Django

184 Video editors

142 Create a real-time CPU


meter with Node.js
and Duino

188 Photo managers

146 Build tic-tac-toe


with Kivy
150 Get started with Gulp.js
154 Master Vim in easy steps

Page
78
D

ual-b
Linux oot
on a
devic ny
e

Linux & Open Source Genius Guide 7

Build your own distro

r creating
fo
e
bl
la
ai
av
ds
ho
et
m
nt
re
ffe
di
e
Discover th
everything
ve
ha
d
an
ro
st
di
d
ise
om
st
cu
n
ow
your
working your way in no time

here a few reasons why you


might want to build your own
distribution. You might want to
build a custom install CD to match
the policy of your organisation. For
example, a GNOME desktop with
Chrome as the web browser might be the standard
desktop where you work. That touches on another
motivation for wanting to create a customised
installer: if the creator of the distribution makes a
decision that you don't like. Canonical's decision to
switch to its own UI, Unity, ranks amongst its most
controversial decisions. However, by using some of
the methods that we explore here, you could
create a distribution that is standard
Ubuntu, but with a traditional desktop
that you are more comfortable with.

8 Linux & Open Source Genius Guide

There are other, niche reasons for wanting to


build your own distribution too. You might need to
put something small and lightweight together for an
older computer. You might need to build a live media
ISO that you are able to carry around with you and
to bring your favourite set of tools to bear when you
need them. You have the exibility to create your
own distro whatever your needs.
The methods of creating a custom
distribution are varied, but they can be divided
into two main categories: you can modify a
running distribution and then distribute it, or
you can modify the installation ISO (called
'remixing') so that it installs your modied
distribution in the way you have specied.
We're going to take a look at four different
approaches.

Linux & Open Source

Genius Guide

Remastersys
Remastersys is a tool that extracts the
conguration from a running Ubuntu
or Debian installation and then turns
this into an installable ISO image. This
means that you carry out the customisation using
the standard tools that you normally use, such
as the package management system and GUI
conguration tools. When you have everything set
up the way you want it, you can clone the system and
deploy it. Additionally, you can use Remastersys to
make a clone of a working system.

Pros
You can use the
standard tools
to congure a
distribution

Cons
Needs the
expertise to
carry out the
customisations,
doesn't work on
all distros, has an
uncertain future

Fetch Remastersys
The development status of Remastersys is currently
in transition. At time of writing, the best policy is to
visit the Remastersys website and to cut and paste
the repository details from there. For example, if you
are using Ubuntu 13.10, download the GPG key and
add it from the command line with:

Tip

wget -O - http://www.remastersys.com/ubuntu/
remastersys.gpg.key
sudo apt-key add remastersys.gpg.key
then add

deb http://www.remastersys.com/ubuntu
precise main
to the end of /etc/apt/sources.lst by invoking a text
editor as root.
Following that, type sudo apt-get install
remastersys remastersys-gui in order to install
Remastersys and its GUI.

Q Booting from the installation ISO

The 'Start the in


staller
directly' option on
the
GRUB menu is m
ore
dependable than
installing via the
live
CD option

Using Remastersys

The future of the GUI portion of the


project seems less certain

When you have the installation set up the way


you want it, launch Remastersys by typing sudo
remastersys-gui. The rst option we need to visit is
the customisation page which is reached by clicking
on the Customise button. From here, you can change
branding options such as the various splash images.
From within this page, click on Copy Settings. This
takes you to a further page on which you can select
the user whose settings will be copied to /etc/skel/. In
other words, these are the settings that will become
the defaults for all new users on the new system. If

you skip this stage, new users will simply have the
default settings for the distribution.
Finally, build the installation ISO simply by clicking
on the Distribution button on the main menu page.
The ISO is deposited into the /home/remastersys/
folder. Use networking to transfer the ISO le to the
outside of the VM. We usually install Filezilla and
transfer to a local FTP server. You can now boot the
ISO on the target machine and carry out a regular
Ubuntu installation.

Q Copying the skeleton information for new users

Q Building the ISO

The future of
Remastersys
The long-time developer of Remastersys
recently decided to give up development.
Fortunately, he has chosen to release the
source code so that other developers can
take up the mantle. The future of the GUI
portion of the project seems less certain,
but Remastersys is also fully functional
from the command line. For the moment,
the binaries of Remastersys are still freely
available from the developer's web site
(www.remastersys.com).
The situation is constantly in ux,
so search around for the latest forks.
The System Imager project, which uses
Remastersys, is a good source of up-todate information, and can be found here:
http://system-imaging.blogspot.co.uk.

Linux & Open Source Genius Guide 9

Build your own distro

SUSE Studio
SUSE Studio allows you to build a
customised SUSE Linux installation
using a web interface. Although it's
easy to use, that doesn't mean it has
compromised on options.
Initially, you choose a base template such as
KDE Desktop or Server. From this point, you begin
the conguration properly. The rst tab is labelled
Software, which is where you choose software
packages with an interface that is categorised and
searchable.

Pros
Could hardly
be easier to
use, sharing of
appliances is built
into the site

Cons
Build speed varies,
you might hit a wall
with really complex
customisations

Tip

You can uplo


RPMs that arenad
the standard re 't in
using the softwpos
are
page

Q Adding in some custom branding

Q Waiting for the ISO to build

Example deployment:
Business desktop
Here we're going to put together an example
appliance. In this case, the appliance will be a
business desktop that based around GNOME. We'll
add a few customisations as we go along, and we
want to nish up with an installable ISO that we can
use for deployment.
Begin by setting up an account on the SUSE Studio
website (http://susestudio.com). You do this by
following the 'Sign In Or Create An Account' link on the
front page, and it is possible to use one of your existing
social networking accounts such as Facebook or
OpenID if you prefer.
Once you have an account, click on 'Create New
Appliance...'. On the next screen, choose the GNOME
Desktop base template, making sure that you are
selecting from the templates that relate to the latest
version of SUSE Linux. Scroll down to the bottom of
the window to choose your architecture and then
give your appliance a meaningful name. Click OK, and
after a short delay, we can start honing the appliance
to match our own requirements.

Start customising
As this is a business desktop, let's add LibreOfce
to it. To do this, select the Software tab and type
the word 'libre' into the search box. The search
is real-time, so you should soon be presented
with a list of matches. Note that they are sorted
by popularity and the package called LibreOfce
should be at the top of the list. Click the '+add'
button to add this package. For a big software suite
such as LibreOfce, it may take a few moments
for the interface to register all of the needed
dependencies. Add Firefox too. Staying in the
Software tab for moment, it's worth noting that you
are able to add extra repositories and even custom
RPM packages.
We'll select the localisation options next.
Proceed to the Conguration tab and select the
General sub-tab. In here, select the English (UK)
option as the language and keyboard layout and
Europe and United Kingdom as the region and time
zone respectively.
Note that you could also have selected Ask on
rst boot for any of these options as well.

It's worth noting that you are able


to add extra repositories and even
custom RPM packages
10 Linux & Open Source Genius Guide

Q Selecting software packages to begin


customising the desktop

Q Booting from the installation ISO

Linux & Open Source

Genius Guide

Tip

The build proces


s and
download speed
vary
according to serv
er
load so get things
ready
before you need
them

Q This screen shows us conguring details such as the users and network settings in the Conguration>General page

We'll leave the network options as they are, but this


is where you would disable DHCP and specify a static
IP address for the workstation, or disable the firewall if
you needed to. At the bottom of the page, we can see a
list of users and groups. It's a good idea to change the
root password from the default. Now click on 'Add new
user...' and create a standard user who is a member of
the Users group.
On the Personalize sub-tab of the Conguration
page, we can add some custom branding. This might
t in well with the policies of your organisation, and it
is also extremely handy for at-a-glance identication
of a desktop within a busy IT environment.
The Files tab is worth a visit if you need to
add custom files to the distribution. You can
add single files or archives. For example, if you
wanted to add a file to the desktop of every
new user, you should upload it and specify
that it should be placed in '/etc/skeleton/
Desktop'. If you wanted to place a file within the
home directory of the user that you have created called
John, add it to '/home/john/'.
As a finishing touch, pop into the
Configuration>Desktop page. Tick the box to

automatically log the user in. Add the command


firefox to the Autostart desktop user log-in section to
automatically start Firefox. Opinions vary, but these
options allow the user to get straight to work.

Build the ISO


The options within the Build tab are particularly
interesting because they allow you to specify the
output format of your custom build. This means
that you don't necessarily have to carry out a full
installation in order to use your custom build.
For example, you can create a virtual machine that
will directly boot within a visualiser. If you want to
work like this, you will probably need to skip back to
the Appliance sub-tab within the Conguration tab
to dene the parameters of the VM. Here, you can
choose options such as allocated memory and set
up the LVM partition arrangement. Apart from the
various VM environments you can directly create, you
can also create a traditional ISO installer, a hard disk
image or a live CD/USB image.
In order to create a traditional installation ISO,
select 'Preload ISO (.iso)' in the Default format
and click on the 'Build' icon. This can take a few

Q The SUSE Studio login page

minutes to complete, depending on how large and


complicated your custom image is. Although it may
take several minutes for your image to build, once
built, your appliances remain on the site and can be
downloaded without delay. The nal tab, Share, is an
intriguing function that allows you share your nished
appliances with other people.
Boot the nished ISO. Conrm that you wish to
erase all data on the hard disk when prompted.

Linux & Open Source Genius Guide 11

Build your own distro

Ubuntu minimal
installation
Canonical provides a minimal Ubuntu
install CD. It's smaller than the regular
installation ISO and it installs a minimal
version of the distribution. At its most
basic, it gives the user a command line, network
connectivity and not much else. From this barebones beginning, it's possible to selectively add
components while leaving out most of the cruft that
tends to come with a standard distribution.
We're going to work from within a virtual machine
for safety and convenience. In our case, we're
going to use Oracle VirtualBox but any of the major
virtualisers will work. Once we have it set up the way
we want it, we can use Remastersys to turn it into an
ISO that can be distributed. We can then transfer this
ISO from within the VM to an FTP server.

Pros
Excellent way of
keeping the distro
standard yet
minimal too

Cons
Time consuming
to carry out from
start to nish

Q Creating a blank hard disk image in VirtualBox

Example deployment:
Minimal Openbox Desktop
Fetch the installation media from http://tinyurl.com/
ygawub and create a new virtual machine. 512MB is a
sensible minimum when allocating memory, but more
memory can also help greatly with speeding things
up. An 8GB hard disk file should be adequate for most
people's requirements. It's usually worth allocating as
many CPU cores to the VM as you can.

Tip

Add the kernel


extensions for your
virtualiser to enable
things like cut and
paste between host
and guest

Begin the customisation


Once you have booted the ISO from within the VM,
begin by lling in the localisation details using the
text mode interface. Next, the installer will attempt
to nd your network using DHCP. Following the
network detection phase, ll in a hostname that
will be used to identify this computer on the
network. Once you've done this, select a mirror that
is geographically close to your location.
The installer should now begin to download
packages. Once the packages have come
through the network, set up the username
for the standard user. You should be able to use
common sense to ask the question that comes
next, regarding your time zone and default user
and password.

Q Starting a minimal installation

Clonezilla
Q Fetching the initial set of base packages

You'll usually find it's worth


allocating as many CPU cores to the
VM as you can
12 Linux & Open Source Genius Guide

Clonezilla (http://clonezilla.org) is a live


CD that can be used to make complete
system backups. It uses an algorithm
that avoids copying the empty space on
a hard disk and produces les that are
as small as possible. You might want to
use Clonezilla as an alternative method to
distribute a customised distribution as a
hard disk image.

Linux & Open Source

Genius Guide

Q Fetching the Ubuntu MinimalCD ISO le

Tip

Using a minimal
install
CD like this will cr
eate
quite a lot of netw
ork
traffic while it is
pulling packages
through
Q Use automatic partitioning

When prompted, allow the installer to allocate


the disk partitioning by selecting the 'Guided - use
entire disk' option. Conrm that you want to write
to the disk when prompted. The actual layout
that you use now isn't important as we will be
producing an ISO that will carry out the installation
of our custom distribution from scratch. Once the
partitioning has completed, the installer will fetch
the packages need for the base installation and
begin installing them.

Customisation decisions
When the base installation is complete, you will
be presented with the Software selection menu.
At this point you have to make a decision. If you
want to, you can select one or more of the provided
templates. For example, you could select Kubuntu
desktop option and have a fairly complete desktop
system from the beginning.
There are other options available to establish
a LAMP web server or a Mythbuntu media
system, and many others too. Most of the rest
of this tutorial assumes that you don't select
any of these options so that we can customise
completelyfrom scratch.
More downloading and installation follows.
Conrm that you want GRUB installed to the MBR
when asked. This brings us to the end of the initial
installation phase. Next, eject the ISO and reboot
the VM when prompted.

Tasksel

Q Tasksel helps assign a specic role to


your distro

First reboot
Upon booting the minimal installation for the rst
time, you should be prompted for your username and
password. We can now start to customise the system.
Install X.org server and Openbox window manager
(feel free to substitute another WM/DE if you prefer)
by typing sudo apt-get install xorg openbox.
When this has completed, type startx to test the GUI.
Click on the backdrop to bring up a menu that will
allow you to launch a terminal window.
Now you can begin customising the system. Make
things as comfortable as you like, but remember that
anything that is installed on this system will end up on
the target system. sudo apt-get install firefox
synaptic lxterminal mousepad lxdm will install
and set up the Firefox web browser, Synaptic (GUI

Tasksel is a system that installs and


congures a series of packages related
to a particular 'role' such as that of a web
server, a full KDE desktop or a media
workstation. You might be prompted for
task selection during an install, but you
can access this feature on a running
installation at any time.
Add the tasksel package with sudo aptget install tasksel, and then type sudo
tasksel to run it. You'll be presented with
the familiar text-mode interface, space to
select an option and tab to switch elds.
Naturally, you can add multiple tasks.
package manager), LXTerminal (more comprehensive
terminal application), Mousepad (a GUI text editor)
and LXDM (graphical login manager). That little lot will
add about 30MB to the installation ISO that you will
create, and about 100MB on the hard disk.
What you actually add is up to you. Apart
from adding packages, you can add desktop
customisations such as changing the backdrop.
When you've got things just how you want them,
create a distribution medium using Remastersys or a
disk cloning tool such as Clonezilla.

Linux & Open Source Genius Guide 13

Build your own distro

Ubuntu Builder
Ubuntu Builder is a GUI application that
allows you to take the contents of a
standard Ubuntu installation ISO and
modify it to create a new, customised
ISO for redistribution. It's a fairly simple application,
however, and not designed for deep modications of
the type that some of the other methods allow.
Ubuntu Builder is a standalone application
that runs on your desktop, and it even runs on
distributions other than Ubuntu. It works by
modifying a standard Ubuntu installation ISO,
downloading and inserting or removing packages
for you.

Tip

Ubuntu builder creates


a /home/ubuntubuilder/ work directory
into which it deposits its
output ISO

Pros
This is a nice,
simple tool there's little you
can do to make this
all go wrong

Cons
Not a huge amount
of customisation
depth. Lacked
polish and felt a bit
buggy in use. ISO
build process is
also a bit slow

Q The end result: Ubuntu 13.10 with MATE, a more traditional desktop

01

The installation

Start the process by adding the Ubuntu


Desktop PPA by typing sudo add-apt-repository
ppa:f-muriana/ubuntu-builder into a terminal. Now
run sudo apt-get update followed by sudo apt-get
upgrade to update the package lists on the system.
Use sudo apt-get install ubuntu-builder to
carry out the installation.

02

Launch Ubuntu Builder

You can now launch Ubuntu Builder by


typing sudo ubuntu-builder in the Terminal or by
clicking on its launcher icon in the launcher menu.
At this point in the process you should be able to see
the basic root interface.

14 Linux & Open Source Genius Guide

03

Fetch ISO

04

Select and unpack the ISO

You can fetch the current ISO by clicking on the 'Get Ubuntu' button in the main interface. However,
it's worth mentioning that we actually found manually fetching the latest standard install ISO from the Ubuntu
website to be more reliable.

Point Ubuntu Builder to the standard


installation ISO by clicking on the 'Select ISO' button.
This should invoke the unpack procedure in a Terminal
window, enabling us to modify the contents of the ISO.
Wait for this process to nish.

05

Add the MATE repository

Next, click on the 'Edit sources.list' button.


This will open a text editor. Cut and paste the
appropriate repository line (beginning with deb)
from the MATE installation guide (http://wiki.matedesktop.org/download).

Linux & Open Source

Genius Guide

Tip

Use the Synaptic


button to reinstall
LightDM, the graphical
login manager, when
changing desktop
environment

06

Select MATE as the


desktop environment

Click on the Select DE/WM button. In the menu,


select MATE as the desktop. This should invoke a
Terminal screen while the packages are replaced.
Allow this process to nish.

07

Create the remixed ISO

Click on the 'Build' button at the top of the


main window. This will open a Terminal window that
displays the progress of the ISO build process. This
might take a long time (an hour or more) depending on
the speed of your machine.

08

Install the ISO

Use the installation disk in the same


way that you would normally install Ubuntu...
and that's really all it takes! You are now ready
to start using your new, customised distribution.
Enjoy!

Linux & Open Source Genius Guide 15

Tips & Tricks


Improve the way you work
18 Host your own webmail server
Cut out the middleman by managing your own
webmail for personal accounts and avoid any
unnecessary downtime

22 Network scanning using Nmap


Discover Nmap and learn how to use tshark to
examine Nmap traffic

26 Create the perfect Windows 8.1


virtual machine
Work in and develop for Windows without ever
having to properly dual-boot it, by creating a
virtual machine

30 Understand regular expressions


Leap beyond wildcard expansion, to detailed
searches of text files and accurate validation of
program output

34 Use TeX and LaTex to create


beautiful documents
TeX and LaTex: typesetting for perfectionists!
TeX isnt just for numbers, its perfect for creating
invoices, books and PDFs too

38 Set up a wireless access point


with a Raspberry Pi
Discover how to wirelessly connect to your
Raspberry Pi, or any existing network connected
to it

40 Configure your PC for remote


maintenance
Learn how to add the ability to log back in and
maintain a Linux system remotely

44 Build your own private cloud


with ownCloud
A fast-track guide to setting up your own
file management system in the cloud using
ownCloud

48 Monitor network traffic


Use tshark to examine network traffic, solve
network difficulties and add network data to
a MongoDB database

52 Access your network


anywhere with a VPN
Work around insecure and restrictive network
access with your own virtual private network

56 Use gnuplot for drawing


Gnuplot is a powerful 2D and 3D program that
can be used to plot mathematical functions
and also show scientific data

60 SSH from a web browser


There are times when you are stuck using a
locked-down machine. As long as you have a
browser, though, you can still connect to your
remote machines

48

16 Linux & Open Source Genius Guide

64 Master version control with Git


Git is quite easy to learn, but you go much further
with an understanding beyond a simple cheat
sheet, so weve combined this how-to with a
little of why

68 Take pictures and video with the


Raspberry Pi Camera
Follow our short tutorial on how to get your new
Pi Camera set up on your Raspberry Pi, and how
to use it

70 Record slow-motion videos with


the Raspberry Pi
Use the updated Raspberry Pi firmware to record
slow-motion videos with the camera module

72 Browse privately with Onion Pi


Turn your Raspberry Pi into a highly secure and
very portable router to keep your system safe
and your browsing anonymous, wherever you are

Well show you


how to get your new
Pi Camera set up on
your Raspberry Pi
and how to use it

60

Linux & Open Source

Genius Guide

Linux & Open Source Genius Guide 17

Tips & Tricks


Open and favourite emails are remembered
between sessions so your unread accounts are
accurate wherever you log in

A traditional mail client layout exists in


Rainloopthatconnectswiththefoldersand
emails of your server

Customise your experience with different


folders, extra accounts and even social
network login support

Thepreviewcanbemadefullscreeninthesame
window,insteadofopeningadifferentpageor
tab this reduces server load

Host your own webmail server


Advisor
Rob Zwetsloot models
complex systems and is a web
developer procient in Python,
Django and PHP. He loves to
experiment with computing

Resources
Rainloop:
http://rainloop.net/downloads

A server

18 Linux & Open Source Genius Guide

Cutoutthemiddlemanbymanagingyourown
webmail for personal accounts and avoid any
unnecessary downtime

While using webmail may be incredibly


convenient, youre also at the mercy of
another companys server and privacy
policies. With the way that people are
connected online today its almost impossible to go
back to the mail client system of old, even if security
and privacy are far superior to Gmail.
But there is another solution that satises both
requirements; by hosting your very own webmail
server you can have both the convenience of
worldwide access while having the privacy of a
desktop mail client. By using Rainloop, a free

webmail application based on PHP, you can quickly


and easily set up your own webmail server with your
own custom settings and email addresses. It has a
modern UI and supports all the latest mail server
protocols, such as SMTP and IMAP.
You will need a server or always-on PC in
order to host your webmail, otherwise it will
only work when your computer is actually on.
Be aware that it may also increase your bandwidth
usage on a monthly basis, so dont send huge les
over it unless you need to.
Interested? Lets get going.

Linux & Open Source

Genius Guide

01

Server
ready

know it. If youre accessing it on a local machine, use


127.0.0.1. Enter the following URL:

http://[IP address or domain]/webmail/?admin

01

Server ready

Before we even begin to look at Rainloop in


depth, we need to make sure the system hosting it is
server ready. Install the following packages from the
terminal:

$ sudo apt-get install apache2 php5


php5-mysql libmysqlclient15-dev mysqlserver poppassd
Some or most of these packages may already be
installed, but its worth checking.

of Rainloop from the website and unzip it. Otherwise,


use wget to download it:

$ wget http://repository.rainloop.net/
v1/rainloop-[current version].zip
And then unzip it to proceed.

04

07

Log in

08

Add domains

09

Domain ports

The username and password for this default


version of Rainloop is admin and 12345 respectively.
Log in to access the interface; our rst task is to
then change the default password. Create a secure
password using standard password etiquette, as the
URL is quite common.

Elevate permissions

Open the terminal if its not already and use


cd to move to /var/www/webmail. Run the following
two commands to elevate the permissions of the
necessary les:

sudo find . -type d -exec chmod 755 {} \;


sudo find . -type f -exec chmod 644 {} \;

02

Set up directories

Well need specic directories created to get


Rainloop to work. You can do this in a le manager, but
it works in the command line like so:

05

Final permissions

Finish off making sure all the permissions are


set by running:

$ sudo mkdir /var/www/webmail

sudo chown -R www-data:www-data /var/


www/webmail

Move to the directory for the next step, as this is


where all our Rainloop les will live.

Rainloop is now just about ready to go, and most of the


rest of the setup will occur within the interface itself.

03

Download Rainloop

Now its time to get Rainloop. If you have


a graphical interface, download the latest version

06

Access Rainloop

Now log into Rainloop from a browser; either


through the IP of the server or its web domain if you

Head over to the Admin Panel where youll


nd domains and then add a domain. Here you can
add your own personal email or work domains and
use the IP or server address in either or both of the
IMAP and SMTP elds. You dont need to add Gmail
or a handful of other web services, as they are already
listed there.

Make sure the ports are correct on the server


addresses: for a local server, the default ports should
be ne. Also, make sure that Use short login form is
checked, name the server as the @ address of your
email (eg example.com) and then click Add to save it
to the list.

Linux & Open Source Genius Guide 19

Tips & Tricks

10

Contacts database

15

To support contacts, we need to add a


MySQL database. Open up the terminal again
and type in:

Log in to
your email

$ sudo mysql -u root -p


Enter your password and youll be dropped into the
MySQL shell to create the database.

11

Create the database

To create our contacts database, enter the


following command:

create database rainloop;


This, as you might have guessed, simply creates the
database named rainloop. To conrm the operation
and quit out of the MySQL shell use:

exit;

You can have both the convenience


of worldwide access with the privacy
of a desktop client
youve added to the server. It will log you in and work
just like any email client, with a column of emails on
the left and a preview pane located on the right. You
can then expand the emails to take up fullscreen.
Rainloop will remember which emails you have read
and connect to drafts and sent folders whenever you
log in.

12

Enable Contacts

Back in the admin panel on Rainloop, go to


the Contacts tab and check the Enable Contacts
box. Below that, change the type of the database to
MySQL. Youll then also need to add the username
and password for the MySQL server.

13

14

Activate password plug-in

15

Log in to your email

Now you need to add the plug-in on the admin


panel of Rainloop. Go to the packages tab and nd the
plug-in on the list; activate it by clicking the arrow next
to release date.

Password changing plug-in

Youll need to add a specic password change


plug-in to be able to change any passwords on a Linux
mail-server. To do this, open up the terminal and rst
install the plug-in to your system with:

$ sudo apt-get install poppassd

20 Linux & Open Source Genius Guide

To log in to your webmail, go to the webmail


address for your server:

http://[IP or domain]/webmail
And use your normal login details for the mail address

16

Add a signature

Click on the gear symbol at the bottom to


access the Settings. From here you can create an
identity, each with an individual signature. This is
useful for if youre using multiple accounts, or if
multiple people using the same email.

Linux & Open Source

Genius Guide

16

17

Add extra folders

admin page. This way you can consolidate all of your


emails into one single login. This can be done from the
main page, or more preferably from the Accounts tab
in the Settings menu.

Located next to the settings cog on the


main screen is the new folder button. You can
add folders and then move emails to them
for better ling, and these will sync with the
server. You can also have parents and children
of folders in your hierarchy.

Manager folders

19

Connect via social media

In settings you can create folders, but you can


also delete, hide and edit the folders too. Theres also
an option to change the system folders, allowing you
to lter spam, sent or other types of messages to predetermined folders. This is currently the only way to
lter email.

Rainloop has an interesting feature of


allowing you to log in to your webmail for specic
accounts via Twitter, Facebook or Google sign
in. Before you have a go at this, make sure that
youre signed in with the webmail account you want to
activate this on.

22

More on social

23

More to come

Having your social media work as a log on


means that you can only use them on the one account
that you've set them up for. It also means that people
can more easily log in to your accounts if they have
access to your regular system and you dont log out.

20
18

Add a
signature

Choose your account

Once youre logged in, go to the Settings


menu and locate the Social tab. In here you
can choose between the three social media types
you basically need to log in under one or all
of them here rst before adding the feature to
the main interface.

21

Add a new account

You can add another account to your


login for that email address, as long as its
included in the accounts you have created on the

There are some features missing at the


moment that you might miss from some regular
clients Out Of Ofce messages and automatic
lters, for example. These features will eventually
come, whether ofcially or via plug-ins, so keep an eye
out for updates.

Linux & Open Source Genius Guide 21

Tips & Tricks

Scanning an
iPad

Scanning a
Linux machine

About
Nmap

Scanning an
HP printer

Network scanning using Nmap


Advisor
Mihalis Tsoukalos is a UNIX
system administrator also procient
in programming, databases and
mathematics. He has been using
Linux since 1993

Resources
Nmap: www.nmap.org
WireShark site: www.wireshark.org
RFCs: www.ietf.org/rfc.html
Internetworking with TCP/IP, Volume
I, Douglas E. Comer, Prentice Hall

22 Linux & Open Source Genius Guide

DiscoverNmapandlearnhowtousetsharkto
examine Nmap trafc
Nmap is an open source tool created by
Gordon Fyodor Lyon that supports port
scanning, operating system detection
and version detection and is very famous
it has even been seen in movies, including The
Matrix Reloaded and Elysium.
Nmap stands for Network Mapper and supports
more than 15 scanning techniques. Nmap can
be useful to network administrators as well as
advanced users and hackers. It can be used for
securing your own network, but it can be also used
for hacking (or cracking) purposes.

When scanning hosts that are not on your


local network, keep in mind that sophisticated
intermediate devices such as routers, rewalls
and proxy servers can mislead Nmap and provide
incorrect information on purpose.
Although regular users can perform various Nmap
scans, particular command line options demand
root privileges to run.
Make sure you experiment with Nmap in order
to nd out exactly which options and scanning
techniques work best for you, but well walk you
through some of the options.

Linux & Open Source

Genius Guide

03

01

Get Nmap

Installing Nmap on a Debian 7 Linux system


is as easy as running the following command with root
privileges:

# apt-get install nmap


When running Nmap, the v parameter provides
additional output to the user about the scanning
progress and can be used in combination with every
other command line parameter. While an Nmap scan
is performed, you can press the v key to increase the
verbosity of the output and the V key to decrease the
verbosity of the output.

02

A simple
scan

ports of localhost (127.0.0.1). As you can see from


the output, the 3306 TCP port is only available
through the 127.0.0.1 IP address, which means
that the MySQL server process accepts local
connections only.

04

TCP connect() scan (nmap -sT)

05

TCP SYN scan (nmap -sS)

The TCP connect() scan is useful when you


do not have root access to a computer. Its weakness
is that it leaves traces in the log les of the remote
computer because it opens TCP session to the
remote machine.

is a good starting point when trying to evaluate


the security status of a computer. Remember
that although TCP SYN scans do not leave any log
info, modern rewalls and capture programs can
detect and report TCP SYN scans.

06

Ping scan (nmap -sP)

07

UDP scan (nmap -sU)

This is a very fast scan as it only sends


ICMP Echo Requests and accepts ICMP Echo
Replies. It is not easily noticeable, especially on
busy networks. It does not need special privileges
in order to run but it does not return much
information and it cannot be used in combination
with other types of scans. Its used for nding the
active machines on a network.

Find Nmap version

You can nd the version of Nmap you are


using by executing the following command:

$ nmap V
Nmap version 6.00 ( http://nmap.org )
Platform: x86_64-unknown-linux-gnu
Compiled with: liblua-5.1.5 openssl1.0.1e libpcre-8.30 libpcap-1.3.0 nmaplibdnet-1.12 ipv6
Compiled without:
$

03

A simple scan

The simplest scan that you can execute on a


Linux system is the following:

$ nmap localhost
The output gives you information about the open

The SYN scan never appears to log les


because the TCP connection is never initialised.
Its disadvantage is that it requires root access
to run. What it gives you is information about the
open, closed or ltered ports of a machine, which

The UDP protocol is used by many services


and this type of scan is the only way to detect open
UDP ports using Nmap. The UDP scan does not
create too much network trafc and works well for
examining machines that are running Microsoft
operating systems, but it needs root privileges to run.

Linux & Open Source Genius Guide 23

Tips & Tricks


12

08

Nmap output
options

Scan multiple hosts at once

The following command scans 256 IP


addresses, from 192.168.2.0 to 192.168.2.255:

$ nmap 192.168.2.0/24

Nmap can be useful to network


administrators as well as advanced
users and hackers

09

saves the output in Normal format, which


is suitable for easy reading and printing.
The Grepable format (nmap -oG <logle
name>) makes it easy to locate information in
Nmap output. Its main advantage is that the
output for each host is automatically stored
on a single le. The XML format (nmap -oX <logle
name>) is best suited for displaying the Nmap
output as an HTML page or processing it by other
software. The Nmap XML document DTD can be
found at www.insecure.org/nmap/data/nmap.
dtd.

Use a text le as input (nmap -iL)

You can use a text le in order to specify


the hosts or the networks that you want to scan
with Nmap. The following example will use the
contents of a le named LUD to perform a UDP
scan:

$ nmap -sU

10

iL LUD

Exclude hosts

The easiest way to exclude hosts is by


using the --exclude option. In order to exclude
192.168.1.1 and 192.168.1.5 from your scanning of the
192.168.0.0/16 subnet, you should run the following
command:

$ nmap sP 192.168.0.0/16 --exclude


192.168.1.1,192.168.1.5
If you have many IP addresses that you want to
exclude, you can put them in a text file (ex_IPs) and use
the --excludefile option as follows:

$ nmap
ex_IPs

sP 192.168.0.0/16 --excludefile

Any exclude option (--exclude or --excludefile) has


higher priority than an include option, so in case of
conflict, the exclude option wins. This does make
sense, as it is better to exclude a host by mistake than
include it and generate a Denial of Service attack or
slow down a network.

24 Linux & Open Source Genius Guide

11

Randomise hosts

Nmap can also choose IP addresses randomly


and try to scan them. This type of scan can put you into
serious trouble if used improperly. The main reason for
using it is for generating random network traffic.

nmap --randomize-hosts

12

Nmap output options

Nmap can save its output in various


formats. The nmap -oN <logle name> option

13

Scan scanme.nmap.org

14

Scan a Linux machine (part 1)

The scanme.nmap.org (or scanme.insecure.


org) host is offered for testing Nmap. This means that
it is not illegal to run Nmap using scanme.nmap.org
as your target.
Please do not use it irresponsibly or for testing a SSH
brute-force password-cracking tool although it
wont lter out 256-999.

The two most helpful scans are the TCP and


UDP scan because they give you a general overview
of a machine, so you should start with them when
examining a new machine. You can immediately nd
out open TCP and UDP ports as well as the network
services that a machine runs.

Linux & Open Source

Genius Guide

22

Capture
the trafc

21

Examine Nmap trafc using tshark

22

Capture the trafc

You will now learn how to study the trafc that


is created by an Nmap UDP scan using tshark, the
command line version of WireShark. First, the network
data will be captured. Then, a tshark display lter
will be applied in order to display the desired trafc.
Finally, specic trafc will be examined and reviewed.

The Nmap command that is going to be used is


the following:

$ sudo nmap

sU 192.168.2.10

Just before running Nmap, the following tshark


command will be executed for capturing network
data:

$ sudo tshark

17

Scan a machine running Mac OS X

18

Scan a Cisco 877W ADSL router

19

Scan a Windows machine

20

Scan an HP printer

As far as Nmap is concerned, a machine


that is running Mac OS X does not differ that
much from a typical UNIX machine, and its
output demonstrates this.

23

w UDPscan.tcpdump

Watch the Nmap trafc

If you want to display the network packets


that go to or come from the host with the 192.168.2.10
IP address and also use UDP port 400, run command:

$ sudo tshark -R "(ip.addr ==


192.168.2.10) and (udp.port == 400)" -r
UDPscan.tcpdata x
You can examine the network trafc of other UDP port
numbers using analogous commands.

As you can see, it took Nmap 16.07 seconds


to scan a Cisco 877W ADSL router and its guess
was pretty accurate! Nevertheless, you have to keep
in mind that Nmap operating system detection is not
always accurate and therefore cannot be trusted all
the time.

15

Scan a Linux machine (part 2)

Next, you should try something smarter,


like nmap A T4. The A option enables
operating system and version detection (know
as TCP/IP ngerprinting), script scanning,
and traceroute while the -T4 option is used for
faster execution. Please note that if you run
the same command without root privileges, you will
get a less detailed output.
The T option is useful for slowing down the scans
in order to avoid the creation of too much trafc
that can slow down or ood a network or a host. The
allowed values of the option are 0-5. A smaller number
dictates a slower scan.

16

Scan an iPhone 5

Nowadays, a mobile phone can have an IP


address and therefore can be scanned using Nmap!
It is interesting to scan new devices to nd out their
open ports and their behaviour during a scan. The
output shows that an iPhone does not have very many
open ports.

Windows machines have the habit of having


more open ports than needed, so it is always a good
idea to examine them using Nmap. Here Nmap was
unable to correctly detect the operating system of the
machine, which dictates the presence of a rewall
somewhere in between.

Network printers also have an IP address, so


Nmap can be used to scan them as well.
Nmap found out that it is an HP printer that runs
HP embedded and has a web server running (ports
80 and 443 are open). Following the Nmap scan, the
printer needed a reboot because all of its lights were
blinking!

24

Discuss Nmap network trafc.

Nmap UDP port scanning starts by


sending empty UDP packets packets that
have no additional data to various UDP ports
waiting for an answer.
You can see that for UDP port 137 (netbios-ns)
there is a reply that states that the host cannot
understand the actual packet, so Nmap knows that
the port is denitely in use!
On the other hand, although a packet
was sent to port 80, the answer was a Port
unreachable packet, so Nmap implies that
UDP port 80 is not in use.
If there is no reply at all from the host for a
given UDP port, then the port is considered
open but ltered (port 138).

Linux & Open Source Genius Guide 25

Tips & Tricks


Use VirtualBox to emulate any other
x86-based operating system or Linux
distro on a host computer

Install Windows under Linux with no


problems thanks to the way VirtualBox
handles virtual machines

Use the tiled interface or the Desktop


interface in Windows 8.1 whatever you
choose to develop for

Share les from the host computer and


attached USB storage to the virtual
machine to speed up le transfer

Create the perfect Windows 8.1 VM


Advisor
Rob Zwetsloot models
complex systems and is a web
developer procient in Python,
Django and PHP. He loves to
experiment with computing

Resources
VirtualBox:
www.virtualbox.org

VirtualBox Guest Additions


ISO of Windows

26 Linux & Open Source Genius Guide

Work in and develop for Windows without


ever having to properly dual-boot it, by
creating a virtual machine

Sometimes
using
Windows
is
a necessary evil. A lot of ofce
environments still use Windows, and
sometimes youll need Windows-specic
applications and software if you want to work from
home. Theres also the problem of developing crossplatform apps constantly switching between
operating systems is time-consuming and can,
frankly, get exhausting. The solution to a lot of these
issues is to never dual-boot your machinein the rst
place, but instead to virtualise Windows, in particular
Windows 8.1, on your computer.

The virtualisation software, Oracle VM VirtualBox,


makes it easy to create virtual machines, and its
what well be using in this tutorial. Its a bit of a
different process to install Windows, though, and
well be covering that in detail. We can also have it
interact with the folders and storage from the host
machine, allowing you to set up a shared folder to
access les and dev builds.
Youll need a decently powered, modern PC to be
able to run the VM smoothly, and make sure you have
plenty of RAM.
Heres how to create your own virtual machine

Linux & Open Source

Genius Guide

01

01

Install VirtualBox

02

New virtual machine

The necessary rst step, VirtualBox is


available in most repositories as the package
virtualbox. Some distros such as Fedora dont
include it in the repositories; however, you can
get binaries along with the source code from the
VirtualBox website listed in the resources section.

Start up VirtualBox and click on New on the


top menu to create a new virtual machine. It will rst
ask you to name it, and will try to t a category to the
name. The category is mostly optional, although make
sure its set to 32-bit or 64-bit for the version you plan
to use.

03

Volatile memory

Youll need to set the RAM that the virtual


machine will take from the host system. 2GB is the
recommended amount for Windows 8; while you

Install
VirtualBox

should denitely try to full that requirement,


we suggest using half of your systems
available RAM.

04

Storage space

05

Storage type

Windows requires a little more hard drive


space than a Linux distro. VirtualBox will suggest
25GB; however, if you can spare the space, you should
up it to 50 or 100 if you plan to use it for a bit more than
basic development or ofce work.

Choose VDI as the type of hard drive


you want, and then make sure its dynamically
allocated. This means the space for the storage
wont be taken up all at once, so if youve created a
100GB virtual hard drive, it will only use the space it
needs at any one time. Name it and click Create.

06

System settings

07

Windows disc

08

First boot

Select the newly created VM and then hit


Settings on the top bar. Go to the System tab rst
and enable EFI if youre using Windows 8 or later.
Next, go to Display; make sure Video Memory is set to
maximum and tick Enable 3D Acceleration.

Click on the Storage tab to add the Windows


installation disc. Click on the empty disc icon
underneath the IDE controller, and then the disc icon
that shows up on the right. If you have the disc, insert
it into your PC and choose host drive from the menu.
Otherwise, nd and select the ISO.

Once nished, click Start while the Windows


VM is selected. It will open a window and boot into
Windows installer. Go to View on the top settings and
click on Fullscreen to set the VM to always launch
and stay in full-screen mode. If its staying at a
prompt screen, you may have to disable EFI.

Linux & Open Source Genius Guide 27

Tips & Tricks

15

09

VirtualBox
additions

Install preparation

The rst step in installing is to set the


language, as you normally would. Click Install and
then agree to the licence. After this, youll be asked
how you want to install; click on the Custom install
option to install from scratch.

You can also access USB storage


devices via the host computer

10

Storage

Unless you plan to do any crazy dualbooting, you will only need to select the empty
space on your hard drive. Otherwise, Windows
gives you tools that lets you create partitions
so that you can install a Linux distro or other
Windows version elsewhere.

12

Personalise

13

Microsoft account

On Windows 8 and 8.1, youll now need to


personalise your install with a name, colour scheme
and update settings. By default, Windows will
download and install updates when it sees t. You
can change the type of updates it will automatically
install, or turn it to manual.

11

Wait for it

Windows can take a while to install, and


will go through several phases, including rebooting
once or twice during the process. Leave it alone and
it will do its thing without any interruptions.

28 Linux & Open Source Genius Guide

Youll need to log into, or set up, a Microsoftbased account to use the latest versions of Windows.
If you already have one, you can enter it here and log
in. Otherwise, youll need to link an email address to a
new account.

14

Final setup

15

VirtualBox additions

Wait a while and it will grab your account


settings and any other data you may have associated
with a Windows 8 install under your Microsoft
account. After this, it will bring up the main tiled
interface for you to start using.

You may have noticed that the screen may


be the wrong resolution. To x this, youll need to
install the guest additions via VirtualBox. Bring up
the VirtualBox menu pop-up at the bottom of the
screen and click Devices and Insert Guest Additions
CD image. It will automatically download it if you dont

Linux & Open Source

Genius Guide

17

already have it. Right-click and select All Apps, then


nd the Computer. Click it and nd the CD.
Open it and click on the relevant Windows .exe for
your system and follow the instructions.

16

Shared folders

17

Share settings

19

Host USB

You can also access USB storage devices,


like an external hard drive or USB stick, via the host
computer. Youll have to do it manually for each USB
device, though. Plug in the rst USB device that you
want to connect to the virtual machine and make sure
its mounted.

Share
settings

should be prominently displayed if not, make sure


youve updated Windows. Otherwise, click the app and
download it. During the upgrade process, youll have to
restart and reconfirm some settings.

One of the better things we can take


advantage of with a virtual version of Windows is
a shared folder that easily allows transfer of les
between the host system and Windows. Create a
specic folder on the host to use a shared folder.

Go back into the VM and click Devices


followed by Shared folder settings. Click the folder
symbol with a plug sign. Click the down arrow and
Other to then browse your le system and select the
shared folder we set up. Name it if you wish.

20
18

Access the share

Make sure youve selected automount and


then press OK. Windows will now easily be able to see
it in the le system; youll need to go to Network then
VBOXSVR to nd the folder. You can then drag it to the
desktop to create a shortcut.

VirtualBox setup

Go into the settings for the Windows VM and


select the USB tab. Make sure the USB controllers
are enabled and then click the USB plug with a plus
symbol. Select your USB device and it will add it to
the VM; whenever its plugged in, it will automatically
mount on Windows.

21

Update to Windows 8.1

If youve installed Windows 8, updating to 8.1


is free. Go to the Store and the Windows 8.1 update

22

Boot to desktop

23

Finish up

Windows 8.1 now has the ability to boot into the


desktop. Once youve completed the update, enter the
Desktop. Right-click on the taskbar and open Properties,
go to the Navigation tab and then select Go to the
desktop instead of Start when I sign in. Thats actually
the name of the option.

After a reboot, that setting will come into


effect. Youll now be able to properly start using your
virtual Windows machine for anything youd use a
normal Windows machine for, without having to dualboot and constantly switch back and forth.

Linux & Open Source Genius Guide 29

Tips & Tricks


Even the simplest regexp can
make you more productive at
the command line

Online regexp testers like 2problems.com let you


rene your regexps though perl pi.bak
also gives you a backup of your original test le

Were going to show you use of the


metacharacters one-by-one, until . * [ ]
^ $ \ ? | { } ( ) is more than just line noise

Regexps are great, but its important to know their


limitations theyre ideal with text les, such as
logs, but steer clear of complex HTML searches

Understand regular
expressions
Advisor
Richard Smedley
started using computers long before
WYSIWYG, and still maintains that
the command-line, and Emacs,
is the most productive working
environment

Resources
Server like LAMP/WAMP/MAMP
ownCloud plug-ins:
apps.owncloud.com

30 Linux & Open Source Genius Guide

Leap beyond wildcard expansion, to


detailed searches of text les and
accurate validation of program input
Were always searching for something
the le where we wrote that recipe
(Python or baking); the comment in
100,000 lines of code that points to an
unnished module; the log entry about an iffy
connection. Regular expressions (abbreviated as
regexps hereafter, but youll also see regex and re)
are a codied method of searching which, to the
unenlightened, suggests line noise. Yet, despite a
history that stretches back to Ken Thompsons 1968
QED editor, theyre still a powerful tool today, thanks
to grep global regular expression print. Using grep
exposes only the limited Basic Regular Expressions

(BRE); grep -E (or egrep) gives Extended Regular


Expressions (ERE). For other languages, most
adopt PCRE (Perl Compatible Regular Expressions),
developed in 1997, by Philip Hazel, and understood
by many languages, though not always implemented
in exactly the same way. Well use grep -P when we
need to access these. Emacs has its own regexp
style but, like grep, has a -P option to use Perlcompatible regexps.
This introduction is mostly aimed at searching
from the shell, but you should easily be able to adapt
it to standalone Perl scripts, and other languages
which use PCRE.

Linux & Open Source

Genius Guide

09

01

Word up!

02

Reserved character

POSIX
classes

Youre probably used to searching a text le


for occurrences of a word with grep in that case,
the word is the regular expression. More complicated
regexps are simply concise ways for searching for
parts of words, or character strings, in particular
positions.

Some characters mean special things in


regexp pattern matching: . * [ ] ^ $ \ in Basic Regular
Expressions. The . matches any character, so using
it above doesnt just nd the full stop unless greps -F
option is used to make the string entirely literal.

05

Mmmmm, cooooool

The other quantiers are + for at least


one of the preceding regexps (_+ nds lines with
at least one underscore) and * for zero or more
(coo*l matches col, cool, coooooooool, but not cl,
useful for different spellings of mmmmmmmmm or
zzzzzzzzzz).

03

Atlantic crossing

08

A to Z Guide

09

POSIX classes

The range in [] can be anything from the


ASCII character set, so [ \t\r\n\v\f] indicates the
whitespace characters (tab, newline et al). [^bd]
oom$ matches all words ending in oom, occurring
at the end of the line, except boom and doom.

The POSIX classes for character ranges


save a lot of the [A-Za-z0-9], but perhaps most
useful is the non-POSIX addition of [:word:] which
matches [A-Za-z0-9_], the addition of underscore
helping to match identiers in many programming
languages.

Extended Regular Expressions add ? | { } ( )


to the metacharacters. grep -E or egrep lets you use
them, as above, where standardise|standardize can
match British or American (and Oxford) spellings of
standardise.

06

No number

07

Start to nish

Feeling condent? Good, time for more


goodies. [0-9] is short for [0123456789] and
matches any element in the square brackets. The ^
inside the brackets is a negation, here matching any
non-number but the other ^?

04

Colourful?

| gives a choice between the two


characters in the parentheses standardi(s|z)e
saving unnecessary typing. Another way to nd
both British and American spellings is ? to indicate
one or zero of the preceding element, such as the u
in colour.

The ^ matches the expression at the


beginning of the line; a $ matches the end. Now
you can sort your document.text from text.doc and
nd lines beginning with # or ending in a punctuation
mark other than a period.

10

ASCII style

Where character classes arent implemented,


knowledge of ASCIIs underpinnings can save you time:
so [ -~] is all printable ASCII characters (character
codes 32-127) and its inverse [^ -~] is all non-printable
ASCII characters.

Linux & Open Source Genius Guide 31

Tips & Tricks


14

11

Beyond grep

12

Nice one Cyril

Bye bye,
IPv4

Find and Locate both work well with regexps.


In The Linux Command Line (reviewed in LUD 111),
William Shotts gave the great example of find . -regex
.*[^-_./0-9a-zA-Z].* to find filenames with embedded
spaces and other nasties.

Speaking of non-standard characters, while


[:alpha:] depends on your locale settings, and may only
find ASCII text, you can still search for characters of
other alphabets from accented French and Welsh
letters to the Greek or Russian alphabet.

Regular expressions are like a


particularly spicy hot sauce to
be used in moderation and with
restraint, only when appropriate
14
Bye bye, IPv4

FOSDEM was all IPv6 this year, so lets not


waste any more time on IPv4 validation, as the future
may actually be here. As can be seen in this glimpse of
IPv6 validators, despite some Perl line noise, it boils
down to checking appropriate amounts of hex.

15
13

Ranging repeat

While {4} would match the preceding element


if it occurred four times, putting in two numbers gives
a range. So, [0-9]{1,3} in the above screenshot nds
one-, two- or three-digit numbers a quick nd for
dotted quads, although it wont lter out 256-999.

32 Linux & Open Source Genius Guide

Validation

By now regexps should be looking a lot less


like line noise, so its time to put together a longer
one, just building from some of the simpler parts. A
common programming task, particularly with web
forms, is validating input is in the correct format
such as dates.
In this case were looking at validating dates,
eg for date-of-birth (future dates could then

be ltered using current date). Note that


(0[1-9]|[12][0-9]|3[01]) checks numbers 01-31, but
wont prevent 31st February.

Linux & Open Source

Genius Guide

23

A regexp
too far

21

Perl -pie

22

Perl one-liner

23

A regexp too far

24

Tagged offender

Aside from grep, Perl remains the most


comfortable t with regexps, as is far more powerful
than the former. With perl -pie on the command line,
you can perform anything from simple substitutions
on one or more les, to

counting the empty lines in a text file (this


from Krumins Perl One-Liners, see next months book
reviews). /^$/ matches an empty line; note Perls use of
// to delimit a regexp; ,, could also be used if / is one of
the literals used.

18

Boundary guard

19

Literally meta

20

Lazy = good

As well as ^ and $ for line ends, word


boundaries can be matched in regexps with
\b enabling matches on, say, hat without
matching chatter. The escape character, \, is
used to add a number of extra elements, such
as \d for numerical digit.

Now you know the basics, you can


build slightly more complicated regexps but,
as Jeff Atwood said: Regular expressions
are like a particularly spicy hot sauce to be
used in moderation and with restraint, only
when appropriate.

Speaking of boundaries, placing \Q \E around


a regexp will treat everything within as literals rather
than metacharacters meaning you can just quote
a part of the regexp, unlike grep -F where everything
becomes a literal.

16

Back to basics

17

Why vi?

Now we have the basics, and can string


them together, dont neglect the grep basics here
were looking at how many attempts at unauthorised
access were made by SSH in a given period. An
unnecessary pipe replaced with grep -c.

Whatever your position in the venerable and


affectionate vi/Emacs war, there will be times and
servers where vi is your only tool, so grab yourself a
cheat-sheet. Vi and vim mostly follow BRE. Here we
see one of the \< \> word boundaries.

Time to think about good practice. * is a


greedy operator, expanding something like <.*> by
grabbing the last closing tag and anything between,
including further tags. <.*?> is non-greedy (lazy),
taking the rst closing tag.

Finally, know the limitations of regexps. Dont


use on HTML, as they dont parse complex languages
well. Here the legendary StackOverflow reply by Bob
Ince to a query on their use with HTML expresses the
passion this question engenders.

Linux & Open Source Genius Guide 33

Tips & Tricks


\usepackage imports a macro package in this case maths
and thats where the power lies. There are macros for everything.
Note too what \LaTeX produces in the right-hand panel

Your text editor will provide highlighting of LaTeX


comments and commands, making it easy to just
read back either text or structural information

Run latex on the .tex text le and it gets to


work doing all the typesetting, so you only
need to worry about structure and content

You can use TeXs xdvi viewer to look at the document, or


convert it to PostScript or PDF for printing and thanks to
tablets, PDFs are undergoing a small-screen renaissance

Use TeX and LaTeX to create


beautiful documents
Advisor
Richard Smedley
started using computers long before
WYSIWYG, and still maintains that
the command-line, and Emacs,
is the most productive working
environment

Resources
TexLive:
www.tug.org/texlive/

Comprehensive TeX Archive


Network:
www.ctan.org/

34 Linux & Open Source Genius Guide

Learn how TeX isnt just for numbers; it


also makes invoices, books and PDFs
You use LibreOfce for your documents,
perhaps Scribus for making a company
brochure, and edit most documents
online. Youve heard of TeX in the
context of mathematical equations, mostly but
why would you use it to lay out everything from
invoices to 100-page PDFs? Quite simply, because
it produces beautiful-looking documents that are
easy to maintain, and using a simple markup that
makes keeping track of document evolution and
changes far simpler than it is with word processors
and desktop publishing packages. Format once for
each type of content you write, then concentrate just
on the content. Thanks to tablets with insanely large

numbers of pixels per inch, beautiful PDFs become


important, so feel the irony as you produce them
from the *NIX command line.
TeX itself is simply a formatting engine, stable and
virtually bug free. LaTeX is a complete document
preparation system built as a macro package on TeX,
and containing templates and add-ons to create
just about any document. Well use LaTeX2e, known
as Standard LaTeX, which has barely changed in 20
years it hasnt needed to.
The commands are straightforward, but there
are many options. Follow our quick intro, modifying
the documents, then move online and build bigger,
better docs as your knowledge increases.

Linux & Open Source

Genius Guide

06

01

Bugs will be features!

02

TeX Live

03

TeX in your browser

Right-to-left

TeX is extremely stable, with years passing


between bug reports! Since release 3, the version
number of TeX is converging to , and is now at
3.1415926 (released March 2008). The version
number will be changed to upon creator Donald
Knuths death, and all future bugs found are to be
regraded as features!

TeX Live is the recommended way to install


the LaTeX packages youll need. Its available for
almost all distros. To try our invoice example, also
install the latex-invoice RPM, or apt-get install
texlive-latex-extra.

You neednt wait until you can download


and install TeX Live. JaxEdit is an online LaTeX
editor with live preview for maths but not much
else yet which you can try at jaxedit.com. You can
also download the JS and run it in your browser.

08

WYSIWYM formatting

09

Its just text

These strengths are a product of the What


You See is What You Mean (WYSIWYM) paradigm.
You write your .tex document in any text editor and
let LaTeX take care of the typesetting: running latex
on the le produces a .dvi, which you can convert to
PostScript or PDF.

05

Just a fraction

06

Right-to-left

07

Booked up

The step 4 example code produces this


output when converted to a PDF. Well look at some
of the commands in a minute, so you can get started
trying LaTeX out, but rst we want to make sure that
you realise its not just for maths.

Remember its just text. With structure.


And a few commands that all begin with \ and may
have braces {} and possibly square brackets []. The
problem for the beginner is knowing where to start
with so many options, so lets start simply.

LaTeX spread through the academic world


and quickly gained popularity with linguists and
language researchers, leading to excellent layout
of non-Latin alphabets. Pictured at the top of the
page is an example of Arabic in ArabTeX (which also
handles Hebrew).

04

e=mc2

Most people have heard good things


about TeX and LaTeX in the context of laying out
mathematical equations. In fact, it was because of
how badly typesetters were laying out mathematical
equations that Donald Knuth felt compelled to write
TeX originally

LaTeX continues to be virtually the default tool


for producing postgraduate theses and other academic
works this is because of its structural strengths, and
consequent abilities at producing bibliographies and
nested contents pages.

10

Macro power

Remember those imported macros on the


title-page screenshot? There are thousands of
user-contributed packages at CTAN. Here, your
advisor is looking at a Dutch .sty for producing
invoices. Useful, but not as powerful as the ofcial
invoice package

Linux & Open Source Genius Guide 35

Tips & Tricks

15

Draught preview

16

Special character

In the pic at the bottom of the page, were


previewing an earlier draught of the invoice. Note
weve used separate programs GEdit, xterm
and xdvi in this case, but done it all within
Emacs on the screenshot of the opening page.
The point is, were dealing with text, so its
easy to t in with your workow, and the tools
with which you are familiar. It might also be a
time to try something new [will you stop pushing
people onto Emacs, already Ed], or even use
LaTeX to generate your slides, or your websites
PDF downloads.

Note the \& in LinuxUser \& Developer


&, like #, $, %, ^, _, {, }, ~ and \, is a special
character, with special meaning to LaTeX, so
must be called like: \# \$ \% \^{} \& \_ \{ \} \~{}
\textbackslash.

11

More macro power

which can do the adding up for you. Having


tried in the past to produce invoices both from
spreadsheet templates and word-processors, it is
with great relief that your advisor turns to something
that just needs text to be lled in, yet produces
readable output.

12

Boilerplate

Everything before \begin{} is whats called


boilerplate, analogous to <head> in an HTML
document. \documentclass must be specied
in the {} braces: {letter}, {article} or {book} are
common types. Options such as twoside for doublesided printing, paper size, and font size go in the []
brackets.

17

Like the hat

Note the {}s after the escaped ^ and ~ in


the previous step. This tells LaTeX that they are
standalone, otherwise it will treat them as accents

and place them on top of the following letter. Find


oodles more in The Comprehensive LaTeX Symbol
List, by Scott Pakin.

18

Error messages

For more complex documents, with


references and tables of contents, you may have
to rinse and repeat the latex command until
everything is fully processed. If theres a problem,
LaTeX will tell you about it. Here, weve left in special
characters by mistake.

The best way to learn is, as with


any coding language, by reading .tex
documents out there and making
your own
12-16

13

Make a date

14

Desktop calculator

\usepackage{}, \title{}, and \date{} are


common boilerplate. \include{lename.tex} will
import other les, so big projects can be split, such
as a short story collection of chapters from different
authors. Text commands can be embedded in elds,
as here with date.

Using the \Fee command followed by


{activity}, {rate} and {units done} for each invoice
item lets invoice.sty work out totals for you and
produces clear, readable invoices (see main picture).
VAT options are also available.

36 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

19

Pretty pictures

20

txt2tex

LaTeX seems to work best with graphics


from EPS les and PDFs, or tools generating vector
graphics from data on-the-y. However, your advisor
doesnt excel at all things visual, and has seen others
do more than dump a pic in a doc, as here.

If youre converting a lot of documents to


TeX, txt2tex will take some of the tedium out of the
job download it and run the demo. But youre by no
means limited to text-based documents in TeX. See
what LaTeX can do with music and games

21

Know the score

The WYSIWYM choice for music is not so


clear, with Philips Music Writer (PMW) arguably
easier to learn than TeXs MusicTeX markup.
Additionally, Lilypond, a standalone MusicTeX fork,
produces beautiful scores. Choices, choices.

23

TeXnicians
and TeXperts

22

En passant

Install one of the chess packages (skak or


TeXmate) and you can draw a board from Forsyth
Edwards Notation:

\fenboard{r5k1/1b1p1ppp/p7/1p1Q4/2p1r3/
PP4Pq/BBP2b1P/R4R1K}
With TeXmate you dont need the numbers to end
lines.

23

TeXnicians and TeXperts

24

Try it out

There are some great guides to using LaTeX


The Not So Short Introduction to LaTeX2 (Or
LaTeX2 in 157 minutes) by Tobias Oetiker, Hubert
Partl, Irene Hyna and Elisabeth Schlegl is a good
concise start, but dont just read a book...

The best way to learn is, as with any coding


language, by reading .tex documents out there and
making your own. If youve got a document youre
about to write, give yourself an extra hour and nd a
.tex with similar structure to build it upon.

Linux & Open Source Genius Guide 37

Tips & Tricks


Create an access
point for an existing
wired network

Create an access point for a


new independent network
that can be used anywhere

Our experts Raspberry


Pi with a suitable
wireless N dongle

Set up a wireless access


point with a Raspberry Pi
Advisor
Liam Fraser
Liam is the creator of the
RaspberryPiTutorials YouTube series
and volunteers as a Linux server
administrator for the Raspberry
Pi Foundation

Resources
n Latest Raspbian Image
raspberrypi.org/downloads

A router or switch
A USB wireless dongle that
supports Access Point mode
A Wi-Fi device to test with

38 Linux & Open Source Genius Guide

How to wirelessly connect to your


Raspberry Pi, or any existing network
connected to it
A Raspberry Pi with a wireless dongle
that supports Access Point mode is a very
versatile tool. As well as connecting to
existing wireless networks, the Raspberry
Pi can become an access point and have multiple
devices connected to it. It can either have its own
network or bridge onto an existing one and make it
wireless. One potential application of this is to create
a wireless access point for guests, with appropriate

rewalling rules to make sure they cant connect to


anything on your LAN. Another is simply connecting
to it over wireless rather than Ethernet, which can
be useful if the Ethernet port is already in use. For
example, a Raspberry Pi acting as a portable PXE
server (used to boot a computer over the network)
would have an Ethernet cable connected to the
computer its booting, but you might need to log in and
see whats going on at the same time.

Linux & Open Source

Genius Guide

01

Install the required software

Log into the Raspbian system with the


username pi and the password raspberry. Get the
latest package lists using the command:

sudo apt-get update


Well be using hostapd, which is a daemon
that handles access point management and
authentication. Well also need bridge utils, which
will be used to bridge the Ethernet and wireless
interfaces together. iw is used to get information
about the wireless interface, and dnsmasq is used
as a DHCP server. Install these with:

sudo apt-get install hostapd bridgeutils iw dnsmasq

02

Check for AP mode

Its crucial that you have a wireless dongle


that supports Access Point (AP) mode. To verify that,
connect the wireless dongle and type iw list. There
will be a section titled Supported interface modes.
Ensure that AP mode is in that list. Our experts
output looked like this:

Supported interface modes:


* IBSS
* managed
* AP
* AP/VLAN
* WDS
* monitor
* mesh point

03

network, but we will set up our own later on. Reboot


using sudo reboot so that the changes take effect.

04

Congure hostapd

We rst need to edit /etc/default/hostapd


to start the daemon on boot. Uncomment the
DAEMON_CONF option and replace it with:

DAEMON_CONF="/etc/hostapd/hostapd.
conf"
We then need to create the hostapd cong le at the
path specied above. It should have the following
contents:

interface=wlan0
bridge=br0
driver=nl80211
country_code=UK
ssid=pinet
hw_mode=g
channel=1
wpa=2
wpa_passphrase=super_secret
wpa_key_mgmt=WPA-PSK
ieee80211n=1
wmm_enabled=1
Note that if you dont have a wireless N capable
device, or are having stability issues, youll want
to leave out the last two lines. You may also need
to change the wireless channel if you are having
stability issues. Once youve done this, you can
start the hostapd daemon with sudo /etc/init.d/
hostapd start.

Bridge interfaces

Well bridge the Ethernet interface and the


wireless interface so they can share the same IP
address and trafc can pass between them. To do
this, edit /etc/network/interfaces (using sudo) and
change it to look as follows:

auto lo br0

05

Connect to the network

The SSID in the cong le is the network


name, and the WPA Passphrase is the password.
When you connect to the network, the connection
will be bridged to the existing wired network and
you should receive an IP address from your existing
router doing DHCP. Congratulations on creating an
access point!

06

Create an independent network

Were now going to change the network


conguration to be independent from your
existing network. The rst thing to do is to pick a
private address range that youd like to use. For
this example, well use 192.168.0.0/24. However,
we recommend using a random address range in
case you ever want to route to other networks. The
rst thing well do is assign a static address to the
bridge interface. Change the bridge section of /etc/
network/interfaces to look as follows:

iface br0 inet static


bridge_ports eth0 wlan0
bridge_waitport 0
address 192.168.0.1
network 192.168.0.0
netmask 255.255.255.0

07

Congure a DHCP server

We installed dnsmasq in the rst step.


Conguring it is really simple. Start by taking a copy
of the default cong:

sudo cp /etc/dnsmasq.conf /etc/


dnsmasq.conf.orig
Then edit /etc/dnsmasq.conf to contain the
following lines:

interface=br0
dhcp-range=192.168.0.2,192.168.0.254,
255.255.255.0,12h
Reboot for the changes to take effect. Youll want to
disconnect the Raspberry Pi from your existing network
at this point, otherwise there will be two DHCP servers
on the network.

08

Try it out

Devices on the wireless interface will be


able to talk to devices on the wired interface, and
any device plugged into the Pi will get an IP address
so that it can talk to the other devices.

iface lo inet loopback


allow-hotplug eth0
iface eth0 inet manual
allow-hotplug wlan0
iface wlan0 inet manual
iface br0 inet dhcp
bridge_ports eth0 wlan0
bridge_waitport 0
At the moment, well be bridging onto an existing

Its crucial that you have a wireless


dongle that supports Access Point
(AP) mode. To verify that, connect
the wireless dongle and type iw list.
Ensure that AP mode is in the list
Linux & Open Source Genius Guide 39

Tips & Tricks


Install Webmin, the
extremely compressive webbased system conguration
and monitoring GUI

Open a terminal on the


remote computer and
because its SSH, its
completely secure

Use VNC over SSH to gain


remote desktop access to
put yourself in full control of
the other computer

Via SSH tunnelling, you can run


full X Window applications on
your desktop while they execute
on the remote computer

Configure your PC for


remote maintenance
Advisor

Michael Reed is
a technology journalist
and hes been hacking
away at Linux for over
15 years

Resources
Client computer running Linux
Remote computer running Linux
LAN

40 Linux & Open Source Genius Guide

How you can add the ability to log back in


and maintain a Linux system remotely
If youve set up a PC using a Linux distro
for a non-expert user or family member
to use, youve made the rst step in
providing them with a secure and wellfeatured computing platform. However, if the user
in question needs a bit of hand-holding, its only a
matter of time before you will need to carry out some
maintenance that cant be explained over the phone.
This project involves installing SSH for commandline access and for secure tunnelling, VNC for remote
desktop access, and Webmin for overall system

conguration. Note that once you have SSH terminal


access up and running, you can carry out the rest of
the tutorial remotely over the LAN. In fact, if you then
skip ahead to the port forwarding section, you can do
the other stages of the tutorial over the internet.
Initially, this tutorial will assume that you have
both the client computer (your computer) and the
remote computer (their computer) attached to
your LAN. Ultimately, we will set up the router in the
owners home and a dynamic DNS server so that we
can nd the remote machine on the internet.

Linux & Open Source

Genius Guide

01

Set up the network

This tutorial presumes that you have


the remote computer and the client computer on
your LAN. Make a note of the login username and
password of the remote computer and its IP address
(type ifconfig to nd it).

a password and a username. Type ssh-keygen -t dsa


to begin. You can hit Enter to accept the defaults when
prompted, but type in a password when asked.

02

Install SSH

To enable remote terminal logins, install


the SSH server by typing sudo apt-get install
openssh-server (or yum -y install opensshserver on Red Hat). Open /etc/ssh/sshd_cong in a
text editor (as root) to begin conguring SSH.

07

You can place les on the remote computer


or fetch les from it with the scp command.

05

Copy key

06

Test SSH

Copy the pubic key over to the machine that


will be used for remote maintenance. You carry this
out by typing ssh-copy-id [remote login name]@[IP
address of remote computer] on the client computer.

03

scp [remote user]@[remote


IP]:[local file]
[destination directory] to fetch the le(s) onto
the remote machine and scp [local file] [remote
user]@[remote IP]:[destination directory].

Congure SSH server

Alter the line PubkeyAuthentication so that


it reads PubkeyAuthentication yes. Make sure that
PasswordAuthentication is set to no. This is because
we want to use encrypted keys rather than usernames
and passwords to remotely log into the machine. Save
the file and type sudo service ssh restart to
restart the server.

04

File copying (SSH)

Generate keys

We need to generate both a public and


private key so that we can log into SSH without using

SSH into the remote box from the client


by typing ssh [remote username]@[remote IP
address]. This should give you command line on the
other machine without prompting you for a password.
Try out a few commands to test it, and type Ctrl+D to
end the session.

08

Web proxy (SSH)

Lets say that the user complains that


they cant access a specic website or the web

Linux & Open Source Genius Guide 41

Tips & Tricks

12

Connect to VNC

13

Install Webmin

There are a few VNC clients, but this example


presumes that you have installed Vinagre. Use it to
connect to the VNC server at localhost. Remember,
we connect to localhost because we set up the
SSH tunnel for all trafc to the remote computer on
port 5900 in the previous step. You should now have
remote desktop access to the remote computer. If
there are any problems with the connection, examine
the text output in the terminal for errors.

in general. We can test the connection by using


SSH tunnelling. Type ssh -D 8888 [username]@
[IP address] to begin. This will establish a local
proxy server that is actually accessing the web via
the remote machine. You can then congure Firefox
to access the web through this proxy by specifying
Socks 5 proxy on localhost. If you can SSH to the
other machine and access the problem site, it must
be a software problem on the other machine, rather
than a networking problem.

09

Run X applications (SSH)

You can use tunnelling to remotely run


X applications. Just the thing to, for example,
edit the conguration in a GUI application
without using full desktop sharing. Type ssh -X
[username]@[remote IP] to open the connection.
You then start the application in the normal
way from the command line. Type firefox &
to start Firefox on the remote machine and
interact with it on your local desktop. You can
reduce things to a single command by typing ssh

-f -T -X [username]@[ip address] [name of


application].

10

apt-get install x11vnc (or yum install vncserver on Red Hat).

Set up the VNC server

Were going to install VNC for remote


desktop access. In this case, were going to ignore
the VNC servers own security features and simply
tunnel it over SSH instead. Begin by installing the
VNC server on the remote machine by typing sudo

42 Linux & Open Source Genius Guide

11

Set up VNC tunnelling


and start server

Set up an SSH tunnel for VNC by typing ssh -L


5900:localhost:5900 [remote user]@[remote
IP]. As well as setting up the tunnel, this launches
a terminal connection. In the terminal, type x11vnc
-safer -localhost -nopw -once -display :0.

Proceed to the downloads section of


the Webmin website (www.webmin.com) and
download the package for your Linux architecture.
In the case of Debian, install the .deb le by
typing sudo dpkg -i [name of .deb you
downloaded]. Follow this up by typing sudo aptget -f install to satisfy the dependencies. In
the case of Red Hat, type yum install [name of
RPM].

In this case, were going to ignore


the VNC servers own security
features and simply tunnel it over
SSH instead

Linux & Open Source

Genius Guide

14

Test Webmin

Webmin runs in SSL mode by default, so


access it by browsing to https://[IP address of remote
computer]:10000. Accept the browser certicate that
it offers. Use either the root username and password
or regular username and password that can run sudo,
in the case of Ubuntu. In the future, tunnel to Webmin
for maximum security without having to open up an
extra port on the router. We use the same technique
as with VNC. Type ssh -L 10000:localhost:10000
[username]@[IP address] in order to set it up and
then browse to https://localhost:10000.

16

Set up static address

How you set up a static IP address varies from


distribution to distribution. On an Ubuntu machine,
click on the Network icon in the Control Panel and
select Edit and then select the IPv4 Settings tab.
Select Manual from the Method: drop-down list.
Now add an IP address that is part of the current
subnet. For example, if typing ifconfig reveals
that the current IP address is 192.168.0.5, choose
something higher such as 192.168.0.100. You may
have to manually add DNS server addresses for your
ISP. Google for them. Accept the changes and reset
the machine.

15

18

Locate remote IP address


on the internet

19

Set up ddclient

You can ask the person who runs the remote system
to simply visit whatismyip.com and email their
current IP address to you, but setting up dynamic
DNS is a more robust solution. Go to the No-IP
website (www.noip.com) and set up a free account.
Now install the ddclient package.

Set up the remote network

Its time to return the computer that we have


set up for remote maintenance to its home base.
Once the computer is reconnected to the network,
check that it can access the internet as usual.

Add/alter the following lines in


/etc/ddclient.conf:

17

Port forwarding

Set up port forwarding by making use of


the documentation for your router (or portforward.
com). You need to forward port 22 to the static IP
address that we set up.

protocol=dyndns2
use=web, web=checkip.dyndns.com/, webskip='IP Address'
server=dynupdate.no-ip.com
login=user name
password='password'
[hostname].no-ip.biz
run_daemon=true
Now restart it by typing sudo service ddclient

restart.

Linux & Open Source Genius Guide 43

Tips & Tricks


ownCloud automatically categorises
the uploaded les. Different types can
be accessed via the links here

Handily, there is a built-in


PDF viewer available
in ownCloud

This is the point from


where you upload your les
to ownCloud

The built-in image viewer can


show the pictures uploaded
to ownCloud

The expanded menu shows


the different admin actions
available

Hover the cursor over a le and a menu


allows you to rename, download, share or
check other versions of that le

Build your own private cloud


with ownCloud
Advisor

Nitish Tiwari is a software


developer by profession and an open
source enthusiast by heart. As well
as writing for leading open source
magazines, he helps rms set up and
use open source software for their
business needs

Resources
Server like LAMP/WAMP/MAMP
ownCloud plug-ins:
apps.owncloud.com

44 Linux & Open Source Genius Guide

A fast-track guide to setting up your own


le management system in the cloud
using ownCloud
ownCloud lets you create your own le
management/sharing/backup
system
without having to rely on a third-party
cloud service. It also provides other
functionality like contacts management, calendar
management, plug-in support, users, groups etc.
All these features make ownCloud a fully edged
enterprise-level le management tool.
In this tutorial we will provide a step-by-step
installation guide to set up ownCloud on your system,
although setting it up on a third-party server (for
example a web server provider) will probably be easy

because you dont need to install the server. We will


then explore some other use cases where it can be
deployed. Then we will go through some of the thirdparty apps/plug-ins available, which can be hooked
onto ownCloud to provide further functionality.
Were using ownCloud version 5.0.13 and the
dependent PHP versions for this tutorial, although
some of the plug-ins discussed here may require
older versions of ownCloud. Please check the
corresponding plug-in documentation link before
trying to install the plug-in. There has also recently
been an upgrade to ownCloud version 6.

Linux & Open Source

Genius Guide

01

Introduction

ownCloud is an open source le sync and


sharing application, available in a free community
edition as well as an enterprise edition. It allows
you to back up, share and manage les uploaded to
the server. With multiple interfaces like the web
UI and Android/iPhone apps it allows you to be in
touch with your data at almost any point of time. The
ownCloud desktop client for Windows, Mac OS X and
Linux lets you sync your les seamlessly with the
ownCloud server, akin to Google Drive or Dropbox
clients. With this full ecosystem support for le
syncing, ownCloud truly is all about Your Cloud, Your
Data, Your Way, as they say in the documentation.

02

Installing the server

There are two possible setups here. You


may want to install ownCloud on your system or in a
small home/ofce setup, where it is accessible in a
LAN. Otherwise you may want it to be available on the
internet. In the latter case, you can skip this step, but
if you are planning the former, you will have to identify
a machine as the server and install a server such as
Apache to that system. But, as discussed earlier, a
server is not the only requirement for ownCloud. You
also need a database and PHP support. So, a LAMP/

WAMP or MAMP server is probably the best way


forward, since all the required tools come bundled
with these servers. Your advisor uses the MAMP
server in his system to host ownCloud.

03

Installing ownCloud

With the server active, we can try to host


the ownCloud application. Before that, download
the application code from ownClouds ofcial
website: just click on the Tar or Zip le link on
the ownCloud install page. Unzip the archive and
place the extracted folder in the root of the server.

Now, access the folder via your web browser. The


welcome page opens up and prompts you to enter
the admin ID and password. Below the prompt
youll also see an Advanced link, where you can
change settings related to the database etc. Once
done, just click on Finish Setup and thats all! You
have your ownCloud.

04

Get going with ownCloud

Now that ownCloud is installed, and the


admin user created, you may want to add more
users and then each of those users may want to
upload their les. In this step we will see how to
do this. First, the user management click on
the admin button in the top-right corner of the
homepage. In the drop-down list that appears,

Linux & Open Source Genius Guide 45

Tips & Tricks

left side of the homepage lets you create events


and share them with other users or groups. It also
supports multiple calendars and syncing with iCal.
The contacts option supports adding and managing
the contact details. Contacts can also be uploaded
as .vcf les and, like les and events, you can even
share the contacts among users or groups.

click users. This takes you to the user management


page where you can add/remove users and also
segregate them in groups. You also have the option
to assign the admin for a group. To upload les, you
can simply click on the new button in the ownCloud
homepage and then select the le to upload in the
upload window.

05

07
credentials. The client then connects to the server
seamlessly and starts syncing. The app is highly
congurable and lets you change many settings,
including bandwidth usage and multiple sync folder
support. Also, it is available for all the major platforms:
Windows, Linux and Mac OS X.

ownCloud sync client

Along with the server application, ownCloud


also provides the desktop sync client, which can be
installed on the users system. The sync client makes
sure any les present in its folder get uploaded
automatically to the server. Just download the client,
point it to your ownCloud server URL and enter your

46 Linux & Open Source Genius Guide

06

Other applications of ownCloud

It may appear straightforward, but there


is a lot more to ownCloud than just le syncing.
ownCloud not only helps sync les between multiple
devices and platforms, it also lets you manage your
calendar and contacts. The calendar link on the

Roundcube mail plug-in

With contacts and calendar support,


you would probably start to think, why isnt email
supported? Thankfully, with support for external plugins, ownCloud lets you extend the functionality in any
way you wish. Open source webmail client provider
Roundcube offers an external plug-in which brings
your mailbox to ownCloud. To install the plug-in, go
to the ownCloud ofcial plug-in portal, download it
and paste the downloaded les to the htdocs/apps
folder. Now, click on the apps link in the drop-down
that appears after clicking on admin (at the top-right
corner of the page). Here youll see all the apps; go to
the Roundcube app and enable it. This enables the
app, but you still need to create the database and
congure the web server. For more details, you can
check the installation guide.

Linux & Open Source

Genius Guide

ownCloud not only helps sync les


between devices, it also lets you manage
your calendar and contacts

once encryption is enabled because if you forget


your password, there is no way to retrieve the data by
default. To protect yourself against such loss, you can
enable recovery key in the ownCloud admin settings
for encryption.

rescue. This plug-in lets you create and store short


links from the web. It comes as a preloaded plug-in with
ownCloud. To start using it, you just need to enable it
from the apps page. Once enabled, you can just drag
and drop the button from the Shorty interface to add
a site to your list. You can also shorten the URLs with a
configurable back-end service like goo.gl, ti.ny etc.

08

Shorty plug-in for


weblinks management

With the internet having grown so vast, its become


difficult to track webpages and the content you like.
On top of that, there are now so many devices: phone,
office PC, home PC etc. So, the link you saw at home
yesterday is difficult to find when you want to show it
to your colleagues in the office. Shorty comes to your

09

File encryption plug-in

Security and privacy have become major


areas of concern in recent years. With ownCloud you
can be sure of using your own server; still, encrypting
the data makes the whole setup even more secure.
ownCloud ships with an encryption plug-in and once
you enable it, all your files get automatically encrypted.
The encryption is done server-side and only the
ownCloud portal can decrypt the data, using a key
that is generated with your password. So be careful

10

Journal plug-in

The last plug-in we will look at in this article


is the journal entry one. This plug-in lets you create
journal entries in your ownCloud calendar. The plug-in
ships with ownCloud, but before you enable it, enable
the TAL Page Templates plug-in (also preloaded with
ownCloud). The entries are saved as VJOURNAL
records in the calendar and can be sorted and filtered
by date/time.

Linux & Open Source Genius Guide 47

Tips & Tricks

Capturing network data


and displaying it on
screen using tshark

Applying a lter during


network data capturing

Applying a display lter


using tshark

Displaying statistics on
the captured data

Monitor network traffic


Advisor
Mihalis Tsoukalos is
a UNIX system administrator
with expertise in programming,
databases and maths. He has
been using Linux since 1993

Resources
tshark:
www.wireshark.org/docs/man-pages/tshark.html

Wireshark:

www.wireshark.org

DHCP, RFC 2131:

www.ietf.org/rfc/rfc2131.txt

Display Filter Reference:


www.wireshark.org/docs/dfref/

48 Linux & Open Source Genius Guide

Python script
avaliable online:
http://bit.
ly/1gH6W0S

Use tshark to examine network trafc,


solve network difculties and add
network data to a MongoDB database
Gerald Combs created Ethereal, the
ancestor of Wireshark, back in 2006.
When he went to work in a new job, he could
not use the name Ethereal any more so he
renamed his tool Wireshark. The rest is history!
This tutorial will present you tshark, the commandline version of Wireshark, which is a very popular
and capable network protocol analyser. The main
advantage of tshark is that it can be used in scripts. Its
main disadvantage is that it does not have a GUI.
You can get tshark either from its website by
compiling its source code or directly from your Linux
distribution. The second way is quicker and simpler.

If you try to run tshark as a normal user, you


may not be able to use any network interfaces for
capturing network trafc due to UNIX permissions.
Your advisor nds it more convenient to run tshark
as root (sudo tshark) when capturing data and as a
normal user when analysing network data.
Before you start capturing, it is better to have
a given issue that you want to solve or examine in
mind. This is the rst step for a successful network
trafc analysis.
If you are already familiar with Wireshark, learning
how to use tshark will be easy for you. Having a good
knowledge of TCP/IP comes in handy too.

Linux & Open Source

Genius Guide

01

Installing and running tshark

In order to install tshark on a Debian 7


system, you just have to run the following command
as root:

# apt-get install tshark


To nd out if tshark is properly installed, as well as its
version, you can execute this command:

$ tshark -v

02

Capturing network
data using tshark

Tshark can be used as a replacement for tcpdump,


which is the industry standard for network data
capturing. Apart from the capturing part where both
tools are equivalent, tshark is more powerful than
tcpdump and therefore if you want to learn just one
tool, tshark should be your choice.
The rst command you should run is tshark D to
list the available network interfaces.
The simplest way of capturing data is by running
the tshark command without any parameters. You
will get the output on screen which, as you can
easily understand, is not helpful at all!

03

Two command-line parameters

The single most useful command-line


parameter is -w, followed by a lename. This
parameter allows you to save network data to a le for
later processing.
The following tshark command captures 500
network packets (-c 500) and saves them into a le
called test.pcap (-w test.pcap):

$ tshark -c 500 -w test.pcap


Another useful parameter is -r, followed by a
lename, which allows you to read and analyse a
previously captured le.

04

Applying lters during capturing

Tshark allows you to lter network data


by capturing specic types of trafc, avoiding the

creation of huge capture les. This can be done using


the -f command-line parameter followed by a lter in
double quotes.
The most important TCP-related eld names are
tcp.port, for ltering the source or the destination TCP
port; tcp.srcport, for checking the TCP source port;
and tcp.dstport, for checking the destination port.
Generally speaking, applying a lter after data
capturing is considered more practical and versatile
than ltering during the capture stage because most
of the time you do not know in advance what you want
to inspect. Nevertheless, using lters during network
capturing can save you time and disk space and that
is the main reason for using them.
Remember that the lter strings should always be
written in lower case.

05

Applying lters
after network capturing

Filters that are applied after data capturing are


called Display Filters by tshark and Wireshark.
You should use the -R command-line parameter
followed by the Display Filter in double quotes.

The http.response.code != 404 display lter


searches for HTTP trafc with a response code
not equal to 404. The tcp.port == 80 && ip.src ==
192.168.2.2 display lter searches for TCP trafc
that both uses port number 80 and comes from the
192.168.2.2 IP address. If you have an error on your
Display Filter, tshark will let you know by displaying
an error message.
As you can easily understand, the possibilities are
endless and only depend on your imagination and
the problem you are trying to solve.
If you deeply understand Display Filters and have
a good knowledge of TCP/IP and networks then
network problems will not be a problem!

06

Exporting captured data


into a readable format

Imagine that you want to extract the frame number,


the relative time of the frame, the source IP address,
the destination IP address, the protocol of the packet
and the length of the network packet from previously
captured network trafc. The following tshark
command will do the trick for you:

Linux & Open Source Genius Guide 49

Tips & Tricks

$ tshark -r login.tcpdump -T fields -e


frame.number -e frame.time_relative -e
ip.src -e ip.dst -e frame.protocols -e
frame.len -E header=y -E quote=n -E
occurrence=f
The E header=y option tells tshark to rst print a
header line, the E quote=n dictates tshark not to
include the data in quotes and the E occurrence=f
tells tshark to only use the rst occurrence for elds
that have multiple occurrences.

07

Solving a DHCP problem

08

More about the DHCP protocol

The problem: some computers on a network


could not connect to the network although other
computers were okay. All computers were using the
DHCP protocol to get their network settings. The IP of
the ofcial DHCP server was 10.0.10.10.
DHCP is short for Dynamic Host Configuration
Protocol and is a protocol that provides configuration
information to hosts on TCP/IP networks. DHCP is
based on BOOTP (the Bootstrap Protocol) and extends
it by adding more capabilities. DHCP and BOOTP
protocols are both using the UDP protocol with UDP
ports 67 and 68.

The Python script inserts network


data into a MongoDB database for
further processing and querying. The
name of the script is insertMongo.py
50 Linux & Open Source Genius Guide

The rst packet of a usual DHCP transaction


between a DHCP client and a DHCP server (IP
192.168.1.1) has a DHCPDISCOVER message from
the machine searching for a DHCP server. Since the
machine does not have an IP address yet, the source
IP of the packet is 0.0.0.0 and the destination IP is the
broadcast IP address (255.255.255.255).
What distinguishes the network card of a machine
from another network device found in the same LAN
is its MAC address, which is unique. Therefore the
DHCPDISCOVER message should include the MAC
address of the device requesting a DHCP server.
The next message is the DHCPOFFER from the
DHCP server (IP 192.168.1.1) and is a broadcast
message since the client machine still has no IP
address.
Then the client machine requests from the
DHCP server conguration parameters with
the DHCPREQUEST message. Next, the DHCP
server sends a DHCPACK message back to the
client machine that includes all the conguration
parameters.
From now on, the DHCP client can use the offered
conguration information and any parameter that is
unique to that particular machine, like the IP address,
is reserved by the DHCP server and is not offered to
any other networked device.

Linux & Open Source

Genius Guide

09

Solving the problem

Tshark output shows that there were two


DHCPOFFER messages on the network from two
different IP addresses (192.168.1.254 and 10.0.10.10)
instead of only one DHCPOFFER message from the
legitimate 10.0.10.10 DHCP server! This was the rst
truly useful hint for solving the actual problem.
As the DHCP server did not get any answer from the
client, it re-sent the DHCPOFFER message (packet
number 6), but as you can see, it was already too late
(packet number 4)!
The IP address of the extra DHCP server was
192.168.1.254. The 192.168.1.254 DHCP server
offered the 192.168.1.60 IP address to the machine.
As you can guess, all computers that could not
properly connect to the network were getting IPs in
the 192.168.1.1-253 range.
The client machine preferred the wrong DHCP
server to get its information. The reason for

choosing the 192.168.1.254 DHCP server was that it


responded rst! Pretty naive reason, yet it caused
many problems.
After nding out that there was a second DHCP
server that triggered the problem, it was easy to
nd out the computer that caused the problem.
This particular computer was running a Linux
virtual machine (VM). The OS on the VM had its
DHCP server running and that was the cause of the
problem! Pretty tricky, dont you think?

10

using the appropriate command-line arguments


using the following Perl commands:

my $command = "$TSHARK_BINARY -r $filename -T


fields -e frame.number -e ip.src -e ip.dst
-E header=y -E quote=n -E occurrence=f";
my @netDATA = `$command`;

Creating a Perl script that uses tshark

The purpose of the checkIP.pl Perl script is


to nd invalid IP addresses. The checkIP.pl script
assumes that the network data is already captured
with tshark.
Several steps are needed in order to solve the
problem. The rst step is reading the le with the
network data. Next, it is running the tshark binary

11

More about the Perl script

The next step is reading the output of the


tshark command that was saved in the @netDATA
variable line by line. After cleaning up input lines
from unnecessary space characters and parsing it,
the script uses the Data::Validate::IP Perl module for
catching erroneous IP addresses and then prints the IP
on screen:

if ( ! is_ipv4($sourceIP) )
{
print "Packet number $frameNumber
contains a bogus source IP!\n";
}
You can alter the script in order to catch the type of
errors you want, such as trafc from unwanted hosts
or trafc to specic TCP or UDP ports.

12

Inserting network data into a MongoDB


database

The Python script supplied online inserts network


data into a MongoDB database for further processing
and querying. The name of the script is insertMongo.
py and it assumes that the network data is already
captured with tshark or tcpdump. The next shell
command runs the Python script with input from
tshark:

$ tshark -r ~/1000.pcap -T fields -e frame.


number -e ip.src -e ip.dst -e frame.len
-E header=n -E quote=n -E occurrence=f |
python insertMongo.py
Total number of documents written: 1000
The 1000.pcap le contains 1,000 network packets
and the script informs you that there were 1,000
documents written in MongoDB, so you know that
everything is okay. You can now start querying the
MongoDB database!

Linux & Open Source Genius Guide 51

Tips & Tricks


Other architectures such
as Microsoft Windows can
connect to our server, using
the ofcial client

You can add the VPN server to


your existing Linux setup. Well
run it manually at first, so that
we can monitor its output

The Linux clients run the


same software but with a
different cong le

Android-powered devices
such as phones and tablets
can also connect

Access your network


anywhere with a VPN
Advisor
Michael Reed is
a technology journalist
and hes been hacking
away at Linux for over
15 years

Resources
A Linux box
A second Linux box on a LAN (optional)
An Android phone (optional)
A Microsoft Windows box (optional)

52 Linux & Open Source Genius Guide

Work around insecure and restrictive internet


access with your own virtual private network
This tutorial will show you how to add a
VPN server to your existing Linux setup.
This means that you can connect, securely,
to a computer that you trust by using a
technique called tunnelling. Using this connection,
you can securely access the internet from your remote
computer, bypassing any filtering that is part of the
policy of the Wi-Fi provider. You can access anything
that your properly set up computer in your home or
your office can access. The connection is a secure,
encrypted one. This is significant, because in theory it is
possible to eavesdrop on the communications that are
carried out using a public Wi-Fi hotspot.

In addition, you can use a VPN (virtual private


network) connection to access any resources on
your network, because effectively you are on that
network. So you need never curse yourself for
forgetting to bring those important les with you
ever again.
Once the server is set up, you can connect to it
using a variety of clients. On a Linux machine, you
will typically use the same service as the server
with a slightly different conguration le. Microsoft
Windows computers and Android devices such as
smartphones and tablets have ofcial and thirdparty OpenVPN clients available for them.

Linux & Open Source

Genius Guide

chmod 400
/etc/openvpn/{server.key,server.
crt,ca.crt,ta.key}.

01

Install OpenVPN on the server

02

Copy easy-rsa scripts

06

Generate three client keys


Use the command ./build-key client1

07

Begin server conguration

08

Locate the example configuration


Use find /usr -name server.conf* to

to generate the rst of the client keys. When


prompted, accept client1 as the common name.
Each key must have a unique common name. Now
repeat the procedure for two more client keys. As
before, agree to sign the certicate and commit.

Install OpenVPN using the package


manager. For Red Hat, type yum install
openvpn or apt-get install openvpn on Debian
(both root). In addition, install easy-rsa, a collection of
scripts that aid in the creation of the certicates and
keys that we need.

We cant run easy-rsa from where it


is installed because we need to alter it. Find
the installation directory with find /usr -name
easy-rsa. Copy the directory to your current directory
with (for example) cp -r /usr/share/easy-rsa/. and
move into it with the cd command.

03

04

Build server certicates and keys


Type source
./vars followed by
./clean-all and then ./build-ca. Fill in the

questions when prompted you should be able to hit


Return to accept the defaults in most cases. Build the
certicates and keys for the server with the command
./build-key-server server. Answer questions as
before and accept server as the Common Name.
Dont specify the challenge password or optional
company name when asked. Answer y when asked
about signing the certicate and when asked to
commit. Dife-Hellman parameters are an element in
the encryption scheme that OpenVPN employs. Type
./build-dh to create the parameter le.

Install OpenVPN on the server

Open the le vars in a text editor. The


ofcial OpenVPN documentation recommends
that you edit at least the KEY_COUNTRY,
KEY_PROVINCE,
KEY_CITY,
KEY_ORG and
KEY_EMAIL elds. Set KEY_CN to server. Close and
save the le.

05

Copy the server keys

Enter the keys directory with the


cd command, and type ls keys to examine
the key les that we have created. Use
cp * /etc/openvpn/ (as root) to copy all of the
keys to where OpenVPN can see them. Then do

The server and each client use a configuration


file located in the /etc/openvpn folder. On the server,
when the OpenVPN service starts, it scans that
directory and launches a service instance for every
valid configuration file it finds. Start by shutting down
the OpenVPN service by typing service openvpn
stop (as root).

locate the example server conguration le. Use


cut and paste within the terminal to copy it to the
current directory with the cp [name of file] .
command sequence. If it has a .gz extension, its
compressed. If so, decompress with gunzip [name
of file].

Linux & Open Source Genius Guide 53

Tips & Tricks

client1.crt to /etc/openvn/ on the client machine.


How you do this is up to you, but bear security in
mind as anyone who has these les can, potentially,
use them to access your server. A memory stick
(delete them afterwards) or a trusted network such
as your LAN should be ne.

12
09

Edit server conguration

Open server.conf in a text editor and give it


a quick examination to get an idea of what you can
change, but well leave most of it as it is. For now, just
add /etc/openvpn/ to the start of the key filenames.

Type openvpn server.conf (as root). If everything is


working okay, the final message should be Initialization
Sequence Completed.

11
10

Start the server

For now, well start the server manually so


that we can see the text output and check for errors.

Open client.conf in a text editor. Change


remote to the IP address of the server. For now,
use the IP address of the server on your LAN (run
ifconfig on it). Later, well change this to the
external IP address so that it can be accessed
from the internet. Change the CA certicate (ca) to
ca.crt, the user certicate (cert) to client1.crt and
the private key (key) to client1.key, each with /etc/
openvpn/ at the beginning of the name. Each client
needs a copy of the CA and unique versions of the
other two les.

Congure the client

Move to another Linux machine on your


LAN (the client). Install OpenVPN using the
package management system. Now run find
/usr -name client.conf to nd the example
client conguration le and copy it to the current
directory. Copy the les client1.key, ca.crt and

We can connect to the server,


via the tunnel, but we cant yet
access the internet
54 Linux & Open Source Genius Guide

Edit the client conguration

13

Test the client


Type openvpn client.conf (as root)
to start the client. Type ifconfig to conrm

the creation of the tun0 device. If you can


ping the server from the client by typing
ping 10.8.0.1, you have now established a
tunnelled, encrypted connection between the client
and the server.

Linux & Open Source

Genius Guide

14

Enable packet forwarding

As it stands, we can connect to


the server, via the tunnel, but we cant
yet access the internet. Start by editing
/etc/sysctl.con and uncomment the line net/ipv4/
ip_forward=1 to enable packet forwarding. Now
restart the server machine.

resolvconf/resolv.conf.d/head and type resolvconf


-u (all as root). You should now be able to ping and
browse to sites on the internet from the client.

17
15

Forward trafc with iptables


On the server, type iptables -t nat
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j
MASQUERADE (as root) to enable forwarding of trafc

Forward trafc with iptables

On the server, copy server.conf to /etc/


openvpn/. We now control the OpenVPN server with
service openvpn [command]. Command can be
start, stop or restart and the check command makes
the service start on boot. Add the iptables we used to
the end of /etc/rc.local.

through the tunnel to the internet-connected


interface (eth0). Install the dnsmasq package to
your system. Add the line interface=tun0 to /etc/
dnsmasq.conf. Restart dnsmasq by typing service
dnsmasq restart.

Set up routing for the client


Add the lines push "redirect-gateway def1"
and push "dhcp-option DNS 10.8.0.1" to server.conf.
This causes the clients to execute the appropriate
route commands to direct all network trafc via the
VPN. You may have to manually add the DNS server
to the client: add the line nameserver 10.8.0.1 to /etc/

Connecting from an Android device

20

Connecting from a Windows client

Most Android phones and tablets dont include


built-in OpenVPN support. Search for an OpenVPN
client on Google Play. Your choice of client may
depend on compatibility with your device, and some
applications require a rooted device. Copy the key files
to your SD card and use the setup page of the client.

18
16

19

Connecting from the outside

Set up your router to forward UDP port


1194 to the IP address of your server. How you
do this depends on the model of your router (see
portforward.com for a comprehensive database
of router models). Update the remote eld in your
client.conf les to point to the external IP address of
your network (use www.whatismyip.com to nd it).
Consider using a dynamic IP service such as No-IP
(www.noip.com) so that you dont have to update the
clients every time the address on your router changes.

The OpenVPN website (openvpn.net) has


a client that runs on Windows. Copy the three client
les to the machine, then install the client and follow
the on-screen prompts. The Windows client can make
use of the push directives that we added to server.
conf, so we dont need to set up DNS manually.

Linux & Open Source Genius Guide 55

Tips & Tricks


Gnuplot can nd
the intersection of
two parabolas

Gnuplot can draw


difcult plots

Gnuplot can plot


many lines on the
same output

Gnuplot can do
3D too

Use gnuplot for drawing


Advisor
Mihalis Tsoukalos is
a UNIX system administrator
with expertise in programming,
databases and maths. He has
been using Linux since 1993

Resources
Gnuplot homepage:
www.gnuplot.info

Gnuplot Python package:


gnuplot-py.sourceforge.net

3D plots using gnuplot:


www.packtpub.com/article/3d-plot-using-gnuplot

56 Linux & Open Source Genius Guide

Gnuplot is a powerful 2D and 3D program


that can be used to plot mathematical
functions and also show scientic data
This tutorial will introduce you to a very
useful scientific tool called gnuplot.
Gnuplot is copyrighted but freely
distributed. There are versions of gnuplot
for UNIX, IBM OS/2, MS Windows, DOS, Mac OS X,
VMS, Atari and many other platforms!
Gnuplot is a command-line tool that was initially
developed in order to allow scientists and students
to visualise mathematical functions and data. It now
supports web scripting and integration as a plotting
engine for third-party applications like Octave.
Gnuplot can draw many types of plots in both two
(2D) and three (3D) dimensions. It supports drawing

using lines, points, vector elds, surfaces, boxes and


contours. It also supports output in many different le
formats, including EPS, FIG, JPEG, LaTeX, METAFONT
(invented by Donald Knuth the creator of TeX), PDF,
PNG and PostScript.
Although gnuplot can show its graphical output
on screen with the help of a windows manager, the
gnuplot commands that are going to be used here
will capture the output directly to a le. This implies
that gnuplot is suitable for scripting. There is also
a Python module for using gnuplot while writing
Python code, called py-gnuplot, which will be
explored in this tutorial.

Linux & Open Source

Genius Guide

01

Install gnuplot

You can install the command-line version


of gnuplot on a Debian system by using the following
command:

# apt-get install gnuplot-nox


Alternatively, you can install the gnuplot-qt package
that is the Qt version, or the gnuplot-x11 package that
is the X-package version. Colin D Kelley and Thomas
Williams rst developed gnuplot back in 1987.
Note: When executing gnuplot commands, please
remember that the gnuplot language is case sensitive.

02

First run

You can nd the version of gnuplot you are


using by executing this command from the shell:

$ gnuplot -V
gnuplot 4.6 patchlevel 0
If you run the gnuplot command without any
arguments you will enter the gnuplot shell, as shown
in the screen above.
Type quit at the gnuplot prompt to leave the gnuplot
shell and return to the UNIX shell.

03

A simple gnuplot example

The rst example will show how to draw the


2*x linear function.

gnuplot> set term png


Terminal type set to 'png'
Options are 'nocrop font "/usr/share/fonts/
truetype/liberation/LiberationSans-Regular.
ttf,12" fontscale 1.0 size 640,480 '
gnuplot> set output "plot2x.png"
gnuplot> plot 2*x
gnuplot> quit
mtsouk@mail:~/docs/article/working/GNUplot.
LUD/code$ ls -l
total 4
-rw-r--r-- 1 mtsouk mtsouk 3958 Oct 26 19:17
plot2x.png
The first command tells gnuplot that the output will use
the PNG file format. It then defines the filename that
will be used for saving the output image. After that, it
defines the function to plot. After quitting gnuplot, you
can find the output file in you current directory.

The default size of the output is 640x480 pixels.


If you want to generate a higher analysis output
(2048x2048 pixels), you can give the following
command instead:

gnuplot> set term png size 2048, 2048


You can dene the size of the output according to your
needs.

04

Plot mathematical functions with


gnuplot part 1

The previous output was very simplistic. In this part,


you are going to learn some commands that will help
you beautify the output, add some text to it and make
it look more professional.

gnuplot> set term png size 2048, 2048


Terminal type set to 'png'
Options are 'nocrop font "/usr/share/fonts/
truetype/liberation/LiberationSans-Regular.ttf,
12" fontscale 1.0 size 2048,2048 '
gnuplot> set output "plotSinCos.png"
gnuplot> set xrange [-2*pi:2*pi]
gnuplot> set title "Linux User and Developer"
gnuplot> set xlabel "x-axis: from -2*pi to
+2*pi"

gnuplot> set ylabel "Setting the y-axis label"


gnuplot> plot sin(x) * cos(x)
gnuplot> quit
As you can comprehend, the set xrange command
denes the graph boundaries for the x-axis. For
a trigonometric function such as sin(x) * cos(x), it
makes more sense to dene boundaries relative to
the number pi.

Linux & Open Source Genius Guide 57

Tips & Tricks

07

Using gnuplot and Python

On a Debian 7 system, you can install the


gnuplot Python package by executing the following
command:

# apt-get install python-gnuplot


In order to make sure that python-gnuplot was
installed correctly, you can run the following one-liner
Python script and check its output for errors:

Running the Python script produces the following


output because of the debug=1 option:

$ ./LUD.py
gnuplot> set term png size 2048, 2048
gnuplot> set output "LUD.png"
gnuplot> plot cos(x)/x title "Using the
Gnuplot Python module"

$ python -c 'import Gnuplot'


If everything is okay, you will immediately see the
shell prompt. If the module is not installed, the error
message will look like the following:

$ python -c 'import Gnuplot'


Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named Gnuplot

05

Plot mathematical functions with


gnuplot part 2

You can further enhance the output by adding


grid lines to it with the following gnuplot
commands:

gnuplot> set xtics ("0" 0, "-180" -pi, "-90"


-pi/2, "90" pi/2, "180" pi)
gnuplot> set ytics ("0" 0, "0.5" 0.5, "-0.5"
-0.5)
gnuplot> set grid

09

Execute the following gnuplot code to get an


impressive 3D output:

gnuplot> set term png size 2048,2048 crop


Terminal type set to 'png'
Options are 'crop font "arial,12" fontscale
1.0 size 2048,2048 '
gnuplot> set output "3D.png"
gnuplot> set param

Please note that gnuplot has a built-in help system.


You can enter, for example, help plot and get useful
information about the plot command just try it!

08

A simple Python and gnuplot script

The next Python script uses the gnuplot


Python module:

06

Plotting two functions at once

Lets say that you want to see two functions


on the same graph in order to compare them. Gnuplot
can assist you with this. The following command
demonstrates how:

gnuplot> plot x*x, x*x*x


As you may understand, all you have to do is separate
the two functions with a comma! If you want to make
the output look even better, you can give the following
command:

gnuplot> plot x*x, x*x*x with filledcurves fs


As you may recognise, the with lledcurves fs does
the job for you it lls the second function with green,
which is the default colour for the current curve.

#!/usr/bin/python
import Gnuplot
gp = Gnuplot.Gnuplot(debug=1)
gp('set term png size 2048, 2048')
gp('set output "LUD.png"')
g5 = Gnuplot.Func(cos(x)/x', title= 'Plotting a
function with py-gnuplot')
gp.plot(g5)
The import Gnuplot command loads the correct
Python module, whereas the gp.plot(g5) command
plots a graph that is called g5. The gp(set term png size
2048, 2048) command defines the output file format
and its dimensions, whereas the gp('set output "LUD.
png"') command dictates the filename that will be used
for saving the output.

Gnuplot gives you the choice of


plotting multiple graphs side-byside on the same output screen
58 Linux & Open Source Genius Guide

Going 3D!

dummy variable is t for curves,


u/v for surfaces
gnuplot> set isosamples 50
gnuplot> set ztics .5
gnuplot> set xtics .4
gnuplot> set ytics .4
gnuplot> set urange [-pi:pi]
gnuplot> set vrange [-pi:pi]
gnuplot> set pm3d depthorder
gnuplot> splot cos(u)*cos(v), sin(u)*cos(v),
sin(u) with pm3d
smooth palette in png: using 160 of 160
available color positions
gnuplot>
The splot command can display a surface as a
collection of points, or by connecting those points.
The set isosamples 50 command sets the number
of points at which a function of two variables will be
evaluated to generate a surface plot.
The special pm3d option was set as pm3d
depthorder. This tells gnuplot to draw the surface
starting with the elements farthest away from our
viewpoint, producing a more pleasant output. The with
pm3d (palette-mapped three-dimensional) parameter
of the splot command is for drawing solid surfaces.

Linux & Open Source

Genius Guide

10

Using multiplot mode

Sometimes it is preferable to show multiple


plots at once. Gnuplot gives you the choice of
plotting multiple graphs side-by-side on the same
output screen. The key point for doing this is the set
multiplot command. If you give this command, the
prompt will change from gnuplot to multiplot. The
following commands generate a very pretty output:

gnuplot> set term png size 2048, 2048


Terminal type set to 'png'
Options are 'nocrop font "/usr/share/fonts/
truetype/liberation/LiberationSans-Regular.ttf,12"
fontscale 1.0 size 2048,2048 '
gnuplot> set output "MultiPlot.png"
gnuplot> set xrange [-1:1]
gnuplot> set size 1,1
gnuplot> set origin 0,0
gnuplot> set multiplot
multiplot> set size 0.5,0.5
multiplot> set origin 0,0.5
multiplot> plot x
multiplot> set size 0.5,0.5
multiplot> set origin 0.5,0.5
multiplot> plot x*x
multiplot> set size 0.5,0.5
multiplot> set origin 0,0
multiplot> plot x*x*x
multiplot> set size 0.5,0.5
multiplot> set origin 0.5,0
multiplot> plot 1/x
multiplot> unset multiplot
gnuplot> reset
The size and origin commands position each graph
in its place and make it take up just a quarter of the
screen. The reset command clears all labels on the
axes as well as all other settings. It is very useful if
you want to start a new graph from scratch.

11

Plotting data les

Imagine that you have a le that contains


groups of data points that you want to plot. Gnuplot

12

Using gnuplot scripts

As you can imagine, it would prove very tedious


to type every single gnuplot command each time you
wanted to plot something. Gnuplot allows you to write
the commands you want to execute in a plain text file
and run it as a script. To run a gnuplot script named
test.script, you just have to execute the following
command from the shell:

$ gnuplot test.script
If youre inside gnuplot, run the following command:

gnuplot> load test.script


Note that it is generally considered good practice to end
a gnuplot script with the reset command in order to reset
the gnuplot environment to its default status.
The following is an example of a gnuplot script:
can help you and plot it for you! The following
command illustrates what gnuplot shows when you
try to plot an erroneous data le:

gnuplot> plot 'badDataFile'


^
Bad data on line 325
Gnuplot prints an error message on screen telling
you the line number of the rst error this does not
indicate that the rest of the le is ne, just that there
is at least one error! A correct data le gets plotted
without problems by using the following command:

gnuplot> plot 'LUDdata'

set term png size 2048,2048 crop


set output "3Dwireframe.png"
set param
set isosamples 20
set ztics .3
set xtics .2
set ytics .2
set urange [-pi:pi]
set vrange [-pi:pi]
set hidden
splot sin(u)*cos(v), cos(u)*cos(v), sin(u)
reset

Linux & Open Source Genius Guide 59

Tips & Tricks

Here you can nd a list of all


of the previous connections
you made through Secure
Shell in Chrome

There is a URL always


available pointing you to a
goo.gl URL containing an FAQ
for Secure Shell

For a given connection, you


can set the hostname, the
username and any other SSH
options you need

SSH from a web browser


Advisor

Joey Bernard As a true


renaissance man, he splits his
time between building furniture,
helping researchers with
scientic computing problems
and writing Android apps

Resources
Secure Shell:
goo.gl/RYHiK

FireSSH:
ressh.net

Shellinabox:
https://code.google.com/p/shellinabox

MindTerm:
www.cryptzone.com/products/mindterm/

DropPages:
droppages.com

Pancake:
pancake.io

60 Linux & Open Source Genius Guide

There are times when you are stuck using


a locked-down machine. As long as you
have a browser, though, you can still
connect to your remote machines
SSH is the de facto way of securely
connecting to remote machines where
you need to get work done. Normally,
this is achieved through an SSH client
application installed on your desktop. Unfortunately,
there are situations where this is just not feasible,
for any number of reasons. In this tutorial we will
look at a few different options for how to regain a
command-line connection to your remote machines.
No matter how locked down a machine may
be, you will almost always have a web browser
available. We can leverage this and get an SSH
connection established through this browser. There
are several different technologies that can be used
to give us this connection. The rst option we will
look at is a purely browser-based application that
requires nothing extra on either the client side or

the server side. Naturally, the available options are


limited, but it is one of the leanest options. The issue
is that you need to use a supported browser. The
second option is a Java-based one. A Java applet is
loaded into your browser to handle the actual SSH
connection management. Unfortunately, this is only
an option if you have Java installed and are allowed
to run Java applets in the browser. The third option
is even leaner on the client side than the rst option,
and has the added advantage of running in almost
any browser. The downside is that it requires you to
install a piece of server-side code to facilitate the
actual SSH connection management.
Hopefully, by the end of this tutorial, you will have
found an option that ts your own situation and
helps you manage your remote machines no matter
where you are.

Linux & Open Source

Genius Guide

01

Finding an SSH client plug-in

Both Chrome and Firefox have SSH clients


in their respective app stores. In this tutorial, we
will be looking at Secure Shell from the Chrome
store and FireSSH from the Firefox store.

where you can enter the host, username and


password.

04
02

Installation

In the case of both browsers, installation


should be straightforward. All you need to do is nd
the relevant app in the browser store and click on the
Install button. Most browsers also require a restart
before the SSH client is ready to use.

03

Open a new connection

For the rest of this tutorial, we will use the


Chrome version. To open a new connection, simply
click on the Secure Shell icon on the browser
homepage. This will open up a connection window

Closing connections

07

Saving connections

08

Finding a Java plug-in client

Close your connection the same way as any


SSH client, by typing in exit. When the connection
closes, Secure Shell offers you the option to
reconnect (R), choose another connection (C), or
simply nish and exit (x). If you choose x, the current
browser window will stay open but will be inactive.

Terminal options

Secure Shell in Chrome does not have a


terminal preferences window yet, so you need to
open a JavaScript console (by clicking the menu item
View>Developer>JavaScript Console) and entering
the changes you want to make. For example, you can
set the background colour with the following:
term_.prefs_.set(background-color, wheat)

05

06

All of your previous connections get stored


as a list available at the top of the connection screen.
Click on a connection to edit the SSH options before
ring off and connecting to the remote machine.

Working in SSH

You can do almost everything with Secure


Shell that you would normally do with a regular
client. You can do port forwarding by including
the relevant options when you make the original
connection. You place these types of options in the
SSH Arguments box.

There is a Java applet that you can


use called MindTerm. In this case, you need to
wrap MindTerm in a simple webpage in order to get
the browser to load it for you and host it somewhere
visible. You can also run it directly as a Java app.

Linux & Open Source Genius Guide 61

Tips & Tricks


You can do almost
everything with
Secure Shell that
you would normally
do with a regular
client

09

Installation

If you need to host MindTerm somewhere


non-local, you can place it on a hosting service if you
have one. If not, you can get a Dropbox account and
host it there as a static webpage. There are services
like droppages.com or pancake.io that will help
you here.

Open a new connection

The screenshot above is made


using the MindTerm jar le standalone. The
behaviour is the same in the browser. When it
starts up, it asks you to enter either a server
alias or a server hostname. If this is a new
machine, it will ask you whether you want to
save it as an alias.

11

Connection options

The advantage of a Java applet is that you


have more tools available to you. Clicking on the
menu item Settings>Terminal will pop up a full
preferences window where you can set the terminal

62 Linux & Open Source Genius Guide

Closing connections

14

Saving connections

15

Client/server browser-based SSH

You close your session with the exit


command, just like with a regular SSH client. Once
the connection is shut down, MindTerm resets itself
and is ready for a new connection to a new host.

type, font type and size, and colours, among many


other items.

10

13

12

Working in SSH

With MindTerm, you also have easy access to


all of the SSH connection options. Clicking on the menu
item Settings>Connection will pop up a new window
where you can set port forwarding, as well as more
esoteric items such as the type of cipher or the type of
compression to use.

Whenever you connect to a new host,


MindTerm asks you whether you want to save it in
the list of hosts under an optional alias. To get access
to these saved connections, you will need to click
on the menu item File>Connect. This will pop up a
connection window where you can select the server
from a drop-down box.

The previous two methods have an


advantage where all of the SSH connections are
essentially only through the client and the server.
This also means that the machine you are working
on also needs to allow network connections on the
ports that you need, most often port 22. But what
can you do if your desktop is locked down to only
allowing HTTP trafc? In this case, you need to move
the workhorse part of your SSH connection off to

Linux & Open Source

Genius Guide

clicking in the terminal window will bring up a set of


options that get saved in a web cookie that survives
over sessions.

another machine, and connect to it over HTTP with


your browser. The most common choice for this is
shellinabox.

remote host. There are tons of options available as


command-line parameters. The most common ones
are options like --port=x, and --verbose.

18
16

Installation

Once you download the source, you


need to install it with the usual ./configure;
./make; ./make install step that should be in
most Linux users repertoire of skills. You do this on
the remote host that you want to connect to.

Connecting to a different machine

Once you log in, you can always


just SSH to another machine. But if this is
something that you always do, you can get
shellinabox to do this for you by using the option
-s /:SSH:mynewhost.com. This means you could
have connection tunnels to multiple different
machines, each with its own port.

17

Starting the server

Once shellinabox is installed, starting it


is done by simply starting up shellinaboxd on the

20

Working with SSH

22

Closing connections

23

Saving connections

24

Where to now?

Unlike the other methods, you do not have


the option to set up more complicated options, like
SSH tunnelling. All you get on the client side is pure
HTML, with nothing else. If you need more, you will
need to use one of the other methods.

Starting the client

To start the client, you simply need


to open your browser and enter the URL
to the remote machine. It will look like
http://mymachine.com:4200, where you might have
changed the port being used. This will open up a
terminal where you can enter your username and
password.

19

21

Connection options

Because this is the leanest of the webbased SSH clients available, you simply dont have
the same level of conguration options. Right-

Like every other SSH client, you shut down


your connection with the exit command. This leaves
your browser window open with a button popping
up in the centre labelled Connect. Clicking on this
button will refresh the screen and reopen the
connection to your remote host.

Unfortunately, there is no real way to store


a set of connection strings to different machines
within shellinabox. The best option open to you is
to congure a series of daemons on different ports
tunnelling to different machines, and then you can
save the URLs to these servers as bookmarks in your
browser.

Hopefully this tutorial will have shown you


some options that are available when you get stuck
with an overly locked down machine. If these dont t
your exact situation, dont be afraid to look for some
of the other options available out in the wild.

Linux & Open Source Genius Guide 63

Tips & Tricks


Join in with any FOSS project
that has a Git repository.
GitHub and other sites open
easy code sharing to all

Which les did I change


since the last commit? The
git status command keeps
track of where youve been

Fork a repository grab


some open source code
to start working on your
own version

Share your code without


sharing your passwords,
logs, autosaves and other
irrelevancies, using .gitignore

Master version control with Git


Advisor
Richard Smedley
As a system administrator,
writer, consultant and
occasional coder, Richard has a
lot of stuff to keep track of, and
now leans heavily on Git

Resources
Git:
git-scm.com/

A GitHub account:
https://github.com

Basic command line skills

64 Linux & Open Source Genius Guide

Git is quite easy to learn, but you go much


further with an understanding beyond a
simple cheat sheet, so weve combined
this how-to with a little of why
Git has become so popular so quickly,
its easy to have missed out on why you
should integrate it into your coding (and
writing) life. Were going to get you quickly
up to speed about why Git is special, and how to get
to grips with using it.
Git is a distributed version control system,
famously written by Linus Torvalds after the
BitKeeper debacle left the Linux kernel in need of a
capable home. Now, through GitHub and a dozen
other popular repositories that make it social to
store and share code, Git is becoming the default

choice for distributed version control and source


management of new projects.
On personal projects that dont need a central
repository, Git is great for keeping track of changes
and experimenting with different approaches in
your project using branching, giving you the choice to
merge changes and/or roll them back.
To really benet from Git you need not just a few
commands on a cheat sheet, but an understanding
of how it works so well touch on a few Git concepts
as we go along, making it easier for you to go and nd
out more later.

Linux & Open Source

CREDIT: Phil Gould cc-by-sa 2.0 http://www.ickr.com/photos/phil_g

Genius Guide

01

A project, any project

Git manages projects in the form of les


so not just code, but anything from a set of recipe
cards to the text of that historical romance youre
writing under a pen name.

git init
A .git directory is created: you now have a working
tree, and youre now ready to commit changes to
your repository. Initialisation is only done once per
repository.

02

Installation

Naturally, your distro includes an easily


installable binary. Most distros separate out
packages for Git-to-Subversion functionality, a
GUI interface, revision tree visualiser, et al. Install
the git-doc package for extremely comprehensive
HTML-formatted documentation.

05

Staging post

Now that you have a working repository,


lets use:
git add
to stage the code. Passing . as an argument adds
everything in the current directory, recursively (ie
including subdirectories), to the index. Now we use
commit and at last our code is in the repository, and
tracked by Git.

07

Where was I?

08

Undone by stages

If you havent committed changes for a


while, git status is useful for showing just what
changes youve made as you tunnel through
Rails, for example, making changes across views,
controllers and helper les. Whats changed in
those les is shown by git diff (see the large
screenshot on the next page).

The two-step commitment process allows


one to change ones mind at the staging step so if
you have added a le youve edited, but dont after
all want to commit it, then
git reset HEAD filename
will remove that le from staging.

03

Who are you?

Initial cong simply consists of a few git


config --global commands, which write your
details into ~/.gitcong. Without the --global
switch, git config writes to ./.gitcong so you can
tailor individual repositories.

04

A tree, initially

Those repositories are created where your


code is; cd into a project folder and initialise your
rst Git repository with:

06

Status

If we beaver away at writing some more


code, then ask Git for a status report, it will tell us
that we have made some changes that have yet to
be committed. Its good practice to commit often
Git is your source code management tool after all,
so give it the changes to manage.

Linux & Open Source Genius Guide 65

Tips & Tricks

09

Not that one


If youve staged a totally new le, then

git rm --config filename


is the better option rm to remove a le, but
--cached to just take it from staging. Here git
status shows the change from stage to untracked
for the le we removed.

10

Commit objects

Time to improve our understanding of


whats going on behind the scenes in Git. Each
commit object contains a snapshot of the les,
references to parent commit objects, and a SHA-1
name (40-character hash of parts of the commit)
that uniquely identies the commit.

12

Branch line

git checkout gets you using another branch.


Its quite important to commit your changes before
you use checkout to switch branches otherwise
some strangeness will result, such as losing all the
changes youve made to the branch, so show some
commitment and commit

14

Releasable code

You can have as many branches as you


like, depending on your team and strategy. One
simple approach is to keep the master branch in
a releasable state and commit your incremental
changes to a development branch until youre ready
to merge your changes back for a new release.

11

HEAD master

While the latest commit object is HEAD


and its parent is HEAD^ that refers to the latest
commit in the current branch. By default we just
have a master to work on, but we can create new
branches from any commit object the above
to work on an earlier version of the projects le
format.

13

Committed to change

Git will prompt you if you have forgotten to


save before changing branch. You can use
git checkout -- nameoffile
to deliberately remove the uncommitted changes
you have made. Commits are cheap so keep
commiting in your own branch.

15
Git manages projects in the form of files
so not just code
66 Linux & Open Source Genius Guide

Track whats changed

Sometimes, simply keeping track of what


youve just done defeats the fallible human brain.
What was that small change which seems to have
broken something?

Linux & Open Source

Genius Guide

merge master into your branch to keep your tree up


to date with changes there.

git diff

21

Pushing code

22

Time-saving aliases

git push is used to push changes to the


remote repository. Using push.default in your
cong, or arguments (eg simple) to the command,
you can get push to refuse to push if the upstream
branchs name is different from the local one.

is your friend, showing you what has been changed


but not yet staged with git add. Not the les
effected (git status does that), but each line added,
removed or edited.
git diff --cached
will look at the changes that have been staged but
not yet committed.
Other useful diff commands to Git include
git diff HEAD^
which shows changes since the commit previous
to the latest commit.

18

Ignorance is bliss

Now, before you work with others on


a project, use the .gitignore le to keep your
password and local datase details out of the
public repository, along with auto-saves from your
text editor, notes to yourself, and anything else
unnecessary for the project but in the directory.
Aliases such as gc for git commit,

ga for git add --all


and gst for git status
save valuable key strokes for frequent users. You
might wish to add your own preferred aliases to your
.bashrc, or download one of the macro collections
to be found on GitHub.

16

Branch log

Keeping track of commits for your current


branch is a simple matter of running git log forthat
branch. This will show the chain of commit objects
since initialisation. The rst few digits of the SHA-1
hash shown in log is a good reference to HEAD on
the branch youre working on, if you want to check if
a colleague is using the same.

17

Conicting code

Git makes merging branches easy, but will


also tell you where there is a conict if a merge
cannot take place usually where both branches
have made changes in the same place. You can also

19

Push me pull you

20

Fetch

git pull origin master will fetch the latest


copy of the remote master and merge it into your
local copy, thereby allowing your local copy to
combine other peoples updates with your own
changes. Whether you pull frequently or just after
specic changes is up to you.

git fetch just gets the remote changes


without forcing a merge. It creates a local copy for
you to work from, so you can examine with git diff
and make changes before merge to handle any
problem conicts you can foresee.

23

Try out commands

24

Go use Git

Grab some code and start trying out the


commands. git clone can be used to make a local
copy of a complete Git repository from elsewhere,
including branches (git branch -r will show them).
Those Fork me on GitHub buttons will all be
competing for your attention.

Now, armed with some understanding of


what Gits about, go forth and spread thy code!
Online there are plenty of tutorials on advanced Git
usage and we hope to return to the topic soon in
LUD, showing you many versatile uses for Torvalds
other creation!

Linux & Open Source Genius Guide 67

Tips & Tricks

Create automated video- and


photo-related tasks easily

The Pi Camera doesn't


take up any USB slots,
instead plugging directly
into the Raspberry Pi,
which also powers it

Use the camera for timelapse photography, a robot's


eyes, or a webcam in any
number of suitable projects

Take pictures and video with


the Raspberry Pi Camera
Advisor
Rob Zwetsloot models
complex systems and is a web
developer procient in Python,
Django and PHP. He loves to
experiment with computing

Resources
Raspberry Pi
Raspbian:
www.raspberrypi.org/downloads

Pi Camera
Ashtons picam module:
https://github.com/ashtons/picam

68 Linux & Open Source Genius Guide

Follow our short tutorial on how to get your


new Pi Camera set up on your Raspberry
Pi, and how to use it
One of the most recent Raspberry Pi
accessories is the tiny Pi Camera board
a small PCB with a camera sensor
mounted to it that connects via a ribbon
to the Raspberry Pi..Because of this, its not exactly
plug-and-play, so youll need to do some extra setup
on your Raspberry Pi to get it to work.
The Pi Camera has multiple functions, such as for
time-lapse photography, using as a webcam, or even

as an optical sensor for a Pi-powered robot, like the


one were making this issue. Because it doesnt take
up any USB slots, and draws very low power, it can
be a lot more versatile than a standard webcam.
The Pi Camera itself is not a low-quality piece
of kit either with a 5MP sensor, its also able to
create up to 1080p quality video, the same as the
Raspberry Pis HDMI output. So grab your Raspbian
SD card and make the most out of your Pi Camera.

Linux & Open Source

Genius Guide

01

Attach Camera

To attach the Camera to the Raspberry Pi,


locate the slot between the Ethernet and HDMI port
and gently lift up the fastener. Insert the ribbon of
the Camera board, making sure to align the ribbons
connectors with those on the Pi.

This will show a ve-second preview of the input of


the camera and then capture the last frame of the
video.

02

Pi preparation

Before we try to enable the Raspberry Pi


Camera, we need to make sure our rmware and
software are all up to date with a quick software
upgrade. In Raspbian, we do this by opening the
terminal and using:
$ sudo apt-get update

05

Record video

To record a video, we use a similar command,


raspivid, like so:
$ raspivid -o video.h264
It will also take ve seconds of video by default.

followed by:
$ sudo apt-get upgrade

03

To take pictures with the Raspberry Pi


Camera, youll simply need to enter:

$ raspistill -o image.png

import picam
i = picam.takePhoto()
i.save(/home/pi/test.jpg)

Advanced photos

You can have it take photos of specic size


and quality with a time-based name by editing the
code to look like this:

$ sudo raspi-config

04

Picam photos

We can now use Python to construct a script


to take photos with the picam module. Very simply, all
you need to do is enter:

08

Once thats nished, run in the terminal or


command line:

Take pictures

07

And running it will take a photo called test.jpg.

Pi cong

to start the standard conguration screen. Navigate


down to Enable Camera, press Enter and then simply
key over to enable and conrm with another press of
Enter. Select Finish and then reboot.

pip install https://github.com/ashtons/picam/


zipball/master#egg=picam

06

Picam

If you want to do a little more with the Pi


Camera, theres a simple Python wrapper currently
available called picam. Youll need to install it rst,
though, and well use pip for that. Install pip with:

$ sudo apt-get install python-pip


and then enter:

import picam
import time
ii = picam.takePhotoWithDetails(640,480, 85)
filename = /tmp/picam-%s.jpg % time.
strftime(%Y%m%d-%H%M%S)
ii.save(filename)

09

Picam video and more

Picam also allows you to take video in a


similar way to the above, with the main difference
being that youll use the recordVideo command. You
can use the code to take photos or video at regular
intervals for time-lapse, or have it trigger during a
specied event.

Linux & Open Source Genius Guide 69

Tips & Tricks


Learn how to record
multiple types of slowmotion videos for yourself

Play it at normal speed for


high frame rate video for
specic video projects

Allow your Raspberry Pi to


play back the video to get
the full effect

Record slow-motion videos


with the Raspberry Pi
Use the updated Raspberry Pi rmware to record
slow-motion videos with the camera module
Resources
Latest Raspbian Image:
raspberrypi.org/downloads

Raspberry Pi camera board


Internet connection

70 Linux & Open Source Genius Guide

The Raspberry Pi camera has been an


interesting little add-on for the Raspberry
Pi, allowing people to have slightly more
control when it comes to projects that
require pictures and video. The functionality of the
camera is always being slightly enhanced and now a
new function has made its way to the camera: slowmotion video. With this you can create short clips as
slow as 33 per cent normal speed, with all the same
control as you have with the normal camera functions.
At the time of writing, our favourite PiCamera Python
module doesn't include the ability to access the slowmotion functions.

01

The rmware updater

The rst thing you need to do is make sure


you update the rmware on your Raspberry Pi. Boot
into Raspbian and open the LXTerminal. To install
the rmware updater, simply type in the following

Linux & Open Source

Genius Guide

command. For newer versions of Raspbian, it will


likely be already installed.

$ sudo apt-get install rpi-update

04

Camera setup

Just
in
case
you're
using
a
new install of Raspbian, you'll need to
make sure that the camera module is
enabled.
Make
sure
that
you've
plugged
in
the
camera
while
it's off, and then in the LXTerminal use:

$ sudo raspi-config
Go to the Enable Camera option and select to
enable it. You may need to restart in order for it to
actually take effect.

05

First tests

We can now do a quick test of the slowmotion capabilities by entering the following:

02

Update the rmware

Updating
the
rmware
is
very
straightforward; back in the terminal just type in the
following:

$ raspivid -w 640 -h 480 -fps 90 -t


5000 -o slowvideo.h264

$ sudo rpi-update

Alternate updating

Problems
have
been
known
to
occur with the rmware update and there
are two main ways you can try and x
them. First of all, try updating Raspbian
with apt-get update followed by apt-get
upgrade. If that method doesn't work,
try using the rmware updater like so:

$ sudo UPDATE_SELF=0 rpi-update

Playing it back

Although your Pi has the means


to record the video, it can't play it back.
You can either grab the video from the
SD card, or install VLC using the following
command, which should have no trouble viewing
the video le.

$ sudo apt-get install vlc

08

Other uses

09

Final warnings

If you do plan to use the slow-motion


function, obviously you can use it as intended.
However, you can also use it to create high frame
rate videos. Video editors such as Kdenlive will let
you increase the playback speed, and you can also
view them in VLC at normal speed.

And it should automatically download and install all


the necessary les. Once this is complete, reboot
your Raspberry Pi to activate the new rmware

03

07

We've told it to make a video at 640 x 480 resolution,


to lm it at 90 frames per second and to do so for
ve seconds. This is the maximum resolution for this
speed of video.

06

Doing these slow-motion videos even


at the low resolution we're using is taxing on the
processor. Try to only capture short clips while
using it and denitely do not take a lot of them in
rapid succession.

Other recording modes

As
well
as
being
able
to
record
video
at
90
frames
per second, you can also go down to
60. You cannot increase the resolution
though, so it's stuck to 640 x 480 for both
90 fps and 60 fps video for the time being.

You can also use slow-mo to create


high frame rate videos. Video editors
such as Kdenlive will let you increase
the playback speed

QBe careful not to overwork the processor

Linux & Open Source Genius Guide 71

Tips & Tricks

This isnt a
hardware hack a
spare SD card can
be used for the Tor
router, and other
SD cards can be
used for different
functions without
any problems

Using not much more than a


Raspberry Pi, you can route
one or more systems through
a Tor-enabled access point,
guaranteeing anonymity

Hook into the internet just


about anywhere theres
an internet connection a
relatives house, hotel
rooms and more

Connect
everything over a
wireless network
no need to
directly connect
to the Pi with a
cable

Browse privately with Onion Pi


Advisor
Rob Zwetsloot models
complex systems and is a web
developer proficient in Python,
Django and PHP. He loves to
experiment with computing

Resources
A Raspberry Pi
Raspbian:
www.raspberrypi.org/downloads

Compatible Wi-Fi adaptor:


www.adafruit.com/products/814

72 Linux & Open Source Genius Guide

Turn your Raspberry Pi into a highly


secure and very portable router to keep
your system safe and your browsing
anonymous, wherever you are
You can easily turn your Raspberry Pi
into a portable wireless router, requiring
very little power and giving you a wireless
network wherever theres the most basic
of internet connections. What if its not enough to
know you can search the web, though? What if you
want to be wholly secure as you do it? Then its time to
upgrade the router with Tor to protect your privacy on
the internet.
This Onion Pi, as dubbed by Adafruit, combines
Raspbian and Tor to create and secure a wireless

access point using just a Raspberry Pi. This project


is fairly straightforward: after setting up the wireless
access point, we install Tor and do some basic setup
tasks so that it routes trafc properly, and securely.
This will keep you anonymous online a handy
feature in a time lled with privacy concerns all
around the web.
When the Pi is not connected to the internet, it
should still function as a wireless router, allowing at
the very least a wireless LAN in your location.
Here, we run you through the simple process

Linux & Open Source

Genius Guide

06

Set up DHCP

Now we need to congure the DHCP server.


Edit the conguration le with:

$ sudo nano /etc/dhcp/dhcpd.conf


And start by putting a # in front of the two option
domain-name entries, then remove the # in front of
authoritative, seven lines down.

07

Server address

At the end of the conguration le, add the


following:

01

Install Raspbian

Raspbian is the Raspberry Pi distro well be


using for the Onion Pi. Download the zip le, extract
the image and then apply it to an SD card using:

04

SSH connection

Plug your USB wireless adapter into the Pi


and turn it back on. On another computer connected
to the same network, open a terminal or type into the
command line:

$ dd bs=4M if=[version number]-wheezyraspbian.img of=/dev/[SD card location]

$ ssh [user]@[IP address]

You can also use NOOBS to install Raspbian if


you wish.

Then enter the password for your Raspbian if it asks


for it.

02

Set up Raspbian

03

Pi IP

subnet
192.168.42.0
netmask
255.255.255.0 {
range 192.168.42.10 192.168.42.50;
option
broadcast-address
192.168.42.255;
option routers 192.168.42.1;
default-lease-time 600;
max-lease-time 7200;
option domain-name "local";
option
domain-name-servers
8.8.8.8,
8.8.4.4;
}
Save and exit.

Go through the initial Raspbian setup


and make sure to turn on the SSH server, and to
disable autoboot to desktop this is unnecessary
and will only use extra power. You can also tell it to
ll up the rest of the card if theres room for it.

Well be accessing your Raspberry Pi via SSH


to set it up. To do this we need to know its IP address
you can nd it by typing ifconfig into the command
line. Make a note of it and turn off your Pi.

05

Install DHCP

To make life easier for any system connecting


to the Pi access point, we need to install a DHCP
server to it. We do this with:

$ sudo apt-get install hostapd iscdhcp-server


DHCP will automatically assign IP addresses to
network-attached devices, meaning you wont need
static IPs.

08

DHCP server

Edit the server conguration les so that its


set to work in conjunction with the wireless adaptor:

$ sudo
server

nano

/etc/default/isc-dhcp-

Scroll to INTERFACES and change it to:

INTERFACES="wlan0"

Linux & Open Source Genius Guide 73

Tips & Tricks


09

14

Incoming Wi-Fi

We need to set up the Wi-Fi adaptor to be


both static and accept incoming signals. First:

Network addressing

Setting up a NAT will allow multiple clients to


connect. To do this, run:

$ sudo nano /etc/network/interfaces

$ sudo nano /etc/sysctl.conf

Put a # in front of iface wlan0 and following lines


with wpa roam, iface default and any other affecting
wlan0.

And add to the bottom of the le:

net.ipv4.ip_forward=1
Save this, and then nish by running:

$ sudo sh -c "echo 1 > /proc/sys/net/


ipv4/ip_forward"

12
10

Static IP

Now give the wireless interface a static IP


after the line allow-hotplug wlan0, enter the following:

iface wlan0 inet static


address 192.168.42.1
netmask 255.255.255.0
Save and exit, and then set wlan0s address with:

WLAN conguration
interface=wlan0
driver=rtl871xdrv
ssid=[access point name]
hw_mode=g
channel=1
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=[password]
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

$ sudo ifconfig wlan0 192.168.42.1

13

Hostapd

After saving and exiting, we need to edit


hostapd to point it to this new le. Open it with:

$ sudo nano /etc/default/hostapd


And then nd the line #DAEMON_CONF="". Remove
the #, and change it to:

15

IP tables

Run the following three commands to make


sure the internet connection is forwarded correctly:

sudo iptables -t nat -A POSTROUTING -o


eth0 -j MASQUERADE
sudo
iptables
-A
FORWARD
-i
eth0 -o wlan0 -m state --state
RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i wlan0 -o
eth0 -j ACCEPT

16

Apply conguration
So that this still works after a reboot, type:

$ sudo sh -c "iptables-save > /etc/


iptables.ipv4.nat"
Then add to the end of /etc/network/interfaces:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

11

WLAN creation

We need to create a new le that holds all the


information for our wireless network. We are going to
make it password protected so that only the people
we want to can access it. To create the le, start with:

$ sudo nano /etc/hostapd/hostapd.conf


And then enter the text from the next step.

74 Linux & Open Source Genius Guide

up iptables-restore
ipv4.nat

<

/etc/iptables.

Linux & Open Source

Genius Guide

17

Wi-Fi nal

Finally, set it up as a daemon so it runs at


boot with the following commands:

sudo
sudo
sudo
sudo

service hostapd start


service isc-dhcp-server start
update-rc.d hostapd enable
update-rc.d isc-dhcp-server enable

And the wireless access point part will be nished.

18

Install Tor

After a reboot, we now need to install Tor. Do


this simply with:

$ sudo apt-get install tor


Once its installed, youll need to edit the Tor cong
le with:

$ sudo nano /etc/tor/torrc


Follow the next step to add all the necessary
information to it.

19

Tor congure
Put this below the FAQ comment:

Log notice file /var/log/tor/notices.


log
VirtualAddrNetwork 10.192.0.0/10
AutomapHostsSuffixes .onion,.exit
AutomapHostsOnResolve 1
TransPort 9040
TransListenAddress 192.168.42.1
DNSPort 53
DNSListenAddress 192.168.42.1

If you want to keep SSH open to connect remotely,


youll need to make an exception for that with:

$ sudo iptables -t nat -A PREROUTING


-i wlan0 -p tcp --dport 22 -j REDIRECT
--to-ports 22

21

Reroute
Route all DNS trafc rst, using:

$ sudo iptables -t nat -A PREROUTING


-i wlan0 -p udp --dport 53 -j REDIRECT
--to-ports 53
And then route any TCP trafc with:

$ sudo iptables -t nat -A PREROUTING -i


wlan0 -p tcp --syn -j REDIRECT --toports 9040

$ sudo iptables -t nat -L


If youre happy with it, save it to the NAT le like
before with:

$ sudo sh -c "iptables-save > /etc/


iptables.ipv4.nat"

23

Logging

We should create a log le in case


you need to debug later. To do this, use these
three commands:

$ sudo touch /var/log/tor/notices.log


$ sudo chown debian-tor /var/log/tor/
notices.log
$ sudo chmod 644 /var/log/tor/notices.
log
You can also check it with:

$ ls -l /var/log/tor

24

Secure the router

Finally, we can activate the Tor service so


that we can start using the access point securely
with:

$ sudo service tor start

20

You can check this if you wish with:

Table ush

We now need to ush the current IP tables


so that we can get the routing to go through Tor.
First of all, do:

$ sudo iptables -F
$ sudo iptables -t nat -F

$ sudo service tor status

22

Check and save


You can check the table setup with:

To make it turn on at boot, you simple add it to rc.d


with:

$ sudo update-rc.d tor enable

Linux & Open Source Genius Guide 75

Masterclass
Become a Linux power user
78 Dual-boot Linux on any device
Give yourself more choice with your computers
as we attempt to dual-boot anything and
everything from Windows to Macs

78

88 Get started with Android Studio


Get your head around the early access preview of
the next stage in Android development

96 Get to grips with Go Program!


An introduction to one of the hottest
programming languages of the moment. Find out
if Go is right for you

88

100 Qt masterclass
Get your head around event-driven
programming, responsive development, and
reading and writing files, all with Qt

112 Program with Erlang


Write, document and debug Erlang programs
and use OTP to create a fault-tolerant server
process

100

Dual-booting
a PC is fairly
simple, but what
about other
devices that
we use or carry
around with us?
76 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

Linux & Open Source Genius Guide 77

Masterclass
Dual-booting
a PC is fairly
simple and
inifinitely
useful but
what about
computer-like
devices that
we use?

78 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

Dual-boot
Linux on any device
Give yourself more choice with your computers as we
attempt to dual-boot anything and everything
Dual booting is not quite one of the dark
computing arts its not like youre
compiling Gentoo or anything but theres
still an element of risk and reward in
the process. Dual-booting a PC is fairly simple, and
innitely useful, but what about computer-like devices
that we use or carry around with us? We decided to see
if we could bring the convenience and power of dual
booting to a range of other everyday devices and give

ourselves the greatest amount of choice possible with


our computers.
Looking around the home and the ofce, we
noticed a few extra devices we wouldnt normally
associate with dual booting: live USB keys,
Raspberry Pis, Android smartphones/tablets and
even a Mac. Adding what we already know about
dual-booting normal PCs, we set about creating
the ultimate guide to dual-booting anything.

Warning!
It should go without saying that everything were attempting
in this feature carries some risk of losing all your data. While
you should always have a backup solution in place, we know
it can sometimes slip your mind. This is your reminder
to make sure everything you need is backed up, saved
elsewhere or ghosted in case of any problems.

Linux & Open Source Genius Guide 79

Masterclass

Dual-boot your PC
Get more out of your PC by adding a second Linux distro
Linux distros are basically designed
these days to work well with each other in
multi-boot situations. Modern GRUB will
automatically detect other bootable distros
during an install and add them to the list. Even if it
doesnt, its fairly simple to add them. Going deeper,
you can mount partitions across multiple distros
for sharing les, proles and space. To get started,
though, youll have to create some space on your hard
drive. If you used the standard installation tools with
your main distro, youre likely using up all the space.

QChoose between installing alongside, or having


full control of the partitions

The rst step in the process is choosing your


second distro what exactly are you trying to
achieve? Are you developing and testing across
multiple platforms, or do you need a work and leisure
distro? Will your system have two inherently different
functions while booted into each operating system?
Once youve made your decision on what exactly you
want and need, youll then need to create a standard
installation medium either a burnt CD or a livebooting USB stick will sufce.
You can install in two different ways from here:
either setting up the new boot partitions manually, or
using the Install Alongside option that a lot of modern
distros include. Youll need to create a root partition
for your new distro we detail how to best go about
this at the bottom of the page. Its best to have a wired
internet connection even if your system has wireless,
just in case it doesnt have the right drivers out of
the box for it. Some distros like to download updates
during the install, or even download the entire distro
during installation, so having a more guaranteed way of
accessing the internet is useful.

QAn fstab file, mounting partitions on boot

During the installation, GRUB will usually be


reinstalled by the new distro so it can boot into both
the new and the old distro, with the new distro set
as a default. We cover the best way to edit or change
it back depending on your preferences on the next
page; however, this doesnt cover shared space or
auto-mounting between the two. This can be done by
adding entries to /etc/fstab, as pictured in the fstab
screenshot above.

Making space
The most efcient way to set up a hard drive and
share multiple distros
Reclaim space

New partitions

At the very least, your system will already have a root partition for
your distro, plus a swap partition. This swap can be shared by both
distros, so all you need to do is trim the storage partition for the
main distro. Most live discs and installers will include partitioning
tools. However, dedicated rescue and admin distros such as
SystemRescueCD also include them.

We recommend having at least 20GB of space for your second


distro, or at least the root partition. Make sure not to reclaim all the
available space from the hard drive, as while the ext3 and ext4 le
systems dont fragment easily, you may still catch an important
or essential le if you shrink too much. You can always try having
dedicated space for your home folder on the hard drive as well.

80 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

Understanding the boot menu


Get GRUB under your control so you
can make startup efcient and easy
GRUB is currently the standard Linux bootloader
that youll use. Its the first thing you see after system
POST and is designed to let you easily and quickly
choose what you plan to do on your computer. Its
highly editable from small things such as length until
timeout and default selection, to colour schemes and
customised boot entries.

easily change the order of menu items by renaming


the les to go in a different numeric order.

Recover
If you do decide to reorder the entries, youll then need
to run:

$ sudo update-grub

Proper order
One of the first customisations youll usually want to
make to a new boot menu is the default selection and
perhaps the timeout. To do this, enter the terminal and
open the grub.cfg file with:

$ sudo nano /boot/grub/grub.cfg


Find the line set default=0 and change the
number to the entry in the bootloader that needs to
be default. Timeout can be found in the folder /etc/
grub.d in the 00_header le. The les here determine
the order in which entries are added to the bootloader,
so as well as changing the default selection, we can

This command can also be used as part of the


recovery of GRUB from the original distro as well
if thats your primary distro on this system. Then
instead of reordering the items or changing the
defaults, it will be better to start by using it to manage
GRUB. Boot into it, open the terminal and reinstall
GRUB with:

$ sudo grub-install /dev/sdaX


where X is the location of your hard drive. This is
more than likely 1 on a single hard drive system.
After, use update-grub to update the boot menu.

Build your own boot scripts


Create custom boot entries in GRUB 2 and even
GRUB to make the bootloader your own
GRUB2
For booting Linux, we point it towards the root of the
distro to boot and the information it needs:

#!/bin/sh -e
cat << EOF
menuentry "Linux distro" {
set root=(hdX,Y)
linux /boot/vmlinuz
initrd /boot/initrd.img
}
EOF

QWhile the config file warns against it, theres no


harm in changing the default selection

chainloader (hdX,Y)+1
}
EOF

GRUB
GRUB 1, while simpler looking, needs more
information to boot into Linux properly:

title Linux distro


root (hdX,Y)
kernel [/path/to/kernel] root=/dev/sdaX ro
initrd [/path/to/initrd]

QGRUBs boot menu also includes recovery and


memory testing options

Which GRUB?
There are two versions of GRUB: the older
GRUB 1 (or GRUB Legacy) and the newer
GRUB 2. Some prefer GRUB Legacy as
parts of it are easier to use; however,
GRUB 2 can be customised a lot more with
themes and images and smarter boot
ordering. Generally though, if your main
or primary distro comes with one version
of GRUB, you should continue using that
rather than installing a different version
over it.

hdX,Y and /dev/sdaX


Windows is a little bit different after telling it where
Windows lives, we then chainload into the Windows
bootloader:

The Windows boot code is again fairly simple, using


the chainloader command to use the Windows
bootloader:

#!/bin/sh -e
cat << EOF
menuentry "Windows" {
set root=(hdX,Y)

title Windows
rootnoverify (hd0,0)
makeactive
chainloader +1

How exactly do we nd out the numbers


to replace these with? In the terminal, rst
use fdisk -l to list the drives and partitions
currently in your system as sda1, sdb3 etc.
For hdX,Y, start counting the drives from 0,
so sda1 would be hd0,1 and so on.

Linux & Open Source Genius Guide 81

Masterclass

Add Linux to Windows


Turn a new laptop or PC into a dual-booting machine with
dual operating systems for maximum coverage

QUbuntu and other distros will recognise and


know how to install alongside Windows

QChainload straight into Windows with no


extra work thanks to GRUB

Fragmentation
Windows Vista and later has
automatic defragmentation tools
built in to cut down on the issues
previous Windows systems used
to encounter. Its not perfect,
though, and NTFS is still prone to
fragmenting more than Linux journal
le systems as les are updated
or changed over time. We suggest
invoking the Defrag tool from within
Windows by going to Start>All
Programs>Accessories>System
Tools>Disk Defragmenter in Windows
7 and later.

QThere are also some open source


tools you can use to defrag Windows

82 Linux & Open Source Genius Guide

As always, we start with partitioning


the hard drive. However, unlike with dual
booting with another Linux distro, there
wont be a swap partition available for the
new Linux distro to use. The swap partition needs to
be the same size as your system memory, so take that
into account when creating space for the Linux distro.
Also consider that Linux can easily see and mount
NTFS partitions, while Windows doesnt like extended
le systems as much, so keeping the majority of
space to be shared on the NTFS partition may be wise.
The swap partition can be placed at the end of the
hard drive; this way its easy to work around if you have
to recreate or resize partitions at a later date. This
partitioning can be done on the Windows side, but
its preferable to use a Linux tool such as GParted to
perform the task. This can be found on some rescue
distros, and a few live-booting distros will have that or
their own partitioning tool. You can either just make the
space before installation, or set up the partitions while
youre at it. One benefit of only making space is that you

can then use an install-alongside option if your distro


has one and it will automatically set up a good swap
partition at the same time.
GRUB will automatically add a chainloading
option to boot Windows during startup using its own
bootloader. Windows will be none the wiser and boot
up as normal, albeit now with a little less space. You
can use the fstab tips from the previous pages in your
Linux distro to automatically mount the Windows
partition and have full access to your les and folders;
however, youll need to look for specic ext drivers for
Windows to access your Linux les.

Secure boot
Secure Boot and how it affects
UEFI and dual booting is still an
issue not all distros work with
it. However, Ubuntu, Fedora and
their derivatives all work just ne on
systems with Secure Boot without much ore than
disabling it in the UEFI menu at startup.

Linux & Open Source

Genius Guide

Add Windows to Linux


Add or just restore Windows to a system
that didnt always need it
In some ways, its currently a little
easier to add Windows to a system
already running Linux than it is the other
way around. Linux systems dont invoke
any Secure Boot privileges, giving you free rein over
the hardware, which in this case means installing
Windows. The main issue arising from this is that
Windows will overwrite the Master Boot Record,
meaning you wont be able to boot into Linux right
after the installation. Well cover a quick way to
recover your GRUB bootloader after installation,
but for a more in-depth guide, look towards the end
of this tutorial.

01

Make space

We advise allowing Windows to have as


much space as possible, as it can ll up a lot quicker
than Linux. Microsoft recommends a minimum of
20GB; however, thats far too low if you plan to use
Windows fairly regularly. 100GB, if you can spare it,
should be more than enough.

03

Choose wisely

The Linux partitions will be labelled as


unknown by the Windows installer; however, it will
recognise the free space on the hard drive. Select the
free space and click Next to continue with the lengthy
Windows install process.

05
02

Install Windows

Insert the Windows installation disc and


reboot your system. Select Boot from disc from
the boot menu and then go through the basic
language setup and licence agreement. Select
Advanced when it asks you how you want to install.

04

Live booting

Well need to recover GRUB now so that


we can boot into both Linux and Windows. Use an
Ubuntu live disc or your original distros live disc if it
has one and boot it up. Check to see if you need to
install GRUB to the live environment, specically the
grub-pc package.

Restoring GRUB

Once youve checked if GRUB is installed, we


can now begin to restore it. First, mount the root Linux
partition and then type in the terminal:

$ grub-install root-directory=/[mount point]


/dev/sdX
where X is the boot hard drive, more than likely sda
in a one hard drive system. Reboot and log into your
Linux distro and nish off with:

$ update-grub

Linux & Open Source Genius Guide 83

Masterclass

Dual-booting USB storage


Create a portable, dual-booting stick or hard drive for
recovery, sysadmin or anything you want
Carrying around a live USB or custom
live-booting USB storage device can be
advantageous if you regularly need to
do system maintenance or dont always
know what computer youll be working at next.
Just one distro may sufce for some but, as weve
mentioned previously, there are many reasons you
might be in need of two at a time. Having two fully
functional distros working on one USB drive or
large USB stick can offer a lot more than an Ubuntu
live USB, and its actually not too different from
setting up a dual-booting system from scratch.

01

Create swap

Make sure youve saved any important data


on the external drive before we go any further. The
rst thing youll need to do is format it and then create
the swap partition make it 4GB in size so that it can
make the most out of more powerful systems. Put it
towards the end of the hard drive like before.

02

Room to share

You can create some shared space for


both distros to use, either independently or as a
shared home directory if youre feeling a bit braver.
Again were saving about 15 to 20GB for the root
directories, so keep that in mind while creating it.
Use ext3 or ext4 as the lesystem, or NTFS if you
want it to be cross-platform.

84 Linux & Open Source Genius Guide

03

First roots

Now create the 15 to 20GB space youll


want for the rst distro in its installer and set it to be
root by selecting / in the menu. It will automatically
mount the swap the partition we created when
booting into itself and our other distro. Carry on with
the installation instructions as before.

05

GRUB reinstall

Boot into a recovery distro (or Ubuntus)


live environment and make a note of what
fdisk -l labels the main boot partition on the
external drive as. Mount it with:

$ sudo mount /dev/sdXY /mnt


And then reinstall GRUB 2 from the terminal with:

$ sudo grub-install --root-directory=


/mnt/ /dev/sda

04

Second distro

Put in the installation medium for the


second distro with the USB storage still plugged
in and boot into its live environment or installer.
Choose the external drive again when selecting
the location, create a custom partition in the space
remaining as ext3 or ext4 and set the mount point to
/. Install as normal.

06

GRUB update

After rebooting, youll be able to get back


into the main distro on the external drive. Once there,
mount the other distros root partition however you
wish and run:

$ sudo update-grub
It will automatically detect the other install and
update the boot menu next time you boot from the
hard drive.

Linux & Open Source

Genius Guide

Dual-boot Android ROMs


If ditching stock Android is not enough for you, why not
try using multiple ROMs on your device?
Android being an open source operating
system has resulted in a number of
custom ROMS that extend or update
the capabilities of your phone or tablet.
Using only one at a time, though, means that if you
want to change, you need to spend an afternoon
ashing and restoring another custom or stock
ROM, leaving you without the device for that time.
For users of Google Nexus products, the
MultiROM Manager is a fairly straightforward app
that takes the pain out of the process, requiring you
to only have your phone rooted for it to work. This
can be downloaded straight from the Google Play
Store and will then ask to be granted root access.
The next step is installing MultiROM Manager to
the bootloader, updating the kernel modules and

anything else it suggests installing. It will likely ask


you to reboot to recovery to nish installation, and
then show up a boot selection on next boot.

Adding ROMs
Find the .zip le of your preferred alternate ROM
and download it to the storage of your Nexus
device. Do not extract it. Reboot the device
to recovery with adb reboot bootloader (on a
connected PC), or turn it on by holding down
volume-up and power and select recovery from
the fastboot menu. Youll be sent to the TWRP
recovery, which now has a new option under
Advanced called MultiROM. Select this, then
Add ROM and hit Next to then locate the zip le
we just put on the device. Swipe to install.

Compatible
devices
Nexus 4

Q

Last years Android


flagship phone is able
to run the Ubuntu Touch
image as well

QNexus 7
(2012)

The 7-inch tablet


that revolutionised
the market also
runs the Ubuntu
Touch image

Nexus 7 (2013)

Q
QMultiROM Manager handles all the flashing and
advanced installing for you

QChoose your distro from the simple


bootloader

Googles latest tablet offering is


much more powerful, but cannot
run Ubuntu Touch

Linux & Open Source Genius Guide 85

Masterclass

Dual-boot a Mac
Apple products can still get a bit of open
source love you just need the right tools
While the quality of Apple software
is debatable on a number of levels,
the hardware inside the rms Mac
computers is often extremely good.
Since the switch to Intel-based architecture
a few years ago, more Linux distros have
technically been able to work on Macs, albeit
with some restricted drivers installed to make
the most of them. Its not as straightforward as
the other setups weve tried in this guide, though,
and involves installing a custom bootloader to
allow us to chainload into live discs and GRUB.
Once this is done, however, you can enjoy a lot
more freedom from your Mac and make full use
of its powerful, high-quality hardware.

01

02

Room for more

Open Disk Utility from the OS X Utilities;


click on the main hard drive and then the Partition
tab. You should see the main startup partition,
which, unless youve changed it, will take up all the
hard drive. Resize it, as weve advised before, and
click Apply.

04

Live boot

05

Restricted drivers

06

Boot order

Boot into your Linux distro from the disc


and follow the normal installation process weve
detailed for other dual boot scenarios either
use install alongside or create your own custom
partition structure. It may take a little longer than
usual, so youll need to be a little patient. When it
asks to reboot, stay at the rEFInd screen.

The wireless drivers wont be included


with most distros for a lot of Macs. Boot into your
distro and update all the packages over a wired
connection, then go to Settings>Driver Manager and
install the suggested Broadcom wireless driver.

Some rEFInement

Head on over to the rEFInd website


(www.rodsbooks.com/rend) and download the
latest binary zip of the tool. Open Terminal in OS
X and cd to the download directory. Its much like
a Linux terminal, so ls will also list the contents.
Install it with ./install.sh

03

Boot from disc

Reboot your Mac and the rEFInd boot


manager should show up at boot. You may need
to reboot a couple of times for it to work. It will
automatically boot into OS X unless you press a
directional key. If the option to boot from CD isnt
there, press Esc to refresh the boot selections until
it appears.

Since the switch


to Intel-based
architecture, more
Linux distros have
been able to work
on Macs
86 Linux & Open Source Genius Guide

You can change the default selection in


rEFInd by editing its conf le. Boot into OS X and
locate the rend folder from the EFI directory.
Open rend.conf in a text editor and locate the line
default_selection. Uncomment it and change it to
2 for your Linux installation, or keep it at 1 for OS X.

Linux & Open Source

Genius Guide

Dual-booting a Raspberry Pi
Short of SD cards for Pi projects?
Put two Pi images on one card
The Raspberry Pi, with access to only
a limited SD card, may not be the most
obvious dual-bootable device. However,
due to the recent release of the New Out
Of Box Software (NOOBS), multiple Pi distros can be
installed onto one card. The catch is that you need
to have used NOOBS in the rst place to get the
dual-booting setup to work.
If you havent already done this, installing NOOBS
onto an SD card is easier than copying an image
directly to it. Using the same kind of partitioning
tools weve used to shrink space and create new
structures, wipe an SD card you wish to use on
the Pi and create a FAT32 partition that lls up the
card. Preferably, as were dual-booting, you should
use as large a card as possible, with at least 2GB
for each distro. Once the formatting is complete,
download NOOBS from the Raspberry Pi website
and extract the les onto the SD card, then plug it
into the Raspberry Pi and turn it on. You can then
select which distros you want to install and it will
automatically install them all to the disc, and create
a bootloader to use at system startup.

Raspberry distros
Home & Projects
Raspbian is the preferred distro of the
Raspberry Pi, with tons of learning
software and easier access to some
of the more interesting Pi projects and
hardware. Its easy to set up and can be
used as a standard desktop OS.
HTPC
Its a tough call between OpenELEC and
Raspbmc both run on the excellent
XBMC. However, OpenELEC has been
around for longer and has some very
special speed optimisations that put it
just ahead of Raspbmc.

QIf you've previously used NOOBs to install a distro, you can get back to the OS selector by holding down
shift as you turn it on

Customisability
Arch Linux on Raspberry Pi allows you to
build it from the ground up, only coming
with a command-line interface and a
basic Linux setup to start with. This allows
you to totally customise it and only use the
packages and software you wish.

Linux & Open Source Genius Guide 87

Tips & Tricks

88 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

Get started with

Android
Studio
Get your head around the early access
preview of the next stage in Android
development

Announced mid-last year at Google I/O,


Android Studio will be the replacement
for the current Android development
solution in Eclipse. This Eclipse setup
is part of the Android Developer Tools, and takes
advantage of the IDEs Java-based development
environment and plug-in support to currently
create the development portion of Android apps.
With the rest of the SDK, virtual devices can be
created and used to test code before deployment
onto real devices.
The Android Studio aims to be an all-in-one
solution for development and testing. Coming with
the entire SDK, and not even requiring traditional

compiling or installation on Linux, Android Studio is


an incredible easy and quick way to get straight into
coding your apps whether youre an experienced
Android/Java dev working directly in code, or a
novice looking to use the graphical, drag-and-drop
approach to app creation.
Android Studio is still currently in beta as a free
early preview well cover some of the current
quirks with the software. However, its denitely a
great way to get used to using this software before
the nal switchover occurs. Well also cover how
you can export from Eclipse if youve already been
using that, and how to then distribute your app once
its nally complete.

Coming with the entire SDK, and not


even requiring traditional compiling or
installation on Linux, Android Studio is
an incredible easy and quick way to get
straight into coding your apps
Linux & Open Source Genius Guide 89

Masterclass
Introducing Android Studio
The new Android Studio is based on IntelliJ IDEA, a much smaller and streamlined
IDE than Eclipse. Android Studio utilises its features to create an all-in-one Android
development environment with a smart visual view that is great for people just
getting into app development, and the standard text editor for those who know
their way around Java and the Android API.

Installation is very simple as well the les contain both the IDE and the
SDK youll need to create your apps, so you wont need to install the SDK via ADT
separately like you will have done with Eclipse if youve developed for Android in
the past. The Studio allows you to edit how the SDKs are utilised, how imports and
exports work, and even has a plug-in manager.

Recent Projects
The recent
projects view
allows you to go
straight into any
project you wish
to work on without
having to look in
the Open Project
prompt

New Project
Start a new project
by going through
the handy Android
new project wizard,
selecting API levels
and setting the icon

Docs and How-Tos


Some tips and
how-tos for
IntelliJ, the base
of Android Studio,
are available
straight from the
welcome screen

Check for updates


Android Studio
has its own update
manager, allowing
you to keep it up to
date. It will notify
you when a newer
version is available

Installation

01

Download

Head over to the Android Developers


website and grab yourself a copy of the early
access preview to Android Studio from here:
bit.ly/1bWrFXz
Youll need to then extract the contents
put it in a folder you have easy access to, as
you need to run the Studio from the folder.

90 Linux & Open Source Genius Guide

02

Runtime
Use cd in the terminal to navigate to

the folder you just created, specically the


android-studio/bin/ directory. To launch Android
Studio, enter into the terminal:

$ ./studio.sh
If it asks about your Java settings, for now
press Enter, as we can x that later if it becomes
a problem.

03

First run

If this is your rst time using Android


Studio, you can safely ignore the prompt to
import any settings. Otherwise, nd the le you
created from before, or your old installation
folder, to get previous settings. We can import
from Eclipse later on.

Linux & Open Source

Genius Guide

Key features

Helpful shortcuts

Development views

Android Studio has handy, configurable


keyboard shortcuts. Heres a list of the
most common to get you started

As the Java for the Android


apps is split into visual and
function elements, you can
use a visual WYSIWIG editor
for the XML code, allowing
you to place buttons and
text and other UI elements
wherever you want on the
screen. The text part is a full
IDE, with code navigation,
debugging, syntax
highlighting and a smart
code analyser that warns you
of any obvious issues with
what youve written.

Command look-up
(autocomplete command name)

Ctrl

Shift

Project quick x

Alt

Enter

Reformat code

Ctrl

Alt

Show docs for selected API

Virtual devices
Test your apps on a virtual
device that you dene, either
by using settings for the main
Nexus devices, or by using
generic settings for differentsized Android phones or
tablets. You can edit the
system settings to limit or
increase RAM, and even allow
it to use the system GPU if you
need it to. Its not perfect, but
its a great way to quickly test
functionality.

Ctrl

Show parameters for selected method

Ctrl

Generate method

Alt

Insert

Navigate open tabs

+
+

Alt
Alt

Jump to source

F4
Delete line

Ctrl

Search by symbol name

Ctrl

Alt

Shift

Build

Debugging
Eclipse migration
For existing Android developers, its easy to migrate
from Eclipse and the ADT to Android Studio. The
tools are already in place to export the necessary
les from Eclipse, and Android Studio includes a
handy feature that allows you to then import them,
and any other Studio le. This is helpful now and
will be essential when it replaces Eclipse.

A specic debugger can analyse the code as


it runs and give you a full rundown of whats
happening when, allowing you to go through the
logs and gure out any possible issues, or work
out where problems are already occurring. You can
pause the program at any point to isolate issues,
instead of having to search through code as it
continues to run.

Ctrl

F9

Build and run

Shift

F10

Toggle project visibility

Alt

Linux & Open Source Genius Guide 91

Masterclass

FAQ
Since Android Studio is still under some
development, there are some bugs that still
need to be ironed out. Here are a couple of
known issues with Android Studio that you
can work around.

Create a new project


Android Studio offers a quick way to start your app:
name, package name, theme, advanced icon options
and the ability to select between different Android
APIs. The latter allows you to put together an app with

specic versions of Android in mind, making newer


API objects available for some.
Well look at a basic setup rst so we can make our
Hello World app later.

01

Create New Project

02

Set image

03

Activity selector

04

Activity name

05

Project structure

06

Views

1) Error: Gradle project refresh failed


Android Studio has a newer version of Gradle
that has some backward compatibility issues.
These can be xed by rst clicking the link to
search build.gradle from the error dialog. From
there, double-click the line under the build.
gradle usage, opening the project build.gradle
le. Edit the classpath to change the gradle
version to 0.5.+. Finally, save the le and
rebuild your project.
2) Error: Failed to import Gradle project
This can be a problem after upgrading Android
Studio, as the project les may point towards
an SDK element that no longer exists. To x
this, youll need to quickly install the Android
Support Repository. First, open Android SDK
Manager, nd and expand Extras, and install
Android Support Repository.

From the Welcome screen, click on Create


New Project to open up the dialog window. Name
the project HelloWorld well be creating the code
for that shortly. Leave all the other default settings
they relate to what versions of Android will be
able to use the app.

Here you can set your image, giving you a


nice preview of how it will look on different types of
devices. You can set padding, background colour
and more to your app icon. For now though, click
the Next icon, as we dont need an icon for this
sample project.

Migration
Its quite easy to export existing projects from
Eclipse and then import them into Android
Studio. First of all, make sure your ADT plug-in
is up to date, then select File and Export. Open
Android from there and select Generate Gradle
les, then your projects to export before clicking
Finish. In the basic Android Studio window, click
on Import project, nd the build.gradle le and
then press OK at the pop-up. You should then be
able to start development on the project within
Android Studio.

For now, well just use the blank activity.


When making other apps, this activity selector can
help you choose how the app with behave such
as being full-screen or having a multi-column ow
to view several items.

Android Studio will create a lot of les


for what will be a simple app luckily we just
need to pay attention to MainActivity.java and
Main_Activity.xml for the moment. The interface
is also initially set up to allow for WYSIWYG visual
development.

92 Linux & Open Source Genius Guide

Here you can name the activity and


customise the way it works, with tabs, drop-down
menus and other actions. For now, well leave this
as is. Hit Finish and the project will be built. It may
take a little while to do this.

The default view is the visual development


interface it includes elements down the side that
allow you to drag and drop functionality. You can
also click the Text tab at the bottom to get straight
into the Java code, with a live preview updating as
you type.

Linux & Open Source

Genius Guide

A view to develop
The two views for Android Studio serve different purposes and people. The
default visual view allows you to drag and drop functions onto the interface, with
guidelines to make sure you keep them aligned with other UI elements. This then
adds the elements to the code, so you can edit them further in there.

Layouts & widgets


Drag and drop
different layouts and
functions to create
your app visually a
great way for novices
to get started

Live preview
See a live
preview of the
way your app
will look, even
while using the
text mode

The text view works like any good IDE, with a hierarchical view of code, code
navigation tools, autocomplete and, of course, proper syntax highlighting.
Whichever view you use is up to you however, you will ultimately have more
control with the text view.

Toolbar
Quick access to
preview and debugging
tools, along with the
ability to change the
theme, activity type
and the device preview

Component tree
Even in the visual
view, you can break
down the code
in the individual
components to see
how they work

Useful resources
There are plenty of online resources that you can
use to help you with your development whether
you need to gure out a more elegant method to
create a specic function, or need to wrestle with
some bugs. Heres a list of some of the places you
should visit:
Android Developer Site
developer.android.com
As well as the place to get Android Studio, the
Developer site has regular posts on interesting

projects, updates to the SDK, and some


troubleshooting tips.
Stack Overow stackoverow.com
One of the best places on the internet for both
enthusiast and professional developers to get
advice on an issue either by searching for a
related question or submitting your own.
IntelliJ IDEA Support
intellij-support.jetbrains.com/home

Support for IntelliJ IDEA, the base for Android


Studio. Some of the support questions may be
more suited to the IntelliJ side of things.
XDA Developers
www.xda-developers.com/
A community built around smartphone
development, the XDA forum is a good way to nd
out about Android-specic tips and tools, and even
to promote your application.

Linux & Open Source Genius Guide 93

Masterclass
Make your rst application
Creating your rst application in Android Studio is really very easy,
and well start by showing you the simplest one of all the classic
Hello World app. All basic apps, when created, have Hello World

01

Visually

To do it visually, simply select Plain TextView,


and place it on the display. Double-click it and type
in Hello World!. All static text can be placed this way
onto the interface, and it will create the necessary
code in the text view as well.

02

Relative text

The relative layout should be included in the


code by default. Create a text parameter by opening
up triangle brackets with:

<TextView

already displayed within them however what were going to do in


this tutorial is learn how to simply make Hello World in Android, and
then well go through what other simple tasks we can do using it.

android:layout_height=wrap_content
android:text=Hello World!
android:id=@+id/textView/>
The layout variables can also be set as xed pixel
width/height.

04

Quit button

05

Imported

Hello code

To add Hello World to the Text parameter,


we put in:

android:layout_width=wrap_content

06

Coded actions

Add the following code below the line


setContentView(R.layout.activity_main);:

The le were editing now is just the display.


To do something, we need to start editing the .java le.
While were still in the .xml, place a button and give it
the caption Quit. Open up MainActivity.java from the
Projects column before we continue.

which we will use to display Hello World.

03

import android.widget.Button;
This allows us to use some extra Android functions to
press and use the Quit button.

Now add these imports to the top of the

code:

import android.view.View;
import android.view.View.OnClickListener;

Button button = (Button)


findViewById(R.id.button)
button.setOnClickListener(new
OnClickListener() {
@Override
public void onClick(View v) {
finish();
System.exit(0);
}
});

Code listing
Package name
The name of your package
according to the Android
system
import functions
Import the various functions
we need to get the code to
work on an Android device
MainActivity class
The MainActivity class includes
all the functions in our main
activity that we created
Main layout
This sets what the main
layout, and displays it
Quit button
The section containing
the code to make sure our
button (with id button)
properly quits the app
with System.exit
Settings menu
This creates the Android settings
menu that appears at the top of
the app

94 Linux & Open Source Genius Guide

package com.linuxuser.helloworld;
import android.os.Bundle;
import android.app.Activity;
import android.view.Menu;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
public class MainActivity extends Activity {
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
Button button = (Button) findViewById(R.id.button);
button.setOnClickListener(new OnClickListener() {
@Override
public void onClick(View v) {
finish();
System.exit(0);
}
});
}
@Override
public boolean onCreateOptionsMenu(Menu menu) {
// Inflate the menu; this adds items to the action bar
if it is present.
getMenuInflater().inflate(R.menu.main, menu);
return true;
}
}

Linux & Open Source

Genius Guide

Working with virtual devices


Create a virtual Android device to test and debug your Android apps, choosing from a selection of generic phones and tablets

Home screen
Emulate a list of
Android devices,
from Nexus
phones and
tablets to generic
phones dened by
screen resolution

Physical buttons
Theres a range of
physical buttons you
can use that work on
certain devices, with
power and volume
working on all

Hardware specs
Edit specic
hardware variables
to better emulate
certain aspects of
a device, or give it
more power via your
host system

Android buttons
Software buttons
are included on
the virtual device,
as they are part of
modern Android

01

Virtual device manager

From the Project View, nd the


Android Virtual Device (AVD) manager icon
along the top toolbar and click it. From here
you can create and edit a range of devices
for testing your apps on. Click on New in the
right-hand panel so that we can get started.

02

New device

In the pop-up dialog, name the virtual


device, select the device you wish to emulate and
then the target API level. Some devices will have a
minimum Android version target. Your can edit this
later, but for our app we wont need to change any
other settings.

03

Virtual reality

Click OK and Android Studio will conrm


the options youre using. Press Start while the
device is selected and choose whether you want to
get a pixel-accurate screen, or a scaled version. It
will then boot up a virtual phone or tablet you can
test on.

Debugging
Android Studio includes tools to debug
the apps you create. To use them, rst
make sure a device is running to emulate
the app. Then, click the Debug button
in the shape of a green bug and the le
will rebuild and launch on the device. This
time, though, it will bring up a full logging
window, a console and a debugger with
full tools to gure out where problems
may be occurring in your apps.

QUse
Android
Studios
debugging
tools to
analyse your
code

Linux & Open Source Genius Guide 95

Masterclass

Get to grips with Go program!


An introduction to one of the hottest programming languages
of the moment. Find out if Go is right for you
Advisor

Sean M Tracey is a
creative technologist at a
leading digital agency. He
spends a lot of his time living
inside of Node.js, Python
and Arduino

In 2007 Robert Griesemer, Rob Pike and


Ken Thompson started writing the
language were going to be talking about
here. Ken and the two Robs felt that, over
the years, languages such as C++ had become too
cumbersome to quickly develop with and, instead of
following the path of developers before that tried to x
these problems, they decided to start again from
scratch. Thats how we have Go. Gos been out in the
real world for a little while now. It was designed to be
quick to learn, quick to write and quick to run and it is
certainly that. At a glance, Go (or Golang) will look very
familiar to those of us who spend our days writing
programs in a C-like language, but Go has a few
gotchas and nuances that were going to look at in this
article. Were going to quickly walk through setting up
Go on a Linux system (weve used Ubuntu 12.04 but itll
run on OS X and Windows), the Go I/O standard library,
variable declaration, loops, structs, slices and various
other little bits that developers will nd useful in their
coding adventures.

Assigning variables
in Go is very
straightforward
and exible

Setting constants
is much stricter, for
obvious reasons

Function denitions
are very user-friendly
in design and
implementation

Installing Go
If youre quite lucky, you might be able to install Go
with one swift command of the terminal:

sudo apt-get install golang


If all has gone well, you will now have Go on your
system. Hurrah! But, for the sake of variety, lets
assume thats not happened. Lets go and install from
source instead so we can all share in the Go goodness.
Head on over to code.google.com/p/go/downloads/list
and grab go1.1.2.src.tar.gz (at the time of writing, this
was the latest version of Go). Extract the tar.gz to
wherever you like and head into the src folder that
youll nd in there.
Assuming you have GCC or another C compiler
installed on the system, were ready to get this thing

96 Linux & Open Source Genius Guide

Notice how Go allows you


to assign two variables by
returning two values

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

//Example 1:
a = 1
b = 2

Example variables &


functions code listing

//Example 2:
a, b := 1, 2
//or
var a, b int = 1, 2
//Example 3:
var a, b, c = 1, true, "Hello"
//Constants can only be set like this
const x = 12
//##### Functions
//Example 4:
func example(x int, y int){
fmt.Println(x + y)
}
//Shorthand
func example(x, y int){
fmt.Println(x + y)
}
//With return
func timesTwo(x, y int)(int, int){
return x * 2, y * 2
}
func main(){
h, i := timesTwo(2, 6)
fmt.Prtinln(h, i) // h == 4, i == 12
}
//Alternative way with namesd variable return
func timesTwo(a, b int) (x, y int) {
x = 1 * 2
y = b * 2
return
}

Linux & Open Source

Genius Guide

going. As it often is, Google has been very kind and


has included a handy Bash script to install and test
Go for us. Fire it off with:

sudo ./all.bash
And then go and make yourself a cup of tea. Like most
source builds, this can take a little while. Just before
the Bash script exits, you should see something along
the lines of:

Installed Go for [YOUR ARCHITECTURE] in


/home/[USERNAME]/go.
Installed commands in /home/[USERNAME]/
go/bin.
*** You need to add /home/[USERNAME]/
go/bin to your $PATH. ***
If you can see that, everything has gone well. Now,
just like the script says, we need to add Go to our
system PATH. Everybody has their own preference
when it comes to their system but you can put it
into your .prole le in your home directory so its
accessible across the system like so:

Wheres it going?
New languages pop up all the time. So what makes
Go different?
Go is a really exciting language: its quick to
learn, fun to write and really fast at running.
Its true, new languages appear all the time.
Anybody and their mother can learn how to
write a compiler, but Go has some of the best
software engineers in the world tinkering
away with it. Its got a long way to go before it
can stand to challenge the likes of Python or
Java and its going to take time for people to
acknowledge it as a genuine competitor, but

remember how seriously people took Node.js


when it was rst released? JavaScript On
the server? Wha? Now look where it is.
Go already has a massive package base to
save developers time, so theres nothing getting
in the way between a developer and a pet
project. As for current real-world applications,
Google recently rewrote its content delivery
system with Go. Now thats what we call
commitment.

library for Go. It can be written so that one statement


imports many libraries or just one at a time like so.

want to write a function that accepts variables, we


can write it like so:

Single statement:

func example(x int, y int){


fmt.Println(x + y)
}

import("fmt", "math", "net")


Multiple statement:

PATH="$HOME/gosrc/go/bin"
Now, enter go in your terminal and you should see the
help dialog listing all the commands we can invoke
when executing a Go script.

import "fmt"
import "math"
import "path"

Notice how we declare the types after we name the


variable were expecting. If the types are the same we
could write instead:

When executed, Go will look to run the main() function


rst.

func example(x, y int){


fmt.Println(x + y)
}

Hello world
Lets write a quick Hello world. Create a le called
lets.go. If you execute:

go run lets.go
You should see Hello, world pop up in your terminal.
Lets step through what weve written here. package
main is basically Gos way of creating a namespace
for the code contained in the le. If we were to write,
say, a module for use in other programs then we could
use most any name we like. The general convention
is that the package name is the same as the import
path. import(fmt) simply imports the stand I/O

1
2
3
4
5
6
7

package main

helloWorld.go
code listing

Variables
Variable declaration is a little different in Go than
it is in other languages. There are multiple ways of
achieving the same effect, If you look at Example
1 on the preceding page, it can also be written like
Example 2.
In each of these instances, a == 1 and b == 2.
Youll notice that when we used the var keyword, we
declared the variable type just before we assigned a
value. When we declare the variable type at the end
of this declaration, all of the variables will be that
type. If we are assigning values in this way and the
types vary then Go will assign them based on the
value passed.
In Example 3, a will become an integer, b will be a
Boolean and c will be a string. Go has constants too.
They can be set like so:

import("fmt")
const x = 12

func main(){
fmt.Println("Hello, world")
}

But you cant declare the type like you can with a
variable. Thats inferred from the value.

Functions
Q
 Now that we have a working environment, we

can run a nice little Hello world

Functions in Go can do some cool things. In our Hello


world we used the most basic implementation. If we

If we wanted to return values we would write


something like this instead:

func timesTwo(x, y int)(int, int){


return x * 2, y * 2
}
func main(){
h, i := timesTwo(2, 6)
fmt.Prtinln(h, i) // h == 4, i ==
12
}

Google has
included a handy
Bash script to
install and test
Go for us
Linux & Open Source Genius Guide 97

Masterclass

Notice how we assigned two variables by returning


two values. If we were so inclined, we could name
these variables and return them like so:

func timesTwo(a, b int) (x, y int) {


x = a * 2
y = b * 2
return
}
Functions in Go can also return closures. But we
must be careful to realise that we arent returning
a function were returning the result of a function
that was executed when the function that was called
was returned. Its a little confusing:

func aFunction(x, y int) int{


randomNumber := 5
return func(x, y int) int {
return x + y + randomNumber
}
}
Notice that in aFunction() were specifying that
an integer is being returned, not a function. If this
was JavaScript, youd be returning the function as
an object that could be executed in the scope from
which it was called with reference to the scope
that it had. In Go, were only ever executing the
function in the context with which it was called;
unlike JS, the function itself is not returned the
result of that function is.

Slices
Arrays in Go are interesting things. For a start, theyre
known as slices. You can create a slice that has a set
capacity but no length. If we know what were going
to put into a slice right away then we can create one
like this:

numbers = []int{1, 2 ,4, 8, 16, 32, 64,


128}
//numbers[3] == 8
Like in C, If we dont immediately have the
information that were going to store in the slice, we
have to tell it how many items it can contain. We do
this by constructing a slice with the make keyword.
This slice will have a length of 5:

numbers := make([]int, 5)
If we assign numbers[0] = 2 it would behave as wed
expect, but if we try to assign numbers greater than
the length of the slice well get an out of range error.
Note that slices can be sliced into smaller slices.
For instance, if we have a slice of numbers, we can

98 Linux & Open Source Genius Guide

create a subset using the following code:

var places = [3]Coordinates{}

numbers = []int{1, 2 ,4, 8, 16, 32, 64,


128}
lessNumbers = numbers[1:3]

Then we could pass through a struct to the slice like


this:

lessNumbers will be a slice of the values from 1 to 3


of numbers.

Packages

Structs
Go has structs which are kind of like classes in C or
objects in JavaScript. We can create instances of
structs to store variables. Lets say we wanted a
way of storing map co-ordinates. Lets look at the
example code below.
When we set the variable, the order in which we
pass the variables into the struct is the order that
the properties within will be assigned. Reassigning
a value after its creation is simple:

places[0] = Coordinates{1.1, 2.2}

You may have noticed so far that we always include


package main whenever we write a Go script. Go
comes with a large catalogue of packages that
can be used in any Go program. The main package
tells our program where to execute from. If we were
writing a package to be used in another script, we
would give it another name. Lets say we made a
Bournemouth package:

package Bournemouth

london.longitude = 0.1075

func isAwesome() bool{


return true
}

If we wanted to have a slice of structs then we would


declare the type that the slice is to contain the struct
like so:

func isSunny() bool{


return true
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

numbers = []int{1, 2 ,4, 8, 16, 32, 64, 128}


//numbers[3] == 8
numbers := make([]int, 5)
numbers = []int{1, 2 ,4, 8, 16, 32, 64, 128}
lessNumbers = numbers[1:3]
//#### Structs
package main
import "fmt"
type Coordinates struct {
latitude float32
longitude float32
}
func main(){
london := Coordinates{51.5072, 0.1275}
bournemouth := Coordinates{50.7200, 1.8800}
fmt.Println(london.latitude, london.longitude) //51.5072, 0.1275
fmt.Println(bournemouth.latitude, bournemouth.longitude) //50.7200, 1.8800
}

Example slices & structs.go code listing


Q A simple implementation of structs to store individual geo co-ordinates

Linux & Open Source

Genius Guide

Using packages together


Go has many well-built packages. This example
shows how they can work in tandem
In this small example were using quite a
few of Gos built-in packages. We have the
HTTP library, the I/O library for writing les,
and the ag library for parsing arguments
made to the script. When you execute go run

requestStore.go

-URL=[ANY

VALID

WEB

Youll notice that were not calling the main function


here. This is because this script isnt going to be
executing any code on its own it exists to be used by
a program. Run:
go build
and if you dont see any errors, then weve
successfully made a package. Run:
go install Bournemouth.go
and Go will move this package to the pkg folder
in our Go build. Now if we were to import the
Bournemouth package into one of our programs,
the methods isAwesome() and isSunny() would be
available to the main package, just like Sqrt() or Pi() is
when we import math.

Putting it all together


So weve covered packages, functions, variable
declarations, structs and slices. That should be
enough to write a simple but useful program. So lets
do that. Create a le called distance.go.
The rst thing were going to import (after the
usual fmt) is the math package. Were going to be
doing some calculations, so were going to need the
square root function that this package exposes.
Next, we create the Coordinates struct, Just like
we did before but this time were including a name
property too.
Now we want to create a places slice. For this
example were going to have three locations (London,
Bournemouth and Glasgow) so well create a slice
with a length of [3]. Were going to be containing
Coordinates structs in this slice so we need to declare
that after the [3]. After that, we enter the geolocation
co-ordinates for each location latitude then
longitude then the name.
The distance function is quite simple. The
calculation will work out the difference between
the two points on a map and then convert that to
miles. (If youd like kilometres, you can multiply the

ADDRESS] it will parse the URL ag, make an


HTTP request to the page specied, get the
response body of that request and then store
it in a le. If you combine the elements of this
code with the distance calculator, you should
be able to pass in new points.

returned value of the distance function by 1.6.)


In the main() function we can assign pointA and
pointB to any item in the places slice. When we run the

script, itll output how far it is between points A and B.


And there we have it, our rst Go program that uses all
the parts of this tutorial.

Finishing up
Its always hard covering the ins and outs of a new
coding language, since every individual aspect of
the latter could easily take up this space on its own.
Hopefully, this tutorial will have given you enough of a
reason to get excited about using Go in some of your
day-to-day projects.

Arrays in Go are
interesting

1 package main
distance.go
2
code listing
3 import(
4
"fmt"
5
"math"
6
)
7
8
9 var places = [3]Coordinates{ Coordinates{51.5072, 0.1275, "London"},
Coordinates{50.7200, 1.8800, "Bournemouth"}, Coordinates{55.8580, 4.2590,
"Glasgow"} }
10
11 type Coordinates struct{
12
latitude float64
13
longitude float64
14
name string
15 }
16
17 func distance(a,b Coordinates) float64{
18
19
return math.Sqrt( ( a.longitude - b.longitude ) * ( a.longitude b.longitude ) + ( a.latitude - b.latitude ) * ( a.latitude - b.latitude ) )
* 69.174; //69.174 miles = 1 degree of latitude on Earth's surface
20
21 }
22
23 func main(){
24
25
pointA := places[0]
pointB := places[2]
26
27
28
fmt.Println("It is ", distance(pointA, pointB), " miles from ",
pointA.name, " to ", pointB.name)
29
30 }
Q The code for distance.go. Using co-ordinates, we can calculate the distance between two points

Linux & Open Source Genius Guide 99

Masterclass

MASTERCLASS
Event-driven
programming with Qt
Advisor

Tam Hanna has been in


the IT business since the days
of the Palm IIIc. Serving as
journalist, tutor, speaker and
author of scientic books, he has
seen every aspect of the mobile
market more than once

Resources
Documentation:
qt-project.org/doc/qt-5.0/qtcore/
signalsandslots.html

QtGui index:
qt-project.org/doc/qt-5.0/qtgui/qtgui-index.
html

QML intro:
qt-project.org/doc/qt-5.1/qtdoc/
qmlrststeps.html

Further reading...
planet.qt-project.org
qt-project.org/doc
qt-project.org/forums
blog.qt.digia.com

100 Linux & Open Source Genius Guide

Keeping your program components separated


simplies testing. Enjoy the benets of eventdriven coding, with minimal extra effort
Did you ever feel like reusing a part of a program,
only to find yourself restricted due to various
dependencies on existing code? Did you ever want
to handle user input in an engine, but didnt want to
make the algorithm dependent on the GUI toolkit?
The signal-slot system permits you to connect
application components to one another using an
event-like paradigm. Instead of having objects call up
one another, signals are emitted to the framework.
These are then routed to so-called slots, which are
invoked in order to handle the processing.
Most computer science problems can best be
understood via an example. In the following tutorial,
we will create a little application which is to analyse
ohmic resistors (whose value is determined by a

series of coloured rings). It will use a computational


engine which communicates by receiving and
emitting signals.
Traditionally, creating an event framework was
a significant undertaking. Many a developer chose
to deploy Qt because of the signal-slot system. This
honour is well deserved: it is difficult to find a use case
which can not be handled by the framework.
In addition to reducing complexity, signals and slots
are needed for interacting with controls provided by
Qts GUI stacks. Finally, most modules and libraries
integrate themselves into the signal-slot system.
So, if you want to use Qt, you will need to know
your way around signals and slots. In real life, an event
is any occurrence of note: the opening of a club, the

QAfter startup, Qt Creator shows a start pane 


with some instructions for getting started

QFour lists and a button are needed

Linux & Open Source

Genius Guide

QAdd some colours to the first three list boxes

launch of an aircraft and the birthday of a friend can all


be considered events. In computer science, a similar
definition can be construed.
When dealing with Qt, an event is any happening
which the class decides to share with the outside
world. Most QtGui widgets emit a wide variety of
events. They can be handled in order to create the
actual application logic. Your own classes can emit
as many or as few events as you see fit. As we will see
later, the framework imposes very few limitations in
this regard.
Due to the widespread usage of the term event,
compatibility issues could arise if Qt used a keyword
called event. This is averted by the use of the signal
and slot monikers. In Qt, a signal is an event. The
framework routes signals to one or more slots, which
then process it further.
Like in the real world, event routing takes place on
an interest basis. If a partygoer alerts his mate to bring
a second crate of beer, it is likely that other partygoers
also hear this call. However, they simply ignore the
statement as it is of no interest to them.
Qt mirrors this behaviour in the registration process.
When a signal is emitted, the framework checks if any
slots have registered interest in it. Notifications get
dispatched only to those objects.

Declaring signals and slots


Our QtGui-based example will constrain itself
to dealing with resistors with four bands. This
provides us with a total of four incoming and
one outgoing events. QObject-derived classes
are automatically integrated into the signal-slot
framework the header of our ResistorCalcClass
looks as following:

class ResistorCalculator: public QObject


{
Q_OBJECT
public:
ResistorCalculator();
public slots:
void ring1Changed
void ring2Changed
void ring3Changed
void ring4Changed

(int
(int
(int
(int

_toWhat);
_toWhat);
_toWhat);
_toWhat);

signals:
void newValue (QString _newVal);
private:

void recalculateValue();
int myVal1;
int myVal2;
int myVal3;
int myVal4;
};
Being based on a QObject-derived class requires the
presence of the Q_OBJECT macro. It is needed for
the moc compiler,.
Slots can be declared publicly or privately. In
practice, private slots are rarely used: some versions
of Qt even permit you to substitute the public slots
declaration with a short form. The actual denition
of the slot takes the form of a void member function,
which receives the parameters that are specied in
the signal.
Declaring signals can prove a bit more difcult.
Signals get emitted by the object, and therefore
have no local representation. Due to that, their
declaration consists of a void function with a
parameter for each bit of data that is to be passed to
the recipient.
Finally, the class needs a few member variables
to track the state of the incoming data.

Linux & Open Source Genius Guide 101

Masterclass

Implementing signals and slots


Slots receive incoming events and are implemented as
normal member functions. It is permitted to invoke a
slot like any other function from your code.
In our case, the ring-changed functions update
the data stored in the model. They also invoke the
common recalculation function. It has the job to
compute the current value, which is then emitted via
the aforementioned signal:

void ResistorCalculator::ring1Changed (int


_toWhat)
{
myVal1=_toWhat;
recalculateValue();
}
Now lets look at the function which determines the
actual return value. Our example uses an abridged
computational algorithm which does not provide valid
values for some input combinations:

void ResistorCalculator::recalculateValue()
{
float retval;
retval=myVal1*10+myVal2;
switch(myVal3)
{
case 0:
retval=retval;
break;
case 1:
retval=retval*10;
break;
...
default:
emit newValue("No Multiplyer");
return;
}
emit newValue(QString::number(retval));
}
In this part of the code, the emit call is the only one of
relevance. It is called to fire off the signal in question.
Please note that emit does not act as a return
statement. Method execution continues normally after
the emitting of the signal.

Creating a QtGui user interface


Now lets create the user interface. Double-click on
mainwindow.ui to open it in the resource editor. Drag

QConnect the signals from the list with your newly created slot

the borders of the form to achieve a decent size. The


toolbar on the left side contains a group of controls:
drag four Item-Based List Widgets and a Button onto
the form. Feel free to arrange the controls in any way
you nd useful.
Adjust the properties displayed in the bottom-right
dialog to create a user interface similar to the one
shown on page 61. Due to the relative complexity of the
editing involved, you can also take the mainform.ui file
provided on the cover disc the editing of the object
names and the button caption is of no concern to the
signal-slot system.
Qt Creator provides a special tool for editing the list
contents. It can be accessed by right-clicking the list.
Then, click Edit items in order to bring up the editor.
Just add a few colours of your choice.
In the next step, it is time to create the signal and
slot connections. This can be done in a special editor,
accessed by clicking the Edit Signals/Slots icon in
the toolbar. Once it is activated, click one of the lists
to mark it as the signal source. As long as you hold
the mouse button, the cursor will change into an
inverse tree signal. Drop it on the forms background
to set the form as the recipient. Create a new slot
by clicking Edit in the popped-up form. Click the +
button under the Slots list to create a new slot with

The four slots must be integrated


by hand into the mainform class
along with corresponding signals
102 Linux & Open Source Genius Guide

corresponding parameters. Click OK to conrm and


close the editor. Select the currentRowChanged and
the newly created properties in order to link them up.
Repeat this for the four other lists. Finally, link the
clicked() signal of the button with the closed() slot
of the form it is shown only if the Show inherited
checkbox is enabled.

Connecting stuff
With that, it is time to connect the QtGui with the
engine. The four slots you created in the editor must be
integrated by hand into the mainform class along with
corresponding signals the editing of the header is
straightforward; the method body looks like this:

void MainWindow::item1Changed(int _newVal)


{

QCreate a slot in the main form by clicking the +


button. Here, three newly created slots are shown

Linux & Open Source

Genius Guide

Creating attractive
user interfaces
with Qt
QtGui framework and QML:
two solutions for a common
problem
When Qt was rst envisioned, JavaScript and
HTML were in their infancy using them to
create complex user interfaces was infeasible.
Furthermore, the then-current computer
systems were unable to cope with the
computational demands raised by animated
GUIs. Instead, developers worried about the
ever-rising screen sizes seen on desktop
workstations.
The QtGui framework (which well use to
create our example application) addresses
these issues. It lets you use native controls
and provides a variety of helpers which handle
resizing and positioning. However, it is not well
suited to animation.
In Version 4.7, Nokia introduced a new GUI
stack called Qt Modeling Language (QML).
This JavaScript- and JSON-based system
offered impressive animation facilities and
initially was intended solely for mobile phones.
In theory, QML elements consist solely of
rectangles, bitmaps and texts which are
rendered using OpenGL.
This approach greatly simplied porting the
framework to new operating systems. Due to
that, Digia decided to make QML available on
all platforms. A set of predened controls will
eventually simplify the creation of commonly
used widgets.
Using QtGui usually leads to quicker results,
as controls do not need to be assembled by
hand. Furthermore, the native integration
ensures that the results look exactly like other
desktop apps.

QThe signal-and-slot mode provides an overview of the connections in your form

emit changed1(_newVal);
}
Here, we forward the signal via a slot. Theoretically, we
could also connect signals to one another this special
case is described in the documentation. The signal from
the engine must also be processed. For this, create
another slot with the following code. Also, include the
QMessageBox via an #include:

void MainWindow::datareceived(QString _val)


{
QMessageBox aMessageBox;
aMessageBox.setText(_val);
aMessageBox.show();
}
Finally, it is time to connect the signals and slots. This is
ideally done in the constructor of the form, which is to
be modified as shown in Fig 1.
The connect macro takes pointers to sender
and receiver. In addition to that, information must

MainWindow::MainWindow(QWidget *parent) :
QMainWindow(parent),
ui(new Ui::MainWindow)
{
ui->setupUi(this);
myEngine= new ResistorCalculator();
connect(this,SIGNAL(changed1(int)),myEngine,SLOT(ring1Changed(int)));
connect(this,SIGNAL(changed2(int)),myEngine,SLOT(ring2Changed(int)));
connect(this,SIGNAL(changed3(int)),myEngine,SLOT(ring3Changed(int)));
connect(myEngine, SIGNAL(newValue(QString)),this,SLOT(datareceived(QString)));
}

Fig 1

be passed in about the signals and slots and their


parameters. In principle, the parameters of the signal
and the slot must match 1:1 special conditions are
explained in the URLs given in Resources (page 60).
By the way, signals and slots created in the editor are
connected automatically.

Analysing the benefits


After all this coding, it is time to recapitulate what
we have done. In addition to learning about the
manipulation of QtGui-based user interfaces, a
component was created which computes resistance
values. ResistorCalculator is completely independent
of the user interface: it just needs four incoming signals
and a recipient for its slot.
For a class as trivial as this one, the benefits might
not be visible immediately. However, lets assume
that ResistorCalculator would implement a complex
computation determining trajectories or amortisation.
In this case, reuse would be a highly attractive
proposition. Exchanging the QtGui front-end for a
QML front-end or even a headless server becomes
a question of adapting a few connect() calls in the
constructor of the host class.
The power of this, alone, justifies the extra effort
needed for implementing signals and slots. However,
the isolation has further benefits. Imagine the creation
of an automatic test script. Thanks to the signal-slot
system, all it has to do is create an instance and fire off
a few signals mock objects are not needed.
In short, using the signal-and-slot system tends
to pay off in most cases. With that, its time to end our
travel through the fascinating world of Qt for this issue.
Next time, well look at layouts and size policies. Until
then, keep coding

Linux & Open Source Genius Guide 103

Masterclass

Responsive development with Qt


Monitors and devices come in all shapes and sizes. Qts layout system
ensures that your app looks good, no matter how big the display
Since Trolltech set out to develop Qt, screen
sizes have increased rapidly. Ten years ago, a
15-inch CRT monitor was considered a decent
screen today, cheap notebooks come with a
17-inch Full HD display. Developers were left
standing in the rain an unadjusted application
looks lost when running on a large screen.
A well-made Qt application looks great on
a large variety of screen sizes. Your advisor is
aware of some products which use the same .ui
file on smartphones and workstations on both
systems, users are satisfied with the results.
This is achieved by a reversal of control. In
traditional user interface systems, controls are
positioned where the developer drops them. Lets
assume a window with a size of 200x200 pixels.
The size of the controls would be specied inside
the window a button is located at 60|60//120|70.

If the user increases the size of the window, the


button remains stuck at its original position.
Additional screen real estate would be occupied
by white space.
An ideal computer analyses the user interface.
Elements would be resized and repositioned
as to their usefulness: content controls would
gain size, whereas modiers and buttons would
remain clickable.
This can be achieved by liberating the layout
engine from the constraints of developerprovided co-ordinates. A exible user interface
is specied in terms of relations and growth
behaviours, dependent on the screen size of the
device being used. At runtime, a layout engine
performs the actual positioning and determines
the actual co-ordinates using the description
provided by the developer.

QtGui-based user interfaces achieve this via


the layout system. Controls are to be placed in
containers which contain information about
the intended arrangement. In addition to that,
invisible meta controls (spacers) expand at
runtime, pushing widgets to the border of their
relative container.
The knowledge gained in the following article is not
limited to the QtGui stack. Cascades, XAML and most
JavaScript frameworks take a similar approach to
dynamic resizing of elements. Web developers refer to
it as responsive web design, but tend to use the same
processes and idioms outlined here.

A first layout
It is now time to start applying all of this knowledge
in practice. Our first example will consist of a group
of buttons which are arranged above one another

A well-made
Qt application
looks great on a
large variety of
screen sizes

QApplying a layout to the form immediately arranges the controls

QOur empty layout shows up as a red line

104 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

Missing form layout!


A tiny issue causing strife and
troubles since day one
When a form is rst created, Qt Creator
does not provide it with a layout. This does
not hinder you from placing widgets and
sublayouts onto it you can even set size
policies and other layout-related attributes.
These will, however, be ignored completely
at runtime. This strange behaviour is caused
by a rarely documented property of the Qt
layout engine: forms which do not contain
a form layout are not managed. Therefore,
attributes such as size policy will not be
adhered to.
Fortunately, identifying this problem
is relatively easy. Resize your form in the
designer and look at the behaviour of the
controls: if they do not adjust their sizes
accordingly, your form is likely missing
a layout

QThe three buttons have become part of the layout

we will then expand it to include a group of other


widgets which will be arranged in a more or less
complex fashion.
Start by launching Qt Creator. The following tutorial
uses Qt Creator 2.8.1, the examples are based on Qt
5.1.1. Older editions of the integrated development
environment will behave similarly the layout system
itself has not been changed much since Qt 4.0.
Layouts tend to be based in forms. Doubleclick the le mainwindow.ui in order to open it in
the WYSIWYG editor. The toolbox on the left side
of the screen contains a group of controls. Drag
a few buttons into the form and drop them off at
a convenient position. Then run the program and
resize the MainWindow by dragging its borders. The
positions of the buttons will remain unchanged.
Close the application after you have convinced
yourself that static layouts cannot adjust to changes
in the form size. Return to the WYSIWYG editor and
take a look at the icons next to the filename chooser
displayed in the top-right corner of the screen.
The sixth button from the left has a symbol made
up of three horizontal lines shown above one another.
Select the form in the tree and click the button in
order to add a horizontal layout to the form.

Your buttons will immediately be arranged above


one another. Changing the size of the form will not
affect this placement the three buttons will stay in
the middle of the window as it keeps changing its size.
This behaviour is shown in the editor and at runtime.

More layouts
Forgetting to assign a form layout disables the layout
system in its entirety. Sadly, most cases require
more than one layout per form arranging a group of
buttons horizontally under a list is not possible with
just a horizontal or a vertical layout.
Qt permits nested layouts. A horizontal layout can
contain a vertical layout which then, in turn, contains
another horizontal layout. The high performance
of contemporary computer systems ensures that
complex forms will still be rendered in real-time. In
the last ten years, your advisor has not experienced a
single case of lag caused by overcomplex layouts.
Creating nested layouts is relatively easy. In our
example, add a text widget to the main form by
dragging it in from the toolbar. Qt Creator will display a
blue line in the form. This informs you that the control
will be placed in a layout and provides indication of its
location in the layout hierarchy. Keep in mind that you

Changing the
size of the form
will not affect
this placement
the three buttons
will stay in the
middle
should not affect the size of the elements of a layoutmanaged form this is achieved by more subtle
means discussed later in the article.
For now, it is enough to place the text box above
the buttons. The toolbar on the left side of the
screen contains a group called Layouts. Select a
horizontal layout and drag it onto the form in order to
add it. Initially, the layout will show up only as a slim
red line.
Due to an internal limitation of the editor, layouts
can not be modied while a form layout is in place.
This can be xed by clicking the form and selecting
the option Break layout in order to strip the form of
its layout intelligence. When this is done, the layout
will show up as a small red square drag the three
buttons into the rectangle in order to add them to the
internal HorizontalLayout.

Linux & Open Source Genius Guide 105

Masterclass

When finished, the three controls will be displayed


inside the red rectangle. The layout can be moved en
bloc: selecting the layout as a whole is easiest if you
drag a selection rectangle around part of the controls.
Actual movement is then accomplished via the arrow
keys.
In the nal step, dont forget to re-enable the form
layout. If this is done, the resulting form is made up
of a large and expanding text box and three slim
buttons glued to the bottom of the screen.

A question of policy
Looking at the text box and the buttons in the example
leads to a puzzling conclusion. The Qt framework
somehow knows that some controls can profit from
extra height, whereas others do not. This information is
used at runtime in order to give additional height to the
textbox, while the buttons remain small.
Each widget has a sizePolicy attribute for the
vertical and the horizontal direction. These properties
can be modified by selecting one of the widgets
and setting up its properties accordingly. The seven
attributes that are supported, as of Qt 5.0, are outlined
in Fig 1.
This, however, is not the only possible way the layout
system can be affected. If controls are intended to
grow at different rates, this can be configured using the
Horizontal and Vertical Stretch properties. By default,
they are set to zero so the controls all grow and
shrink together.
Assigning a value to horizontal stretch and/or
vertical stretch permits you to modify the size-gaining
behaviour. When controls are laid out, each widget is
assigned the minimum required space. If additional

Fig 1
QSizePolicy::Fixed

This widget can neither


shrink nor grow

QSizePolicy::Minimum

This widget can


theoretically grow, but
the minimumSize is
more than enough

QSizePolicy::Maximum

This widget can be


reduced in size if other
widgets can use the
space better it can
NOT grow

QSizePolicy::Preferred

This widget can be


shrunk if needed
growing is possible
but does not lead to
advantages

QSizePolicy::Expanding

This widget can prot


from extra space, but
can be shrunk if the
need arises

QSizePolicy::MinimumExpanding

This widget can prot


from extra space, but
can NOT be shrunk if the
need arises

QSizePolicy::Ignored

This widget tries to


grab as much space as
possible

106 Linux & Open Source Genius Guide

QInflating the size of the form does not affect the arrangement of the widgets

display real estate is available, it is shared according to


the stretch values set in the corresponding direction.
For example, if you had three buttons with their
horizontal stretch factors set to one, two and three
they would correspondingly take up one, two and three
times as much space at the bottom of the created form.

Minimum and maximum sizes


In an ideal world, the computer would be able to
automatically detect the minimum and maximum
sizes needed by widgets. Sadly, practical attempts at
using artificial intelligence for determining minimal
and maximal heights has, by and large, proven to be
unworkable. Instead of that, Qt provides developers
with two properties called minimumSize and
maximumSize.
Their behaviour is largely logical. A control will
never become smaller than the values specied
in its minimumSize, and never exceed the size of
maximumSize. A newly set up control usually has a
minimumSize of 0 and a maximumSize of 16777215
this means that the framework can compute the
minimal size from the content and can make the
widget as large as possible.

Attention,
excessive width
Manufacturers of monitors
have their own agenda
In the last years, the diagonals of screens have
grown considerably. At the same time, the
aspect ratio has become atter and atter:
today, screens tend to be much wider than
they used to be. This new situation provides
considerable challenges to user interface
developers. Very wide texts are difcult to read
it is for good reason that most books and
newspapers are printed in portrait mode. When
creating an application optimised for very wide
displays, try to think about how the additional
horizontal screen real estate can be used.
Android tablet apps tend to provide useful
inspiration thanks to their fragment layouts,
which are used to give additional context.
For example, an email client would display
a hierarchy of folders next to the actual
message.

Analysing and comparing the sizes


of the widget to the base size allows
you to compute the ideal font size

Linux & Open Source

Genius Guide

Fig 2

class Ui_MainWindow
{
public:
QWidget *centralWidget;
QVBoxLayout *verticalLayout;
QTextEdit *textEdit;
QHBoxLayout *horizontalLayout;
QPushButton *pushButton_3;
QPushButton *pushButton_2;
QPushButton *pushButton;
QMenuBar *menuBar;
QToolBar *mainToolBar;
QStatusBar *statusBar;

Analysing the process behind


the WYSIWYG editor

void setupUi(QMainWindow *MainWindow)


{
if (MainWindow->objectName().isEmpty())
MainWindow->setObjectName(QStringLiteral("MainWindow"));
MainWindow->resize(395, 228);
centralWidget = new QWidget(MainWindow);
centralWidget->setObjectName(QStringLiteral("centralWidget"));
verticalLayout = new QVBoxLayout(centralWidget);
verticalLayout->setSpacing(6);
verticalLayout->setContentsMargins(11, 11, 11, 11);
verticalLayout->setObjectName(QStringLiteral("verticalLayout"));
textEdit = new QTextEdit(centralWidget);
textEdit->setObjectName(QStringLiteral("textEdit"));
verticalLayout->addWidget(textEdit);

Qt provides developers with two


additional layout components
Advanced layouts
Most computer problems can be solved by using the
vertical and horizontal layouts. Sadly, some more
complex layouts can not be addressed in this fashion.
Qt provides developers with two additional layout
components which help to simplify the creation of
advanced forms.
Grid layouts permit you to arrange widgets in a
shape similar to the ones used in Excel or LibreOfce
spreadsheets. After dragging one of the widgets onto
the screen, the layout will pop up on the screen as an
empty rectangle. Then, controls can be dragged in by
one another. When dropping the widget, a blue line
will be shown on the right or the left side of the layout
dropping it there adds a new row or column to the
grid layout.
Form layouts are a specific from of the
aforementioned grid. A form is made up of two
columns: the one on the left contains the labels,
whereas the control on the right side is intended for
collecting the actual input.

Spacers for efficiency


Creative use of layouts permits you to create a large
variety of attractive forms. Sadly, some effects can

Where is the
intelligence?

not be achieved without the use of spacers. Spacers


are a design-time-only component which is not visible
to the user.
At layout time, spacers have the habit to try to
extend to the maximum size possible. Placing a
spacer on the left side of a control ensures that it
will be pushed to the right side of the screen as far
as possible. Aligning a control to the middle of the
screen is possible by using two spacers at the sides
of the control.

When a .ui le is saved, the Qt toolchain


transparently creates a ui_*.h le. This header
usually contains a setupUi method which
assembles the windows described in the ui le
out of the classes provided in the QtGui stack.
As an example, take a look at the snippet in
Fig 2, which was taken from one of the forms
created as this article was written.
Please keep in mind that these les are
automatically regenerated whenever the form
is saved. Due to this, manual changes in the ui
le are not recommended developers using
source control systems are advised to keep
these les out of the CVS at all expense.

Conclusion
Due to a long tradition of Visual Basic, Windows users
do not expect their applications to show an overly large
amount of flexibility as forms are resized. Due to the
widespread adoption of Qt, Linux and UNIX users are
far less tolerant of applications which do not resize
themselves intelligently.
Creative use of the layout system provides you with
the aforementioned logic with minimal extra code.
Arranging your controls in layouts can be done in mere
seconds, but leads to significantly better applications.
Practical experience shows that the layout
system can be daunting due to the large amount of
functionality offered. Most developers tend to get
acquainted with the handling within few hours let this
article be your guide to more attractive and responsive
user interfaces.

Font sizes?
As screen densities increase, font sizes eventually
become problematic. Keeping them inline is
currently not possible in the framework. Fortunately,
addressing the issue is easily done in code. Most
controls contain an event handler which will be
invoked whenever the layout engine changes the size
of the elements of the screen.
Analysing and comparing the sizes of the widget to
the base size allows you to compute the ideal font size.
The actual correlation can be done in a linear or in a
stepped fashion advanced information on this can be
found in classic typography literature.

QThe three buttons have become part of the


layout

Linux & Open Source Genius Guide 107

Masterclass

Reading and writing files with Qt


Reading and writing les stored on disk is essential for more complex
applications. Qt provides a common interface which eliminates wrangling
with the native APIs
Even though the C standard library provided some
sort of common API for disk access, creating a truly
cross-platform application remained a daunting
task. This is caused by differences in the underlying
operating system. C:\ is a valid path under Windows,
but does not work for most UNIX systems.
The C/C++ le system API was specied in
the 1970s and 80s. Back then, databases and
XML les were unknown data was stored in at
les containing a group of entries back-to-back.
Accessing XML or SQL records in vanilla C/C++
requires an awful amount of glue code.
Fortunately, Qt provides a variety of classes which
help you develop applications swiftly and effectively.
If you design your programs correctly, they can move
from Linux to Windows and/or Mac OS X without
rewrites Qt handles the idiosyncrasies of the
different le systems. Furthermore, a group of utility
functions is made available for accessing complex
data formats such as XML and SQL databases.
This time, we will walk through the creation of
a le-nding utility. The intent is to teach you a
variety of procedures needed when accessing the
le and directory system. In addition to that, we
introduce and analyse common pain points which

cause problems for beginners. Future parts of this


series will introduce you to databases, XML and
other forms of advanced storage.
Our program will start out by displaying a list of the
files and folders found in the home directory of the user.
Sadly, finding its location is not as straightforward as
one might think at first glance keep in mind that our
application might run under Windows, Linux and Mac
OS X at some point in the not-so-distant future.
Qt provides a class which lets us determine the
location of the users desktop path. So, the first act of
our program consists of determining where to start our
journey through the file system:

MainWindow::MainWindow(QWidget *parent) :
....
{
....
myDesktopPaths=QStandardPaths::standardLocat
ions(QStandardPaths::DesktopLocation);
myCWD=myDesktopPaths[0];
}

Enumerating the directory


Once the CWD has been determined, we must proceed
to showing the user an overview of its contents. This is
done via a process called enumerating the directory.
Performing an enumeration is quite simple: on query,
the file system will return a list containing all of the
entries in the folder. We can process these for display or
further analysis. In our case, well use them to populate
a QListWidget displayed in the main window. The code
needed is as in Fig 1.

Calling the static standardLocations method of


QStandardPaths lets you query the system for a

QHead to the Qt project website (http://qt-project.org/) to download the latest version of Qt. On the
website you can also access helpfuflu l forums, development topics and tools

108 Linux & Open Source Genius Guide

variety of commonly used storage locations. As some


operating systems store data in multiple folders, a
StringList is returned. We simply grab the rst entry in
the list and set it as the working directory. myCWD is a
declared member of the MainWindow class.
QStandardPaths was introduced with Qt 5.0. When
dealing with legacy versions of Qt, you will sometimes
see code using static methods of the QDir object. By
and large, they behave similarly to the module used in
this feature here.

QThe file sleuth displays the contents of the


authors desktop

Linux & Open Source

Genius Guide

Fig 1
void MainWindow::enumerateCWD()
{
myDir.setPath(myCWD);
enumContents=myDir.entryInfoList(QDir::NoFilter, QDir::DirsFirst);
for(int i=0;i<enumContents.count();i++)
{
if(enumContents[i].isDir()==true)
{
ui->listWidget->addItem("[" + enumContents[i].fileName() + "]");
}
else
{
ui->listWidget->addItem(enumContents[i].fileName());
}
}
}

For reasons of completeness, we will also print


the header containing the declaration of the member
variables of the MainWindow class:

class MainWindow : public QMainWindow


{
...
private:
Ui::MainWindow *ui;
QString myCWD;
QDir myDir;
QFileInfoList enumContents;
};
Qt represents directory entries as instances of the
QDir class. Therefore, our code starts off by providing
myDir with the path which it has to enumerate. The
entryInfoList method is responsible for returning a
list of the contents of the currently set path.
It can, furthermore, sort and filter the returned
entries. In our case, we disable filtering by passing in
the NoFilter constant. However, we pass DirsFirst
in order to make the method return directory entries
before the files contained in them.
When the list has been obtained, our code proceeds
to dumping its contents into the list widget. The ui
object contains references to all widgets in the form
its handling was discussed in the previous issue.
In addition to the entryInfoList method, Qt also
offers a simpler method. It returns a StringList
which does not provide further information about
the entries in question. Using it is sensible if the
output gets ltered in our case, we would not be
able to distinguish les from directories.

to add some further logic which permits the user to


traverse the actual file system.
For this, we need to handle clicks on entries in the
list. These can be routed to the form by connecting
the corresponding signal with a slot. Then, the
following event handler will be invoked on clicks:

void MainWindow::myItemClicked
(QListWidgetItem* _inItem)
{
if(_inItem->text().compare("[..]")==0)
{//Go up
myDir.cdUp();
myCWD=myDir.absolutePath();
ui->listWidget->clear();
enumerateCWD();
}
else if(_inItem->text().compare("[.]")==0)
{//Do nothing
}
else
{//Go down
int targetItem=ui->listWidget>currentRow();
if(enumContents[targetItem].
isDir()==true)
{
myDir.cd(enumContents [targetItem].
fileName());
myCWD=myDir.absolutePath();
ui->listWidget->clear();
enumerateCWD();
}
}
}

Navigating across the file system


Displaying a list of files on the desktop might be an
useful proposition, but does not justify the effort of
creating a dedicated application. So, it is now time

The itemClicked signal comes with a pointer to


the list widget item which was clicked. We analyse
its text in order to determine the correct action. If

Recursion
What goes around tends to
come around
In computer science, many problems can be
solved easily by breaking them down into little
steps. For example, let us compare the strings
aaaaaab and aaaaaac. Even though most
programmers would attack this via a loop, let
us, for now, suggest a different approach. Let
there be a comparison function cmp which
compares two characters and returns a value
indicating whether they match. A program
could then be constructed which compares the
current characters, then reduces the string
by one and passes them on to cmp(). Inside
cmp, the characters are compared again if
they match, the string is reduced again and
cmp() is invoked on the new result. This process
continues until the end of the string (or a
disparity) is reached, in which case the instance
of cmp() would return the corresponding code.
Once the recursion has been broken, the
computer will automatically unwind its call
stack. This means that the value from the
innermost cmp() function will be propagated
out until it nally arrives back at the caller.
When applied right, recursive solutions tend
to be very compact. The overhead caused by
the extra stack frames is insignicant on most
modern computer systems.

the user clicks the .. item, it is time to traverse up


by one level. This is achieved by calling the cdUp
method of the myDir object. After that, the CWD is
updated and the list is repopulated.
Clicking the . item is of no importance to our
application. Clicks to actual entries lead to the
analysis of the corresponding item in the le-entry
info structure. If a folder is found, the QDir instance
is moved into it. Finally, the contents of the list are
updated.
Some tutorials change the directory by performing
string manipulations on the current working
directory. This method is not ideal for cross-platform
applications: Windows uses backslashes to delimit
les, whereas other operating systems tend to use a
normal slash.

Determining the file system root


Even though a co-operative user can now use our
application in order to traverse the les found on
their desktop, a glaring omission remains. An evilminded person could traverse above the root of the
le system, thereby causing application crashes
or worse. We have now reached a point where the
differences between the various operating systems

Linux & Open Source Genius Guide 109

Masterclass

come home to roost. Under UNIX, the root of the


system is /. Windows instead provides the user
with a group of drives which are not unied under a
common root.
Fortunately, handling this is relatively easy. The
cdUp function returns false whenever the file system
does not provide a parent folder. Even though this can
also be caused by lack of writing privileges, we will for
now assume it to be an indication for the user hitting
the root folder. In that case, we will provide the user
with a list of drives:

void MainWindow::myItemClicked
(QListWidgetItem* _inItem)
{
if(_inItem->text().compare("[..]")==0)
{//Go up
if(myDir.cdUp()==true)
{
myCWD=myDir.absolutePath();
ui->listWidget->clear();
enumerateCWD();
}
else
{//Enumerate drives
enumContents=myDir.drives();
ui->listWidget->clear();
for(int i=0;i<enumContents.
count();i++)
{
ui->listWidget>addItem(enumContents[i].absolutePath
());
}
myDriveListModeActive=true;
}
}
....
Calling drives() returns a data structure listing all
drives found on the current machine. In the case of
UNIX, this usually is just the root, which is denoted by
a slash. Under Windows, the application would instead
receive a group of elements. Either way, the contents
are written into the list using a process similar to the
one outlined above.
In addition to that, the click handler routine also
needs some special logic for handling clicks to drives:

void MainWindow::myItemClicked
(QListWidgetItem* _inItem)
{
if(myDriveListModeActive==true)
{
int targetItem=ui->listWidget>currentRow();
myDir.cd(enumContents[targetItem].
absolutePath());

110 Linux & Open Source Genius Guide

QSelect what kind of widgets Qt Creator should use to pre-populate your layout file

The handler for clicks on drives is


different to the one used normally
myCWD=myDir.absolutePath();
ui->listWidget->clear();
enumerateCWD();
myDriveListModeActive=false;
return;
}
....
The handler for clicks on drives is different to the one
used normally, as it sets the working directory to the
entire path pointing to the drive. Then, the enumerate
file method is invoked in order to display the contents
found therein.

Opening files for display


So far, our routine ignored clicks targeting list items. It
would be much more interesting if a click on a file would
open its contents in a text box. In order to achieve this, a
new form needs to be added to the project.
In current versions of Qt Creator, this process is
greatly simplied by the Add File or Project Wizard
available from New >New File or Project. There,
select a resource of the type Qt>Qt Designer Form
Class. In the next step, select the template Main
Window. Finally, rename the class to FileWindow

and complete the wizard to create the les needed.


In the next step, add a text box to the form. Then,
update its constructor as shown below:

FileWindow::FileWindow(QString _pathToRead,
QWidget *parent) :
QMainWindow(parent),
ui(new Ui::FileWindow)
{
ui->setupUi(this);
QFile myFile(_pathToRead);
myFile.open(QIODevice::ReadOnly |
QIODevice::Text);
QTextStream in(&myFile);
QString collector = in.readAll();
ui->plainTextEdit->setPlainText(collector);
setWindowTitle(_pathToRead);
}
Integrating the routine into the main window is
straightforward. If the element clicked is not a file, a
new FileWindow is opened with a reference to the full
path to the file in question:

FileWindow* myFileWindow=new
FileWindow(enumContents[targetItem].

Linux & Open Source

Genius Guide

absoluteFilePath(), this);
myFileWindow->show();

Traversing the file system for real


With that out of the way, it is now time to go searching
for files and folders on the computer. Our example will
limit itself to comparing the names of files and folders
to a pattern entered by the user. For this, the routine
in Fig 2 is responsible. It can be triggered via a button
added to the bottom of the file navigator.
Understanding this routine requires you to have
grasped the concept of recursion outlined on the
second page of this tutorial. MySeekClicked starts
out by allocating a QList which is to contain all files
returned from the seeking algorithm.
In the next step, the actual seeking function
seekThisFolder is invoked. It requires a pointer to
the common storage and a QDir instance pointing
at the folder which must be scanned. The calling
of entryInfoList then returns a list of the contents
found.
We proceed to inspecting each element returned.
If it is a file, its name is compared to the search string
entered; matching files are then dumped into the QList.

If we are looking at a directory, the parsing logic


is a bit more complex. First of all, we must ensure
that the le system will not send us to a path already
seen: the virtual directories . and .. are sought out
and eliminated beforehand. Then, the method is
invoked again to analyse the contents of the actual
folder in question.
Debugging recursive routines is not easy. A great
way to simplify the process is outlined in the example
above: we use qDebug to emit info about every
invocation of the recursive function. This way, a sort
of call stack is printed into the Application Output
window as the program is run.
Due to space constraints, our program will not emit
the list of found files. Instead, set a breakpoint at the
myResultList=myResultList line in order to see the
contents.

Conclusion
Even though our example program is not particularly
complex, it shows all important obstacles faced by
developers creating applications which access the file
system. The procedures outlined here can be used to
realise a wide variety of utilities.

void MainWindow::mySeekClicked()
{
QList<QString> myResultList;
seekThisFolder(myDir,&myResultList);
myResultList=myResultList;
}
void MainWindow::seekThisFolder(QDir _where, QList<QString>* myResultStore)
{
qDebug() << _where.absolutePath();
QFileInfoList localContents=_where.entryInfoList();
for(int i=0;i<localContents.count();i++)
{
if(localContents[i].isDir()==true)
{//Folder, so recurse
QString filePath=localContents[i].absoluteFilePath();
if(localContents[i].fileName().compare("..")==0)continue;
if(localContents[i].fileName().compare(".")==0)continue;
QDir myNewDir;
myNewDir.setPath(filePath);
qDebug() << "Spawning into" << filePath;
seekThisFolder(myNewDir, myResultStore);
}
else
{//File, so scan
if(localContents[i].fileName().contains(ui->lineEdit->text()))
{
myResultStore->append(localContents[i].absoluteFilePath());
}
}
}
}

Fig 2

A common example would be the analysis of log


les created by servers. Slovakian rm Tamoggemon
Holding uses a Qt program to determine the pain
points faced by users of TouchCalc manually
analysing the data would be a prohibitively
expensive process due to the large number of
transactions involved.
With that, this part of our journey through the world
of Qt ends.

Our example
program shows
all important
obstacles faced
by developers
creating apps
which access the
le system
File access can and
will fail
Peter Principle applied what
can fail will do so
In this story, our example code did not handle
possible le-system errors in order to be
shorter and more compact. When creating
commercial code, it is highly recommended
that all le-system routines are protected by
error handlers. Users can disconnect memory
cards or just go ofine while using your program
if this causes a crash, a bad rating is almost
guaranteed.
Fortunately, most methods which interact
with the le system tend to provide a relatively
simple error-handling system. Checking return
values and onFail callbacks might be tedious
while coding, but has signicant benets due
to the vastly improved user experience faced by
your clients.
Of course, one can also be overzealous.
However, experience shows that adding too
much error-handling code usually is less
critical than too little.

Linux & Open Source Genius Guide 111

Masterclass
The nal version of the
sqFile Erlang module

Running Erlang from


the UNIX shell

Documentation of an Erlang
module in HTML format

The Erlang
graphical monitor

Program with Erlang


Advisor
Mihalis Tsoukalos is
a UNIX system administrator
also procient in programming,
databases and mathematics. He
has been using Linux since 1993

Resources
Erlang:
www.erlang.org

Try Erlang:
www.tryerlang.org

Mnesia DBMS:
www.erlang.org/doc/apps/mnesia/

MochiWeb:
https://github.com/mochi/MochiWeb

112 Linux & Open Source Genius Guide

Write, document and debug Erlang


programs and use OTP to create a
fault-tolerant server process
This feature will introduce you to the
Erlang programming language, which is
used to build massively scalable
software real-time systems with
requirements for high availability. Some of its uses
are in telecoms, banking, computer telephony,
eCommerce and instant messaging. The runtime
system of Erlang has built-in support for
concurrency, distribution and fault tolerance.
Erlang was developed at Ericsson Labs
in order to serve the switching of telephone

equipment. Although it looks like a special-purpose


programming language, it is not. If you are using a
NoSQL data store such as CouchDB or Riak, you are
already using Erlang without even realising it.
The rst thing that you need to do if you are used
to programming with computer languages such as
C, Perl or Python is forget everything you know about
classes, variables and assignment of variables. You
will need to think more in terms of pattern matching,
pathways and message passing. It may sound weird
but it is also powerful and fun!

Linux & Open Source

Genius Guide

01

Introducing Erlang

Your Linux distribution probably includes a


ready-to-install Erlang package that you will have
to install. Alternatively, you can compile Erlang and
install it manually.
On a Debian 7 Linux system, about a hundred
new packages will be installed after executing the
following command (as root):

02

The Erlang
shell

# apt-get install erlang


The Erlang programming language makes hard
things easy and sometimes makes easy things
hard. It was not designed for big data, but it was
designed for massive concurrency and is a great
language for fault tolerance and distributed
applications. The syntax of Erlang does not
look like a C-based language. The Erlang way of
programming concentrates more on data ow and
you should learn to design your programs based
on recursion and message passing. Erlang can
also program web services with the help of the
MochiWeb library.
Variables start with capital letters in Erlang
and words starting with lower-case letters are
atoms. Atoms are constants with their own name
as their value.

02

The Erlang shell

The shortest interaction you can have with


the Erlang shell is the following:

$ erl
Erlang R15B01 (erts-5.9.1) [source] [64-bit]
[smp:8:8] [async-threads:0] [kernel-poll:false]
Eshell V5.9.1 (abort with ^G)
1> q().
ok
2> mtsouk@mail:~$
As you can see, if you want to exit the Erlang shell,
you will have to type q(). and press the Return/Enter
key. Make sure that you do not forget to put a dot at
the end of the command because Erlang commands
always end with the dot character.
The q() function is the easiest way to exit the
Erlang shell, but keep in mind that the q() function
quits everything Erlang is doing. If you are working
locally then there is no problem, but if you are
working on a remote system its better to quit by
typing Ctrl+G and then q. The reason is that you
may shut down the Erlang runtime on the remote
machine when quitting using q()!
The Erlang shell is perfect to test most of your
stuff before putting your code into a module.
The built-in line editor, erl, is a subset of Emacs.

03

Erlang numbers and variables

Erlang uses two kinds of numbers: integers


and oats. 10 is an integer whereas 21.0 is a
oating-point number. Integer numbers are always
exact, but oating-point numbers may get truncated
a little bit depending on their value. When you are
using oats, you should always have a number to the
left of the decimal point. In other words, you cannot
write .6; you should write 0.6.
When dening a new variable inside the Erlang
shell, you cannot change its value afterwards, as the
following example demonstrates:

1> MyVar = 12.


12
2> MyVar = MyVar + 5.
** exception error: no match of right
hand side value 17
3> MyVar = 5.
** exception error: no match of right
hand side value 5
Erlang interprets MyVar = MyVar + 5 as 12 = 12 +
5, which is obviously wrong. Erlang uses the single
assignment model, which allows each variable to
be assigned a value only once in a given session or
context. Your rst exposure to Erlang may confuse
and frustrate you, but you will soon feel captivated
and excited.

04

The Hello World


program in Erlang

Writing the Hello World program in the Erlang


programming language can prove to be a little
tricky because you will have to create an Erlang
module in a separate le, along with a function that
prints your message.
Then, you will have to compile the le and run
the function to nally get the Hello World message
displayed on your screen.
The name of the le must match the name of
the module; therefore, if the module will be named
hello, the le should be called hello.erl. You can
call the function any name you want; helloWorld
is a representative name.
Then, you will have to type the following inside
the Erlang shell:

1> c(hello).
{ok,hello}
2> hello:helloWorld().
hello, world
ok
3>
The rst command compiles the hello.erl source
le and the second command calls the helloWorld()
function of the hello Erlang module. The output is
shown on screen.

Linux & Open Source Genius Guide 113

Masterclass

05

More information about Erlang

A key part of the Erlang Run-Time System


(ERTS) is the BEAM processor. BEAM stands for
Bodgan's Erlang Abstract Machine, a virtual machine
that interprets optimised BEAM code. When you
compile an Erlang program, the compiler converts the
code into a BEAM le.
The Erlang syntax may look strange the rst time
you see it, but if you start writing Erlang programs
more often, you will stop thinking about its syntax in
less than a week!
The Erlang shell is the perfect place to test most
of your code before creating a module, but you can
also compile an Erlang program using the Erlang
compiler (erlc) without entering the Erlang shell:

$ erlc hello.erl
Similarly, you can run an Erlang function without
entering the Erlang shell:

$ erl -noshell -s hello helloWorld -s


stop
hello, world

init

Please do not forget that Erlang cannot solve every


programming problem efciently you may need to
use another programming language for a specic type
of problem. Always use the best tool for the job.

06

An Erlang module

When
creating
programs,
Erlang
programmers put their code in modules. Modules
help you organise, store and share code easily and
efciently and are the preferred way of writing Erlang
programs.
The source code for the square module, which
calculates the perimeter and the area of a square
given its side, is the following:

-module(square).
-export([length/1, area/1]).
length(Side) -> 4 * Side.
area(Side) -> Side * Side.
If you use side instead of Side as the variable name,

you will get the following warning message when you


try to compile the square module:

1> c(square).
square.erl:8: Warning: this expression
fail with a 'badarith' exception
{ok,square}

will

This happens because variables must start with


a capital letter in Erlang. The last statement of a
function is also the return value of the function. So, the
area() function returns a number.

07

Explaining the code

At the top of the module, the -module


directive dictates the name of the module, which
is square. The -export directive denes the list
of functions that are visible to the world public
functions in object-oriented terminology. The
square module exports two functions, called length
and area. Note that both the name and the total
number of parameters of a function are mentioned.
Erlang considers functions with the same name
but a different number of parameters as totally
different functions. All module code must be
included in functions.
Listing the available functions as well as other
useful information about a module can be done using
the following command:

1> square:module_info().
An alternative way of getting information about a
module is the following:

-import(square, [area/1]).
There is no command to import all functions at once
from an existing Erlang module. If a function of a
module is not used inside the module, you will get a
descriptive warning message.

09

Reading les in Erlang

This part will make the previously developed


square module able to read the contents of a text le.
Each line of the text le should be a number that will
represent the length of the squares side. The name
of the new module will be sqFile. Fig 01 shows the
added code.
The le:open built-in function (BIF) inside the
readlines() function opens the le for reading. Each
line is processed by the process_line function. Finally,
the save_line function prints some output on screen
using the io:fwrite() BIF to make sure that the le was
read correctly.
The io:fwrite() function is analogous to the printf()
function found in C. The trim_whitespace function
uses regular expressions to remove empty spaces
from its input.

10

Writing les in Erlang

This part will make changes to the previously


developed sqFile module to support the writing of
data to a new file. Each line will contain three numbers
separated by a space character: the side of the square,
the perimeter and the area of the square. The input
file a file called data will be used contains float
numbers. The filename used for saving the output will
be output its name is hard-coded. Opening a file for
writing is performed using the following line of code:

2> m(square).
{ok, FILE} = file:open("output", write)

08

Using the square module

If you look inside the directory where square.


erl resides, you will also see a binary le called
square.beam, which is the compiled module code in
BEAM format. You should compile your Erlang code
every time you make changes to the source code.
If you want to use the area function from the
square module in another module that you create,
you should include the following command inside
the new module:

You should not forget to close the le after all write


operations are done, by using the following line of
code:

file:close(FILE)
The actual writing of the output to the output le is
done using the following Erlang code:

io:format(File,"~.3f ~.3f ~.3f~n",[Number, L, A]).

Erlang programmers put their


11
code in modules. Modules help you
organise, store and share code easily
and efficiently

The Erlang Debugger

Erlang comes with a Debugger! You have to


follow a number of simple steps in the Erlang shell
in order to start the Debugger. For debugging the
sqFile module, you will have to execute the following
commands:

1> c(sqFile, [debug_info]).

114 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

{ok,sqFile}
2> im().
<0.42.0>
3> ii(sqFile).
{module,sqFile}
4> iaa([init]).
true
5> sqFile:readlines("data").
The rst command compiles the Erlang module and
adds some extra debugging information, which is a
common practice in every programming language.
The second command starts a new graphical
monitor that is the main Debugger window. The third
command interprets the code of the Erlang module
that is going to be examined. The fourth command
tells Erlang shell to automatically attach the
Debugger to any process that is executing interpreted
code when that process is started. The last command
executes a module function and makes the Debugger
start running.

12

Documenting Erlang code

Fig 01

readlines(FileName) ->
{ok, Device} = file:open(FileName, [read]),
try process_line(Device)
after file:close(Device)
end.
process_line(Device) ->
case io:get_line(Device, "") of
eof -> [];
Line -> save_line(Line),
Line ++ process_line(Device)
end.
save_line(Line) -> io:fwrite("**** ~s~n", [trim_whitespace(Line)]).
trim_whitespace(Input) -> re:replace(Input, "\\s+", "", [global]).

11

The Erlang
Debugger

The extra lines you are going to put in the


sqFile module for documenting your code do not
affect the functionality of the module. Erlang needs
them for generating the HTML output. Documenting
your code is an excellent practice that you should
always follow even if you think you do not have enough
time to write the few extra lines. In order to document
the mylength function, you can put the lines from Fig
02 before its implementation. And the lines in Fig 03
add general documentation for the entire module.
In order to generate the HTML files and put them in
a directory called documentation, you should execute
the following command in the Erlang shell:

edoc:files(["sqFile.erl"], [{dir,
"documentation"}]).
Remember that documentation is produced for
the functions that are in the exported list even if
you write documentation code (@doc) for other
functions.

13

Fig 02
%% @doc This function calculates the length of the perimeter of a square.
%% For a square with a side length of A, the length of its perimeter is 4 * A.
mylength(Side) -> 4 * Side.

Reading les in Erlang

The Erlang Process Manager is a useful


tool for keeping track of your processes and
understanding what is going on. You can start it using
the following command:

1> pman:start().
Everything that runs using Erlang, including
the Erlang shell, is a process and has a process
identier (pid) that you can see with the help of
the Process Manager. If you want to see more

Fig 03
%% @author Mihalis Tsoukalos <mihalistsoukalos@gmail.com>
[http://www.mtsoukalos.eu/]
%% @doc A module that calculates the perimeter and the area of a square.
%% It also writes the output to a file called "output".
%% @reference from <a href="http://www.linuxuser.co.uk">Linux User & Developer</a>.
%% 2014, Issue 137
%% @copyright 2014 Mihalis Tsoukalos
%% @version 1.23
%%

Linux & Open Source Genius Guide 115

Masterclass

information about a process, you can double-click


on it. If you want a shorter list of processes, you
should check the Hide System Processes option.
The Process Manager is a tool that you will
denitely need when you get more comfortable with
Erlang and OTP. During the learning process, it will
mostly help you discover what is going on behind the
curtains.

14

The serverLUD module


and its supervisor

The name of our module that uses OTP is serverLUD.


The commands in Fig 04 show you how to interact with
the serverLUD server process.
Now, it is time to write an Erlang program (module)
that will supervise the serverLUD server. Its name
will be simpleSUP. The important thing is that by
writing Erlang code to supervise the server, if the
serverLUD server goes down for some reason, the
supervisor will automatically start a new serverLUD
process, offering high availability to your Erlang
server which is the main reason that Erlang was
used at the Ericsson labs.
What you should keep in mind is that the
supervisor is responsible for the serverLUD server
without serverLUD knowing about it and without the
programmer having to change a single line of code
inside the serverLUD.erl le!

16

Reading les
in Erlang

Erlang OTP

OTP stands for Open Telecom Platform and


comprises Erlangs open source libraries and tools.
Despite its name, OTP is about much more than just
telephones! It is the most important and vital part
of Erlang. If half of Erlang's greatness comes from
its concurrency, distribution and error-handling
capabilities, then the OTP framework is the other half
of it.
OTP was designed for building projects with big
teams. It has been rewritten twice before, so it is
now in its third version. OTP is about taking all the
generic components, putting them into libraries,
making sure they work ne and reliably, and then
reusing that code when possible. So, what is left
for the programmer? The programmer has to deal
with things that will change from application to
application. If you are going to write real-world
Erlang software, you will eventually have to learn
the OTP framework.

15

13

Explaining the serverLUD


OTP module

If you are not familiar with Erlang code, you are likely
to find the code on both modules difficult to read and
understand.
Line 5 tells Erlang that the module will implement
the gen_server behaviour. A behaviour pre-denes

116 Linux & Open Source Genius Guide

the mechanisms you will use and interact with


processes. They are a way of formalising common
patterns. Line 6 denes a macro that tells the
Erlang compiler that every time it sees the word
?SERVER, it should replace it with ?MODULE. So,
macros are just simple text replacements, as in C.
In line 7, the count of the number of calls that were
made to the server is kept. In line 22, a function that
prints a message on screen is dened. In line 14,
the dened function calls the start_link function of
gen_server in order to start up the process.

17

tcp:send() function to actually send some data to the


TCP server. After finishing with the gen_tcp:send()
function, you should return to the TCP Server and run
the flush() command to display the received data.
To close the TCP server and client connections,
youll have to run the gen_tcp:close() function. The
f() function executed on the Erlang shell of the TCP
server tells the shell to delete and forget any bindings
it has.
Connecting using the UDP protocol is almost
identical you just have to use the gen_udp Erlang
module instead of the gen_tcp module.

Using TCP in Erlang

This part will show you how to create a simple


server that listens to a TCP port and interacts with a
TCP client using the Erlang shell. You will need two
Erlang shells: one shell for the TCP server and one for
the TCP client.
The TCP server is created using the
gen_tcp:listen() Erlang function. Next, you have to
set the socket to an accepting state by calling the
gen_tcp:accept() function. After executing the gen_
tcp:accept() function, the Erlang shell waits for a TCP
connection without allowing you to type any more
Erlang commands.
The TCP client tries to connect to the TCP server
by calling the gen_tcp:connect() function. When
specifying an IP address in Erlang, you should use
commas instead of dots to separate the four parts of
the IP address. After executing the gen_tcp:connect()
function, the gen_tcp:accept() function of the TCP
server returns. Next, you should execute the gen_

18

The Mnesia database

Mnesia is an extremely fast, distributed


DBMS written in Erlang that is part of the standard
Erlang distribution.
In order to create a new Mnesia database, you will
have to run the mnesia:create_schema() function.
The mnesia:create_schema([node()]) command
creates a Mnesia database in the current node
only because of the [node()] parameter. After the
mnesia:create_schema() function is executed,
Mnesia creates a new directory structure to host its
database.
Mnesia queries are similar to SQL and list
comprehensions. List Comprehensions is an Erlang
feature that allows you to create lists without having
to use some other language features (funs, maps or
lters).
Populating and querying an Mnesia database is
beyond the scope of this feature.

Linux & Open Source

Genius Guide

19

Multiple CPU cores and Erlang

As Steve Jobs once said, The way the


processor industry is going is to add more and more
cores, but nobody knows how to program those things.
I mean, two, yeah; four, not really; eight, forget it. The
good news is that Erlang supports multiple CPU cores
natively!
An Erlang program is implemented as a large
number of small processes that do simple tasks
and communicate with each other. Functional
Programming languages forbid code that has side
effects. Erlang processes interact with one method
by exchanging messages without sharing any data
with other processes, therefore allowing Erlang code
to utilise multiple CPU cores without problems.
You only have to make sure during the designing
phase of your Erlang application that it has lots of
processes that do not interfere with each other.
As a result of all the above, an Erlang program can
run four times faster on a quad-core CPU without
changing a single line of code!

20

What to do next?

One article cannot cover every aspect of a


programming language such as Erlang. You should
experiment and write many lines of Erlang code to learn
the Erlang way of thinking and programming. First,
learn how to use the Erlang shell well and then try to
write your own programs using your favourite editor and
compile them using erlc.
If you are really into Erlang and want to master
it, it would be wise to buy Armstrongs book Joe
Armstrong is the creator of Erlang.
Erlang will certainly change you as a programmer
and definitely make you look at programs in a totally
different way.

17

Using TCP
in Erlang

Fig 04

1> serverLUD:start_link(). %% start server


{ok,<0.34.0>}
%% run a server function
2> serverLUD:say_hi().
Hi Linux User and Developer!
ok
3> serverLUD:get_count(). %% get the number of connections so far
1
4> serverLUD:say_hi(). %% make another connection
Hi Linux User and Developer!
ok
5> serverLUD:get_count().
3
6> whereis(serverLUD). %% find the process ID of the process
<0.34.0>
7> serverLUD:get_count(). %% get the number of connections so far
4
8> serverLUD:stop().
%% stop the server
ok
9>
=INFO REPORT==== 25-Jan-2014::17:28:10 ===
terminating server
9> whereis(serverLUD).
undefined
10>

%% the server process is not running any more!

20

What to do
next?

QWritten by Erlang creator Joe Armstrong, this

is a highly recommended read

Linux & Open Source Genius Guide 117

Developer guide
Essential tools for coders
120 Write once, use anywhere with
Qt
Qt is the most flexible cross-platform
framework for C++ and can even create crossplatform GUIs. Learn how

124 Answer real-world questions


with Qt
Computer simulation can solve a variety of realworld problems, ranging from retail purchases
to trading card game modelling

142 Create a real-time CPU meter


with Node.js and Duino
Get a Node.js-powered Arduino to display
the current processor usage of a computer in
real-time

146 Build tic-tac-toe with Kivy


Ease into the workings of Kivy by creating this
pen-and-paper classic in just over 100 lines
of Python

150 Get started with Gulp.js


Explore the many benefits of using the
Gulp.js task runner in your web projects
to help streamline your workflow and
optimise your code

154 Master Vim in easy steps


Take a step ahead, learn some advanced
features and get more productive with the
Vim editor

128 Handle Twitters three-legged


oAuth process
Building and signing requests according
to OAuth can be tricky. We explain the
authentication process for Python and Flask

132 Create dynamic templates with


Flask, Jinja2 and Twitter
We create a dynamic webpage with Twitter and
Flasks rendering engine, Jinja2

136 Build a blog with Django


Learn how to program and publish a Django
photo blog site in easy steps

132

118 Linux & Open Source Genius Guide

146

Linux & Open Source

Genius Guide

Linux & Open Source Genius Guide 119

Developer guide

Write once, use


anywhere with Qt
Advisor
Tam Hanna has been in
the IT business since the days
of the Palm IIIc. Serving as
journalist, tutor, speaker and
author of scientic books, he has
seen every aspect of the mobile
market more than once

Further reading...
planet.qt-project.org
qt-project.org/doc
qt-project.org/forums

Qt is the most exible cross-platform


framework for C++ and can even create
cross-platform GUIs. Heres how
During the initial development of C,
creating GUI applications was considered
a non-use case: it was developed on
a computer system attached to nongraphical dumb terminals. The average developer
spent his time churning away on engine code which,
if designed correctly, would result in an easily portable
application.
As time went by, operating system vendors
extended C/C++ with a variety of custom libraries.
This led to an ever-decreasing level of portability:
apps for Windows, Linux and Symbian are built with
C++, but the non-domain-level code cannot be
shared between platforms. This ired two Norwegian

QAfter startup, Qt Creator shows a start pane with some instructions for getting started

120 Linux & Open Source Genius Guide

programmers, who started to develop a product


called Quasar toolkit (later known as Qt). Work on the
framework started in 1991 and the product managed
to achieve a cult following. The tremendous success
of the Qt-based KDE framework led to concerns
about the closed nature of the framework.
Qts creators Trolltech addressed this issue with
the introduction of a dual-licensing scheme. From
that date onward, the framework was provided in two
forms. Commercial users could buy the code, which
permitted them to keep changes to themselves, while
a GPLd version also became available.
In addition to that, a legal trapdoor was
established to protect the availability. It is

Linux & Open Source

Genius Guide

Installing Qt
Getting Qt up and running is really easy

QAfter starting a project and choosing where to

store it, select Desktop on this screen


responsible for the ood of updates which have
shaped the frameworks history: if the current
commercial maintainer does not update the free
version at least once a year, it is automatically
placed under a very liberal open source licence.
Nokia purchased the framework with the intent
to unify its various developer offerings. This project
came to nothing. Qt is now owned by an IT consulting
company named Digia. Even though the free licence
has long been replaced by the LGPL, a commercial
version remains available. Ports for iOS and
Android are in development Qt is available almost
everywhere.

IDE la Qt
Digia provides users of the framework with a custom
IDE. Qt Creator works best when run in superuser
mode. Start it by entering the following commands
into a terminal hosted in /opt/Qt/ the soft asserts
emitted are normal and do not point to mistakes in
conguration:

Due to the ubiquity of the Qt framework,


most Linux distributions contain a package
containing the aforementioned IDE. However,
this version tends to be well out of date the
safest and sanest approach is to obtain the
latest version directly from the Digia download
servers. As of this writing, the current version
is 5.1.1 the following steps are based on it.
The easiest way to install Qt can be had by
visiting the URL qt-project.org/downloads.
Then, click the option for the Qt Online
Installer which best ts your system. After
the download is complete, open a terminal
and execute the following commands:

sudo qtcreator
SOFT ASSERT: "i != -1" in
file ../../../../src/plugins/
projectexplorer/taskmodel.cpp, line 172
Qt Creator has a slightly unusual layout, which can
best be understood by looking at the GUI in its default
state. The IDE is subdivided into multiple submodules,
which can be activated via the ribbon on the left-hand
side of the screen. Click any of the icons to activate the
corresponding view it will show up in the remainder
of the GUI. The bar at the bottom allows you to nd
out further information about compile processes and
application status by default, it is hidden to save
screen real estate.

sudo chmod u+x qt-linux-opensource1.4.0-2-x86-online.run


sudo ./qt-linux-opensource-1.4.0-2x86-online.run
This will then start the automatic installer.
It will provide sensible default values for
all settings for the average developer,
just clicking Next repeatedly will provide a
workable installation.
The download itself can take up to 30
minutes, during which you can continue to use
your system normally.

First steps
It is now time for us to start working on a little
Hello world application. Click File>New File or
Project in order to open the Project Creation
wizard. Then, select Applications>Qt Gui
Application and click Choose. In the next step,
the product will ask for a name and a storage
location.
After that, a relatively important dialog shows
up. Qt Creator contains a relatively advanced crosscompilation framework, which permits you to target
multiple different versions and platforms with one
click. For our basic project, just select Desktop we
will use this dialog more in later parts of the tutorial.
The Class Information dialog is not signicant and
can be closed by clicking Next. Then, click Finish to
start the generation of the project skeleton. Then,
click Play to start the program it will display the
project skeleton in Edit mode.

A new build process

QQt Creator supports a large variety of project types

When clicking the Play button at the bottom-left


corner, the framework compiles and runs the
application. In the background, a quite complex
process takes place understanding it is paramount
to using Qt successfully.
A Qt-based application is compiled by the native
toolchain intended for the operating system. The
process is controlled by makeles which, however,
are generated automatically by a tool called qmake.
It analyses the .pro-le of a solution and uses that
information to set up the toolchain.
This cumbersome-sounding process offers a
signicant benet to the developer: as the .cpp
les are processed by the native compiler, they can
contain any kind of code which the native operating
system understands. It is fully possible to embed
a native API call into an event handler or a similar
portion of code if you do not need

Linux & Open Source Genius Guide 121

Developer guide

Going
cross-platform
Taking our application to
another operating system
is really easy

QQtGui-based forms can be edited using a WYSIWYG editor

cross-platform compatibility, there is no need


to provide protection around the native code.
Should you ever want to port your app to another
platform, just use the C preprocessor to blot out the
problematic code section.
The .pro le of our example can be opened by
clicking it in the le list on the left side of the screen. It
will look like this:

QT
+= core gui
greaterThan(QT_MAJOR_VERSION,
widgets
TARGET = untitled
TEMPLATE = app
SOURCES += main.cpp\
mainwindow.cpp
HEADERS += mainwindow.h
FORMS
+= mainwindow.ui

4):

QT

+=

In principle, a .pro le contains a list of les and


settings which are needed to build the product.
Qt is made up of multiple modules including the
ones needed via the QT-Statement. The target- and
template-Statements determine the type of binary
which is to be generated. The HEADERS-, SOURCESand FORMS-Blocks list the les of each type. When a
new le is added, Qt Creator updates these sections
automatically.

122 Linux & Open Source Genius Guide

GUI from file


Qt currently contains two different GUI stacks, which
can be used co-operatively. QtGui (Qt 5 renamed this
module to widgets) uses a group of controls mapped
to the native operating system. Qt 4.7 introduced
a second system called QML, which uses a JSONderived language to describe the modules making up
the user interface.
Even though rumours pertaining to the death of
QtGui tend to pop up every now and then, Digia has
repeatedly conrmed that it is strongly committed to
keeping the system around for the foreseeable future.
Using QtGui tends to bring results quicker. As the
base project we created was based on QtGui, we will
continue to use it during the rest of the tutorial.
QtGui-based forms are created entirely in source
code. However, Qt Creator provides a WYSIWYG
front-end which simplies the management of the
user interface. Double-click any .ui le in the project
overview in order to activate the design view, as
pictured above.

Currently, we have no non-framework code,


so there are no obstacles from a code point of
view. Even though Qt Creator contains decent
cross-platform capability, it tends to be limited
to compiling within the same platform. This
means that a UNIX-like system could easily
create a binary for a different UNIX-like system
running on a different processor type (producing
an executable le for Windows requires
considerable effort).
When targeting a different desktop operating
system, bringing up an instance and installing Qt
Creator, there is the fastest way. Digia provides
complete installers for Windows, Mac OS and
various avours of UNIX. Just copy the project
les over, open the .pro le and start a compile
process.
The situation in the mobile space is a bit
different. Here, cross-compilation is a matter
of necessity. The choice of desktop operating
system is sometimes made by the manufacturer
of the device in question. BlackBerry 10 and
Android can be programmed with any desktop
operating system, whereas Windows Phone
and iOS tend to require a device running the
corresponding desktop operating system.

The toolbar on the left of the screen contains a


variety of controls, which can be dragged onto the
form in the middle. The list at the bottom right shows
the properties of the currently selected control.
The tree view above it provides an overview of the
structure of the user interface.
During compilation, a program called uic (User
Interface Compiler) will transform the .ui les into
header les containing the code needed to generate
the actual form. In the case of our skeleton program,
the le will be called ui_mainwindow.h (it can be
found by opening the le mainwindow.h, rightclicking the relevant inclusion and then instructing Qt
Creator to follow the declaration of the symbol).

Qt Creator provides a WYSIWYG frontend which simplies the management of


the user interface

Linux & Open Source

Genius Guide

In addition to that, this listing also demonstrates


the use of the Q_OBJECT macro. It is required in all
classes which are derived from a QObject-derived
class its use will be discussed in the next part of
the series. For now, it sufces to say that it must be
there to avoid compiler errors.

#ifndef MAINWINDOW_H
#define MAINWINDOW_H
#include <QMainWindow>
namespace Ui {
class MainWindow;
}
class MainWindow : public QMainWindow
{
Q_OBJECT

QThe source code of main.cpp is shown with syntax highlighting

Its structure will be similar to this code snippet:

class Ui_MainWindow
{
public:
QMenuBar *menuBar;
QToolBar *mainToolBar;
QWidget *centralWidget;
QStatusBar *statusBar;
void setupUi(QMainWindow *MainWindow)
{
if (MainWindow->objectName().
isEmpty())
MainWindow->setObjectName(QString
::fromUtf8("MainWindow"));
MainWindow->resize(400, 300);
menuBar = new QMenuBar(MainWindow);
...

pushed onto the screen. Finally, exec() is invoked: it


will return only when the program is to be terminated:

#include <QApplication>
#include "mainwindow.h"
int main(int argc, char *argv[])
{
QApplication a(argc, argv);
MainWindow w;
w.show();

public:
explicit MainWindow(QWidget *parent = 0);
~MainWindow();
private:
Ui::MainWindow *ui;
};
#endif // MAINWINDOW_H

return a.exec();
}
The form itself is made up of two les. The c++ class
contains a more-or-less default constructor which
invokes the setupUi method of the ui object. Other
than that, it does not contain much of signicance:

QVarious les are produced during the

compile process

Perfect for every scenario


ui_*.h les are generated automatically whenever
the corresponding .ui le changes. Therefore, it is not
recommended to modify them by hand in any way;
doing so will lead to unreliable and undened project
behaviour. Furthermore, it is recommended to
keep them (and any moc les) out of version control
systems: the frequent changes tend to overload the
servers quickly.

Coding, for real


Now it is time to take a look at the source code. The
entry point is in a le called main.cpp. The method
found there is a standard construct. It starts out by
instantiating the QApplication class which contains
the event loop and other elements needed for
program operation and OS integration. In the next
step, an instance of the MainWindow is created and

#include "mainwindow.h"
#include "ui_mainwindow.h"
MainWindow::MainWindow(QWidget *parent) :
QMainWindow(parent),
ui(new Ui::MainWindow)
{
ui->setupUi(this);
}
MainWindow::~MainWindow()
{
delete ui;
}
The actual intelligence of the form resides in the header
le. Its derived from QMainWindow, and contains a
pointer to the aforementioned ui object. It is created
by uic, and resides in the aformentioned ui_*.h le.

From a commercial point of view, using Qt is a


no-brainer. Being able to port your application
quickly lets you reach new customer groups
with little effort. Should the need arise to bring
your product to a mobile operating system, the
framework is already there.
Using Qt, however, also makes sense for a
non-commercial application which will not be
ported anywhere. Qts compact and expressive
interfaces minimise the time needed for
implementation; features such as the signalslot system and the garbage collector ease
development.
In conclusion, it can be said that learning and
using Qt always makes sense. Once the learning
curve has been climbed, enormous time savings
can be achieved.

Linux & Open Source Genius Guide 123

Developer guide

Answer real-world
questions with Qt
Advisor
Tam Hanna has followed the
mobile phone market since the year
2004. His rm Tamoggemon Holding
k.s. uses Qt to maximise code reuse
across different mobile and desktop
operating systems.

Further reading...
planet.qt-project.org
qt-project.org/doc
qt-project.org/forums
blog.qt.digia.com

Computer simulation can solve a variety


of real-world problems, ranging from retail
purchases to trading card game modelling
Did you ever wonder how many booster
packs must be bought to have one copy
of each card in a set? Did you baulk at
purchasing expensive stockings due to
the high total cost of ownership? Did you ever feel
overwhelmed by analysing the impact of a nancial
decision you have to make?
Even though most problems can be solved by
trying to gure out the mathematical formulas,
brute-force simulations can bring useful results far
more quickly. The method outlined in this article can
be used for all problems, which can be broken down
into a group of easy-to-analyse passes. Running
enough of them leads to a result eld, which can then
be analysed further.
This story was inspired by a promotional
campaign run by Slovakias Tesco retailer. The
company gave its customers a domino gem for
each purchase. Each of the stones was imprinted
with a picture of a different smurf character.
Customers were encouraged to complete an
album by nding one copy of each of the stones a
task made more complex as the packages did not
reveal their contents until after they were opened.
But how many packages would need to be
purchased in order to have a realistic shot at
having a full, completed album?

RNG-Output

Assigned
elements

QIn this case, the first three elements get more

random ones than the fourth one. This causes


an imbalance in the result

124 Linux & Open Source Genius Guide

Slot for elements

Let us start our observations by looking at the


situation of a well-off Slovak who enters his Tesco
store to pick up his rst stone. Each one of the 36
available pictures is new, as our friends collection
is completely devoid of smurf dominoes at this point
of time.
The situation looks far less satisfactory on the
second run to the supermarket. As the collection
now contains one stone, the chance for nding a
new one is now reduced to 35/36ths. As more and
more stones get collected, returns diminish: more
stones reduces the odds for nding a new one.

Well-off computers
Modelling this situation on a computer is
comparatively easy. The collection of each individual
can be represented by an array of 36 booleans: if one
of them is false, the element in question is not yet
found in the users collection.
The actual simulation is performed by using the
random-number generator to pick a domino stone.
Its ID is compared against the collection in order
to determine if the stone is already there if it is
not there yet, the ags state is changed and the
number of missing stones is reduced by one.
A loop is used to keep adding stones until
all stones are found in the collection. Counting

Linux & Open Source

Genius Guide

the number of iterations will inform you about


the amount of packages that are needed by the
individual in question.
We will start our work by implementing a single
pass. It shall be encapsulated in a method that
returns the number of attempts needed to its
caller, as follows:

int MainWindow::runAttempt()
{
//Reset
int missing=36;
int attempts=0;
int whatDoIHave[36];
for(int i=0;i<36;i++)whatDoIHave[i]=0;
Processing begins with the declaration of the
variables needed for storage. Using method-local
storages is helpful when it comes to creating multithreaded simulations, as there is less potential for
race conditions.
In the next step, we commence with the actual
population process. It is implemented in the form of
a while loop, which runs until the amount of missing
stoned reaches zero:

Multithreading and race conditions


Speeding up simulations by executing multiple runs
in parallel
In the last few years, thermal and physical
limits have set constraints on the maximal
performance achievable from a single
core. Chip vendors have approached this
problem by creating multi core processors
containing multiple cores on a single die.
Developers can use these systems to
run a group of threads in parallel. If they
are designed correctly, tremendous speed
increases can result some algorithms
scale in an almost ideal fashion.
As long as the individual tasks are
independent from one another, no weird
behaviour will occur. Sadly, practical
programs tend to suffer from race

condition, which happens whenever two


threads access the same location in
memory at the same time.
In this case, the local value stored in one
thread will overwrite the results produced
by another one. The collection process
would then end up with wrong data, which,
in the worst case, could cause a crash.
This situation can be remedied by
using Mutexes and Semaphores. They
ensure that the threads do not access the
same sequence at the same time. Sadly,
creating reliable multithreaded code
tends to be complex please consult the
corresponding literature.

//Start to populate
while(missing!=0)
{
attempts++;
The individual attempts are modelled by invoking the
qrand function. It returns a random integer ranging
from zero to the highest value of the random number
generator. Its value can be found by looking at the
RAND_MAX constant, which is declared in stdlib.h.
For our example, we are satised by knowing that
the highest number returned is much larger than
the number range used for modelling. If the ranges
were to end up being close to one another, you
might nd that mapping difculties could occur.
We process the returned number via the modulo
operator. The resulting value is then used as index to
the array if the element is not yet set, the missing
counter is decreased accordingly:

int myResult= qrand() % ((35 + 1));


if(whatDoIHave[myResult]==0)
{
whatDoIHave[myResult]=1;
missing--;
}
}
return attempts;
}
With that, the code is done.

QGauss Curves describe randomised processesimbalance in the result


Feel free to run the code from a command line
or GUI. Each invocation of the method will return a
different number, determined by the sequence of the
random numbers returned by the RNG.

Increase accuracy by
collating results
Colleges and schools tend to provide mathematical
problems which have one or, in the worst case a
small amount of permissible and correct results. As
our simulation is based on a random process, it can
return a large variety of different and valid results. An

insanely lucky Slovak might nd a new gem on each


purchase, whereas a very, very unlucky one might
purchase a million packages and still end up with
missing ones.
Processes that are by and large dependent on
random subhappenings tend to align themselves in a
pattern described as Gaussian standard deviation,
as can be seen displayed in the above graph.
Our program will return the data needed to
determine the various supporting points of the
diagram. This can be achieved by running the same
process over and over again until a large enough

Linux & Open Source Genius Guide 125

Developer guide

sample of data has been collected. Most softwarebased random number generators need a starting
value, which initialises the numerical processes
contained. It is customary to pass in the current time
in a production environment.

QTime time = QTime::currentTime();


qsrand((uint)time.msec());
In the next step, a binning array is created. Their
role can be difcult to grasp on the rst glance.
The table below contains a schematic description,
which should be consulted as you work through the
following example.
Each attempt is classied by the amount of
purchases needed to complete the collection. We
band these up so we end up with a group of slots
of ten each. The binning array is then made up of a
group of ints that count the amount of occurrences
of each band. For example, the slot zero would
contain a count of the runs where zero to nine
purchases were needed.
Our routine uses a total of 27 different slots,
handling a range of ten each. So, the zeroth element
of the array would be responsible for all iterations
ranging from zero to nine, and onward. Here, the
upper level was determined by good luck higher
results are collated into the highest element via an if
clause as follows:

int myBinningArray[27];
for(int i=0;i<27;i++)

myBinningArray[i]=0;
for(int i=0;i<50000;i++)
{
int idx=runAttempt()/10;
if (idx>=27)idx=27;
myBinningArray[idx]++;
}
Finally, data must be normalised in order to simplify
the processing. This is achieved by dividing the
number of iterations seen by each slot of the bin
with the total. The result is then multiplied with 100
to achieve the percentage of iterations that fell in the
bucket in question:

for(int i=0;i<27;i++)
qDebug() << (float)myBinningArray
[i]/(float)50000*100;
Divisions tend to require quite a bit of computing
power. Due to that, C treats all division operations
involving two integers as int the normalisation
process above would not work out, as the result of
the division would be rounded to either zero or one.
Casting the two parameters to oat ensures that the
arithmetical operation will be carried out with oat
data types, thereby preserving accuracy.
The example that we have used dumps the
computed results into the command line. You can
copy them into a spreadsheet such as the one
included in LibreOfce. This method was used to
generate the chart shown in gure ve it is clear

that customers must expect to buy a lot of packages


before their collection is complete.

How many iterations?


Our example analyses a total of 50,000 individual
ows. Determining the ideal amount of iterations is
a science of its own there is no universal principle
governing the number of runs needed to have a
satisfactory result.
Due to the tremendous power of todays
computer systems, most real-life problems can be
overanalysed in a few seconds worth of processing
time. Letting the example run for a minute or so will
usually lead to an amount of data so large that the
results can be considered statistically signicant.
Looking carefully at the graph below reveals that
the diagram contains three different coloured lines.
The bright blue one was achieved by analysing 500
iterations, whereas the yellow one shows the result
of 50,000 runs.
In principle, you can be satised with the results
of the analysis if the values start to converge and
change less as the number of runs increases.

Stockings galore
The methods outlined here can be applied to a
wide range of real-life problems. One amusing
problem solved recently involved stockings.
Purchasing a single pair of expensive stockings
is perceived as a waste of money, as a ladder in
one of the stockings would render the other one
completely useless. Purchasing multiple packs of

...
20-29
10-19
0-9
QA binning array is made up of

multiple slots for data

126 Linux & Open Source Genius Guide

Q Customers must expect to purchase a large number of gems to get a complete set

Linux & Open Source

Genius Guide

hosiery will reduce the impact of a single ladder.


However, it also increases the amount of money
bound in underwear.
Before getting into the actual model, we must
clarify one thing in advance. The results of this
program will not return an ideal value deciding on
the amount of stockings to buy. Instead, the data
used will serve as input into the decision process
made by the owner.
Lets set out on modelling the actual problem at
hand. For our program here, a transaction can be
started as long as two intact stockings can be put
to use. They will be assumed to break down with a
random probability of three per cent when worn
aging will not be a factor for brevity reasons:

int MainWindow::modelStockingWearing()
{
int lostStockings=0;
//Left stocking
int myResult= qrand() % ((100 + 1));
if(myResult<=3)lostStockings++;
//Right stocking
myResult= qrand() % ((100+1));
if(myResult<=3)lostStockings++;
return lostStockings;
}
modelStockingWearing is a relatively simple
function. It invokes the random number generator
twice in order to gure out whether the left and/or
the right stocking was damaged during the outing.
This result can then be processed further in a
mother routine. See Fig 01.
In principle, this method is easy. It runs the
modelStockingWearing-Function
until
the
number of remaining stockings falls below two.
The amount of possible iterations is then dumped
into the command line for further processing. See
Fig 02.
The improved version of the harness routine runs
the internal test a thousand times. It then divides the
counter by thousand in order to receive the average
value per iteration.
A chart with the results of our computation (right)
shows that the price per wearing session does
not decrease signicantly when more than four
packages are purchased in one sitting.

QTime time = QTime::currentTime();


qsrand((uint)time.msec());
for(int numberOfPacks = 1; numberOfPacks < 11; numberOfPacks++)
{
int stockingsInStore=numberOfPacks*2;
int counter=0;
while(stockingsInStore>=2)
{
counter++;
stockingsInStore -= modelStockingWearing();
}
qDebug() << numberOfPacks << " " << counter;
}

Fig 01

for(int numberOfPacks = 1; numberOfPacks < 11; numberOfPacks++)


{
int counter=0;
for(int i=0;i<1000;i++)
{
int stockingsInStore=numberOfPacks*2;
while(stockingsInStore>=2)
{
counter++;
stockingsInStore -= modelStockingWearing();
}
}
qDebug() << numberOfPacks << " " << counter/1000;
}

Fig 02

Conclusion
While using computers to solve non-virtual
problems by brute force is a process that is not
currently perceived as acceptable and this story
glosses over some mathematical details the
results returned by a modelling system based on the
methods described tend to be accurate enough for
everyday use.

QThe price of each wearing stops to reduce after the fourth package was bought

Linux & Open Source Genius Guide 127

Developer guide

Handle Twitters
three-legged OAuth process
Building and signing requests according to OAuth can be tricky.
We explain the authentication process for Python and Flask
Advisor

Sean M Tracey is a
creative technologist at a
leading digital agency. He
spends a lot of his time living
inside of Node.js, Python
and Arduino

Resources
Python 2.7+
Flask 0.10.0:
ask.pocoo.org

Flask GitHub:
github.com/mitsuhiko/ask

A Twitter account
Your favourite text editor

QThe OAuth Settings page for your new Twitter app youll fill in various bits as you go
Once upon a time, Twitter was a favourite
API for developers looking to write a
quick weekend hack. Setup was quick,
documentation was plentiful and the
data available was vast. Fun times! But now things
are a little different. With the Twitter API v1.1,
authentication is not quite as easy as it once was
(though the documentation is still plentiful and
the data vast). Twitter now requires adherence
to OAuth protocol v1 for all requests made to its
API. Its tempting to install a library and ignore all
the complicated stuff and entirely appropriate
if youre putting together a quick hack project.
However, understanding whats going on in the
authentication process will help you along when it
comes to debugging problems you may encounter
later if you have to start handling multiple users or
requests. For this tutorial, well be using Flask a

128 Linux & Open Source Genius Guide

great, lightweight Python server framework to dene


routes, create sessions and handle parameters
passed to and from Twitter. Flask is perfect for this
project: as well see shortly, dening routes and
handling parameters is a piece of cake and saves us
having to deal with a lot of the overheads that come
with frameworks such as Django.

01

Install Flask

So, before we start writing code, lets deal


with the prerequisites. If you have pip on your system
then installing is as simple as entering sudo pip
install flask into your terminal. If youre into
building from source, you can head over to Flasks
GitHub and grab the latest version there.

02

Create a Twitter app


Before we can do anything with Twitter, we

need to create an app for their service. Go to https://


dev.twitter.com and sign in. If you look in the topright corner, youll see your Twitter prole picture;
hover over it and a drop-down will appear. Select My
Apps and then click the Create new app button that
appears on the next screen. Follow the instructions
to set up the app. The information you enter here
isnt important right now at the end of the process
youll see a eld titled OAuth Settings with a table of
settings. Well need those in a little bit.

03

Importing Python modules

These are the modules well be using for


this project. With the exception of Flask, all of the
modules should be included with all versions of
Python 2.7+. These modules are all used in the
process of generating, signing and encoding each
request we send to and from Twitter in the auth

Linux & Open Source

Genius Guide

process. We need to be specic when importing


sessions, requests etc from Flask, otherwise
those aspects of the library wont be included in
the scope of the endpoints.

Full code
avaliable online:
http://bit.
ly/1gH6W0S

flask.py (cont. on page 131)


import urllib, urllib2, collections, hmac, binascii, time, random

04

Create global variables


for Twitter keys

Back in step 2 we created an app with Twitter.


Now we need those variables that we saw
in the OAuth Settings area of the Create
my app page. Fill in the consumer_key and
consumer_secret variables. For now, leave the
oauth_secret and oauth_token variables as blank
strings well assign those later.

With the
Twitter API v1.1,
authentication is
not quite as easy
as it once was
05
Assigning a namespace
to the Flask app

Now we create an instance of the Flask class. This


variable will be how we access all of the classes and
properties associated with the Flask package. The
__name__ property in this instance gives our Flask
class an idea of the scope its operating in.

06

Creating our rst route

In order to authorise our user, we need


to create an endpoint that they can visit to trigger
the process. This is where Flask comes in handy.
Using the routes decorator (@app.route()) we can
determine paths that trigger functions or actions.
The rst endpoint we want to create is @app.
route(authenticate). The function we declare
immediately after the decorator will be executed
when that endpoint is called. Note that although
weve done so here, you do not need to name the
function the same as the endpoint. Flask can
deliver static les as well as use template rendering
engines, but for the sake of brevity were going to use
return at the end of the function to deliver a simple
text response showing our progress.

from hashlib import sha1


from flask import Flask, redirect, request, session
#Twitter OAuth Variables
consumer_key = Srq7dzdIDule3RxSz9Ew
consumer_secret = Bsurvs8Bk41VHjhLWzZCUHvZw9btpdxNZVgxjG8b0
app = Flask(__name__)
@app.route(/authenticate)
def authenticate():
session.clear()
session[oauth_secret] =
requestParams = {
oauth_callback : http://192.168.1.5:5000/authorised,
oauth_consumer_key : consumer_key,
oauth_nonce : str(random.randint(1, 999999999)),
oauth_signature_method : HMAC-SHA1,
oauth_timestamp : int(time.time()),
oauth_version : 1.0
}
theSig = sign_request(requestParams, POST, https://api.twitter.com/oauth/
request_token)
requestParams[oauth_signature] = theSig
request = urllib2.Request(https://api.twitter.com/oauth/request_token, )
request.add_header(Authorization, create_oauth_headers(requestParams))
try:
httpResponse = urllib2.urlopen(request)
except urllib2.HTTPError, e:
return e.read()
responseData = getParameters(httpResponse.read())
session[oauth_token] = responseData[oauth_token]
session[oauth_secret] = responseData[oauth_token_secret]
return redirect(https://api.twitter.com/oauth/authorize?oauth_token= +
session[oauth_token])
@app.route(/authorised)
def authorised():
if request.args.get(oauth_token, ) == session[oauth_token]:
verifyRequestParams = {
oauth_consumer_key : consumer_key,
oauth_nonce : str(random.randint(1, 999999999)),
oauth_signature_method : HMAC-SHA1,
oauth_timestamp : int(time.time()),
oauth_version : 1.0,
oauth_token : session[oauth_token]
}
signVerification = sign_request(verifyRequestParams, POST, https://api.
twitter.com/oauth/access_token)
verifyRequestParams[oauth_signature] = signVerification

07

Creating a session with Flask

In order to maintain various properties


and values specic to multiple users, were going
to be storing our values in Flask sessions. A Flask
session is almost exactly the same as using a

verifyRequest = urllib2.Request(https://api.twitter.com/oauth/access_token,
oauth_verifier= + request.args.get(oauth_verifier))
verifyRequest.add_header(Authorization, create_oauth_

Linux & Open Source Genius Guide 129

Developer guide

cookie to store values, but with sessions we get a


layer of cryptography to help keep user credentials
secure. The values in our sessions behave just like
a dictionary, so we can call all of the methods that
dictionaries have. For example, we can call session.
clear() if we want to log a user out of a service or
restart the authentication process. We need to make
sure that before we make any calls to our sessions,
weve set our session key, otherwise well get an
error. In this example weve chosen to insert the key
just before app.run() so it will be set just before our
server is executed. Like most keys, you must keep
this key secret.

08

Preparing to authorise our app

Now we need to start thinking about how


we are going to submit data to Twitter. The OAuth
protocol expects every request to be signed with a
SHA1 hash so Twitter can check the integrity of the
data it has received before allowing access to any
personal information.
In order to sign the request properly, we need to
order the parameters of the request alphabetically,
URL-encode them, concatenate them and then sign
against them each time. This is where the headache
of OAuth begins. Because well be doing this for most
every request, were going to write a convenience
function that does all of this for us: sign_request().

09

Creating our signature

So, whats going on in our sign_request()


function? First, we need to URL-encode the
base URL of the request, which we can do with
urllib.quote(). Next, we create an ordered dictionary
(OrderedDict) which sorts the parameters weve
passed through to the function alphabetically
according to the key value. This is crucial as each
request must be signed with all of the parameters
concatenated alphabetically in order to be valid.
Next, we create the variable requestString to build

A little about
oauth_nonce
Most of the values included in the OAuth
headers and signing process are selfexplanatory. However, one isnt so obvious
in its purpose. OAuth_Nonce is a randomly
generated value often alphanumeric
but as there is no dened convention, an
integer between 1 and something large
will serve its purpose. It exists so Twitter
can check that its not been sent the same
request twice.

130 Linux & Open Source Genius Guide

Flask is perfect for this project:


defining routes and handling
parameters is a piece of cake
the rst part of our request body. This is made up
of the method well be using for the request and the
URL well be submitting to. We then iterate through
the keys in the OrderedDict, concatenating and URLencoding the key value pairs of the dictionary so we
have something like key=value. Finally, we check
whether or not were at the last key in the dictionary;
if we arent then well append a & to the parameter
string in preperation for the next one to follow.
Ultimately, well have a string that will appear as
GET&http%3A%2F%2Fapi.twitter.com%2Fetc%2F
?key1=value1&key2=value2&key3=value3.

10

The signing key

11

Hashing the key

Now we need to create a signing key


for our signature. As the code shows, its just
our consumer_secret + oauth_secret with an
ampersand in the middle. However, you may
notice that we dont have an OAuth secret yet.
Thats okay: for this rst signing request we
dont need one. You may have noticed in our
/authenticate endpoint that after we popped the
session values, we reassigned session[oauth_
secret] to . This way we wont get a KeyError
message when we try to sign our request. When
we get our callback in our /authorised endpoint,
well assign our session[oauth_secret] for
future requests.

Finally, were ready to hash the key. Using


Pythons HMAC library, we pass our signing key,
our result string (the concatenation of our escaped
base URL and method with our parameters string)
and from Pythons hashlib library, the SHA1 hash
function. This results in a binary buffer with our
signature thats great, but we cant send a buffer to
Twitters server. On the next line, we pass our hashed
variable through to the binascii library to convert it
to a Base64 value. Now we have our signature which
well return to the function that called it.

12

Obtaining a request token

With Twitter API v1.1, we can no longer just


request an access token to access user data.
First, we need to obtain a request token and then
exchange that for our rst access token. We dont
need the user to interact with anything at this point,
so we can trigger this on the back-end and handle
it ourselves. In the requestParams dictionary, we

prepare the values well need to sign against and


then submit these to Twitter to receive a request
token. The oauth_callback eld wont do anything
right now, but its where our app will redirect after
we obtain our rst access token. We want to point
it to our route /authorised. On line 27 we pass
the requestParams through to the sign_request
method. We then add the signature returned to the
requestParameters that passed through to the
signing function, as well be using these to build our
authorisation headers next.

13

Building the OAuth


authorisation headers

14

Making the rst request

15

Redirecting the user

Much like when we sign our requests, we need


to properly encode and format our authorisation
headers, too. The main difference between
sign_request() and create_oauth_headers() is
that the signature base string (the string we build to
generate our SHA1 signature) is all of the variables
joined with &, whereas our auth headers are joined
with , and prepended with the word Auth. Other
than that, the two serve the same purpose. Here
weve separated the two out to emphasise the
importance of the difference if either of these two
functions returns a malformed string, the request
will fail. Using the urllib2 module, we can prepare the
request well submit. Notice the second parameter
of the request function on line 31. If we omit this
parameter, the request method will be GET; if it is
included regardless of the content it will be a
POST request. The parameters main function is to
assign the body property of a request; its a little
clumsy, but its something well need in a bit.

Once weve gotten the prepared auth


headers back, we add them to the HTTP request and
then call the open() method and wait for a response.
If all has gone well, we should receive a string in the
request response with our rst auth token and auth
secret. To save us having to individually parse the
string for properties, the function getParameters()
will split the string up and return a dictionary we can
access. Well then store these values in our session
for later.

Now we have our rst set of credentials, we


can redirect our user to the Twitter authorisations

Linux & Open Source

Genius Guide

screen to let them grant access to their data.


Because we havent sent anything to our client yet,
we can use Flasks redirect() method to send our
user to the Twitter authorisation view. This request
is one of the few that we can use URL parameters to
send tokens to Twitter.

Full code
avaliable online:
http://bit.
ly/1gH6W0S

headers(verifyRequestParams))
try:
httpResponse = urllib2.urlopen(verifyRequest)
except urllib2.HTTPError, e:
return e.read()
responseData = getParameters(httpResponse.read())

16

Handling access

If the user grants our application access,


they will be redirected to the callback URL we
dened in step 12. The redirection will include URL
parameters that are our access token and a token
verier string. The Twitter documents state that
we should check the access token we received
in the URL parameters is the same as the one we
obtained moments ago. If all has gone well, Our /
authenticate endpoint will have stored our token
in the session dictionary. If our stored token and
the one Twitter returned matches, we can prepare
to exchange our access token (which currently only
has restricted privileges) for a fully edged access
token with read/write permissions (depending
on the permissions you requested when you rst
created the app).

17

Third (and nal)


token exchange

So, same as before: we prepare our parameters


for submission with sign_request(), build our
auth headers and then re off our request. Just
like when we rst retrieved our request token, this
is something that we can do in the background
without user interaction. Providing everything has
gone swimmingly, you will then be returned a fully
authorised access token that you can start building
your own requests with.

18

Fire up your server

As you can see, we have ended our ask.py


le with:

if __name__ == __main__:
app.secret_key = 
R4nDOMCRypt0gr4ph1cK3yf0R5355i0N
app.run(host=0.0.0.0, debug=True)
If we didnt include any parameters in the run
function then our server would run and be accessible
at 127.0.0.1: 5000, but it wouldnt be accessible to
external machines that kind of defeats the point of
having a server. By passing host=0,0,0,0, Flask will
listen on any IP address it can for potential requests
with a default port of 5000. debug=True is an error
message beautier that Flask includes.
Rather than crashing our server, an error page will
be delivered detailing what went wrong in place of
our expected return.

session[oauth_token] = responseData[oauth_token]
return Authorised + session[oauth_token]
def getParameters(paramString):
paramString = paramString.split(&)
pDict = {}
for parameter in paramString:
parameter = parameter.split(=)
pDict[parameter[0]] = parameter[1]
return pDict
def sign_request(parameters, method, baseURL):
baseURL = urllib.quote(baseURL, )
p = collections.OrderedDict(sorted(parameters.items(), key=lambda t: t[0]))
requestString = method + & + baseURL + &
parameterString =
for idx, key in enumerate(p.keys()):
paramString = key + = + urllib.quote(str(p[key]), )
if idx < len(p.keys()) - 1:
paramString += &
parameterString += paramString
result = requestString + urllib.quote(parameterString, )
signingKey = consumer_secret + & + session[oauth_secret]
print signingKey
hashed = hmac.new(signingKey, result, sha1)
signature = binascii.b2a_base64(hashed.digest())[:-1]
return signature
def create_oauth_headers(oauthParams):
oauthp = collections.OrderedDict(sorted(oauthParams.items(), key=lambda t: t[0]))
headerString = OAuth
for idx, key in enumerate(oauthp):
hString = key + =\ + urllib.quote(str(oauthp[key]), ) + \
if idx < len(oauthp.keys()) - 1:
hString += ,
headerString += hString
return headerString
if __name__ == __main__:
app.secret_key = R4nDOMCRypt0gr4ph1cK3yf0R5355i0N
app.run(host=0.0.0.0, debug=True)

Linux & Open Source Genius Guide 131

Developer guide

Create dynamic templates


with Flask, Jinja2 and Twitter
We create a dynamic webpage with Twitter and Flasks
rendering engine, Jinja2
Advisor
Sean M Tracey is a creative
technologist at a leading digital
agency. He spends a lot of his time
living inside of Node.js, Python and
Arduino

Resources
Python 2.7+
Flask 0.10.0:
lask.pocoo.org

Flask GitHub:
github.com/mitsuhiko/ask

A Twitter account
Your favourite text editor

QThe template uses a loop to generate a list of Twitter tweets

In the previous tutorial, we looked


at using Python and Flask to handle
the Twitter OAuth process and build
requests to obtain tokens. Weve used
Twitter in these tutorials because of the large
amount of easily digestible data available. However,
since Twitter adheres to the standards set out by
OAuth 1.0, the code weve used to sign and build
requests can be modied to work with any thirdparty API using the same standard without a great
deal of work.
For years PHP has been a mainstay of template
generation, but now with well-documented
frameworks such as Flask, Sinatra and Handlebars,
the ability to use powerful scripting languages
greatly improves our ability to make great web
services. To continue on from last time, were going
to use Python, Flask and its templating engine to

132 Linux & Open Source Genius Guide

display tweets. Flask comes with the super-nifty


Jinja2 templating engine, If youre familiar with
Node.js or front-end JavaScript, the syntax will look
very similar to the Handlebars rendering engine.
But, before we dive into that, we need to rearrange
some of the code we wrote last time.

Cut and paste them into a new le called helpers.py


in the root of your project folder. At the top of this le
we want to import some libraries.

01

from hashlib import sha1

Rearranging our code

Server code can get messy and


unmaintainable quickly, so the rst thing were going
to do is move our helper functions to another le and
import them into our project, much like you would
a module. This way, it will be clear which functions
are our server logic and endpoints and which are
generic Python functions. Open your server.py le
(or whatever you decided to call it) that we wrote
in our last tutorial and locate the getParameters,
sign_request and create_oauth_headers functions.

import urllib, collections, hmac, binascii,


time, random, string

Now we can head back over to server.py and import


the helper functions back into our project. We do
this by simply calling import helpers. Because
Python is smart, It will look in the current directory
for a helpers.py le before it looks for a system
module. Now every function included in helpers.py
is accessible to our project. All we need to do to call
them is prepend our the methods we called before
with helper.function_name and it will execute.

Linux & Open Source

Genius Guide

For sign_request, well need to pass our


oauth_secret and consumer_secret for each request
rather than accessing it from the session. Adjust the
function declaration like so:

def sign_request(parameters, method,


baseURL, consumer_secret, oauth_secret):

02

server.py modules

With a lot of the modules needed in this


project having been moved to helpers.py, we can
now remove most of them from server.py. If we
amend our rst import statement to be

import urllib2, time, random, json

{% if session[oauth_token] != %}
<h1>Already Authorised</h1>
<div class=dialog>
<p>Hello, Youve authenticated!<br>Lets <a href=/get-tweets>get some tweets</a></p>
</div>
{% else %}
<h1>Authorisation required</h1>
<div class=dialog>
<p>We need to <a href=/authenticate>authenticate</a></p>
</div>
{% endif %}

QThe BSD-licensed Flask is easy to set up and

Fig 01

Full code
avaliable online:
http://bit.
ly/1gH6W0S

use check out the website for more info


our project will continue to function as it did
before. Note the addition of the json module: well
be using that later as we start handling Twitter data.
Having Flask use a rendering engine is supersimple. Flask comes packaged with the Jinja2
template rendering engine, so we dont have
anything to install we just need to import the
package into the project. We can do this by adding
render_template to the end of our from flask
import [] statement.

03

Our rst template

04

Rendering our template

Now that we have a rendering engine, we


need to create templates for it to use. In the root of our
projects folder, create a new folder called templates.
Whenever we try to render a template, Flask will
look in this folder for the template specied. To get
to grips with templating, well rewrite some of our
authentication logic to use a template, rather than
manually requesting endpoints. In templates, create
an index.html le. You can treat this HTML le like any
other included in the resources for this tutorial is
an index.html that includes all of the necessary head
tags and <!DOCTYPE> declarations for this le.

In server.py, lets create a route for / to


handle the authorisation process.

@app.route(/)
def home():
if not oauth_token in session:
session.clear()
session[oauth_secret] =
session[oauth_token] =
return render_template(index.html)
Its a simple function: all we want to do is check
whether or not we have an oauth_token already

Linux & Open Source Genius Guide 133

Developer guide

and create those properties in the Flask session


so we dont throw an error if we try to access it
erroneously. In order to send our generated template
in response to the request, we return render_
template(index.html).

05

Template variables

We can choose to send variables to our


template with render_template(index.htm,
variableOne=value, variableTwo=Value) but
in this instance we dont need to as each template
has access to the request and session variables.
Open index.html. All code executed in a Flask
template is contained within {% %}. As this is our
homepage, we want to direct users accordingly, So
lets check if weve got an access token (Fig 01).
Between the ifs and else of the template is
standard HTML. If we want to include some data
for example, the access token we can just add
{{ session[oauth_token] }} in the HTML and it
will be rendered in the page. Previously, in our /
authorised endpoint, we would display the OAuth
token that we received from Twitter; however, now
that we have a template, we can redirect our users
back our root URL and have a page rendered for us
that explains the progress weve made.

06

Getting lost
(and then found again)

With every server, some things get misplaced or


people get lost. So how do we handle this? Rather
than dening a route, we can dene a handler that
deals with getting lost.

@app.errorhandler(404)
def fourOhFour(error):
return render_template(fourohfour.html)
If a page or endpoint is requested and triggers a 404,
then the fourOhFour function will be red. In this
case, well generate a template that tells the user, but
we could also redirect to another page or dump the
error message.

07

Static les

Pretty much every webpage uses


JavaScript, CSS and images, but where do we
keep them? With Flask we can dene a folder
for use with static content. For Flask, we create
a static folder in the root of our project and
access les by calling /static/css/styles.css or
/static/js/core.js. The tutorial resources include a
CSS le for styling this project.

08

Lets get some tweets

So now we know how to build templates,


lets grab some tweets to display.

134 Linux & Open Source Genius Guide

Now we know how to build


templates, lets grab some tweets
to display
In server.py dene a new route, get-tweets:

@app.route(/get-tweets)
@app.route(/get-tweets/<count>)
def getTweets(count=0):
Youll notice that unlike our other authentication
endpoints, weve made two declarations. The rst
is a standard route denition: it will intercept and
handle the path get-tweets. The second lets us
dene a parameter that we can use as a value in
our getTweets function. By including count=0 in
our function declaration, we ensure that there
will always be a default value when the function is
executed; this way we dont have to check the value
is present before we access it.
If a value is included in the URL, it will override the
value in the function. The string inside the <variable
name> determines the name of the variable. If you
want the variable passed to the function to have a
specic type, you can include a converter with the
variable name. For example, if we wanted to make
sure that <count> was always an integer instead of a
oat or string, wed dene our route like so:

@app.route(/get-tweets/<int:count>)

09

Checking our session


and building our request

Before we start grabbing tweets, we want to run a


quick check to make sure we have the necessary
credentials and if not, redirect the user back the
authorisation ow. We can do this by having Flask
respond to the request with a redirection header,
like so:

if session[oauth_token] == or
session[oauth_secret] == :
return redirect(rootURL)
Assuming we have all we need, we can start to
build the parameters for our request (Fig 02).
Youll notice that the nonce value is different
from that in our previous requests. Where the
nonce value in our authenticate and authorise
requests can be any random arrangement of
characters that uniquely identify the request, for
all subsequent requests the nonce needs to be a
32-character hexadecimal string using only the
characters a-f.

If we add the following function to our helpers.


py le, we can quickly build one for each request.

def nonce(size=32, chars=abcdef +


string.digits):
return .join(random.choice
(chars) for x in range(size))

10

Signing and sending our request

Weve built our parameters, So lets sign


our request and then add the signature to the
parameters (Fig 03).
Before we create the authorisation headers, we
need to remove the count and user_id values from
the tweetRequestParams dictionary, otherwise
the signature we just created wont be valid for
the request. We can achieve this with the del
keyword. Unlike our token requests, this request
is a GET request, so instead of including the
parameters in the request body, we dene them
as query parameters.

?count=tweetRequestParams[count] &user_
id=tweetRequestParams[user_id]

11

Handling Twitters response

Now were ready to re off the request


and we should get a JSON response back
from Twitter. This is where well use the json
module we imported earlier. By using the
json.loads function, we can parse the JSON into
a dictionary that we can access and well pass
through to our tweets.html template.

tweetResponse = json. loads(httpResponse.


read())
return render_template(tweets.html,
data=tweetResponse)
Whereas before, we accessed the session to get
data into our template, this time were explicitly
passing a value through to our template.

12

Displaying our tweets

Lets create that template now, exactly


the same as index.html but this time, instead of
using a conditional, were going to create a loop to
generate a list of tweets weve received.
First, we check that we actually received some
data from our request to Twitter. If weve got

Linux & Open Source

Genius Guide

something to render, were ready to work through


it, otherwise well just print that we didnt get
anything back.
Once again, any template logic that we want to
use to generate our page is included between {%
%}. This time were creating a loop; inside the loop
well be able to access any property we have of
that object and print it out. In this template were
going to create an <li> element for each tweet we
received and display the users prole picture and
text of the tweet (Fig 04).
In our template we can access properties using
either dot notation (.) or with square brackets ([]).
They behave largely the same; the [] notation will
check for an attribute on the dictionary or object
dened whereas the . notation will look for an
item with the same name. If either cannot nd
the parameter specied, it will return undened.
If this occurs, the template will not throw an
error, it will simply print an empty string. Keep
this in mind if your template does not render the
expected data: youve probably just mis-dened
the property youre trying to access.
Unlike traditional Python, we need to
tell the template where the for loop and if/
else statements end, so we do that with
{% endfor %} and {% endif %}.

13

Flask lters

Sometimes, when parsing from JSON,


Python can generate erroneous characters
that dont render particularly well in HTML.
You may notice that after tweet[text] there is
|forceescape, This is an example of a Flask lter; it
allows us to effect the input before we render in
this case its escaping our values for us.
There are many, many different built-in lters
that come included with Flask. Your advisor
recommends a full reading of all the potential
options.

14

Wrapping up

So, that is pretty much it for templating


with Flask. As weve seen, its insanely quick and
easy to build and deploy dynamic sites. Flask is a
great tool for any Python developer looking to run
a web service.
Although weve used Twitter to demonstrate
Flasks power, all of the techniques described can
be used with any third-party service or database
resource. Flask can work with other rendering
engines, such as Handlebars (which is superb), but
Jinja2 still needs to be present to run Flask and
conicts can occur between the two engines.
With such great integration between Flask and
Jinja2, it makes little sense to use another engine
outside of very specic circumstances.

tweetRequestParams = {
oauth_consumer_key : consumer_key,
oauth_nonce : helpers.nonce(32),
oauth_signature_method : HMAC-SHA1,
oauth_timestamp : int(time.time()),
oauth_version : 1.0,
oauth_token : session[oauth_token],
user_id : session[user_id],
count : str(count)
}

Fig 02

tweetRequest = helpers.sign_request(tweetRequestParams, GET,


https://api.twitter.com/1.1/statuses/user_timeline.json, consumer_secret,
session[oauth_secret])

Fig 03

tweetRequestParams[oauth_signature] = tweetRequest
makeRequest=urllib2.Request(https://api.twitter.com/1.1/statuses/
user_timeline.json?count= + tweetRequestParams[count] + &user_id=
+ tweetRequestParams[user_id])
del tweetRequestParams[user_id], tweetRequestParams[count]
makeRequest.add_header(Authorization, helpers.create_oauth_
headers(tweetRequestParams))
try:
httpResponse = urllib2.urlopen(makeRequest)
except urllib2.HTTPError, e:
return e.read()

Fig 04

{% if data %}
<ul id=tweets>
{% for tweet in data %}
<li>
<div class=image>
<img src={{ tweet[user][profile_image_url_https]
}} alt=User Profile Picture>
</div>
<div class=text>
<a>{{ tweet[text]|forceescape }}</a>
</div>
</li>
{% endfor %}
</ul>
{% else %}
<p>We didnt get any tweets :(</p>
{% endif %}

Linux & Open Source Genius Guide 135

Developer guide

Django test web


server running

Using Django
Admin to create a
new blog post

Displaying the SQL


commands that Django uses
for creating a database table

Executing the python


manage.py syncdb command
for the rst time

Build a blog with django


Advisor

Mihalis Tsoukalos is
a UNIX system administrator
also procient in programming,
databases and mathematics. He
has been using Linux since 1993

Resources
Django: www.djangoproject.com
Django built-in template tags and
lters: https://docs.djangoproject.com/en/dev/ref/
templates/builtins

Django tutorials: www.mikesdjangotutorials.


co.uk/

SQLite: www.sqlite.org
Apache: www.webforefront.com/django/
setupapachewebserverwsgi.html

136 Linux & Open Source Genius Guide

Learn how to program and publish a Django


photo blog site in easy steps
Django is a high-level Python web
framework that encourages rapid
development and clean, pragmatic
design and allows you to build
web applications quickly. Django focuses on
automating as much as possible and adhering
to the DRY principle. Since Django is a collection
of libraries written in Python, in order to create a
site using it you basically write Python code that
uses the Django libraries. If you already have a
good knowledge of Python, then you only have to
understand how the Django methodology works.

Django follows a slightly changed version of the


MVC (model-view-controller) design pattern called
model-template-view (MTV). The MTV handles the
controller work by the core and all the other work
is done in models, templates and views. According
to Django philosophy, what is truly important is
not terminology but getting things done quickly.
Join the Django community at https://www.
djangoproject.com/.
This tutorial will go through all the required steps
needed for creating a simple photography blog, so
make yourself comfortable and enjoy the ride

Linux & Open Source

Genius Guide

01

Installing Django

On a Debian 7 system, you can install Django


using the following command:

# apt-get install python-django


To make sure that everything works as expected,
type the following Django command, which prints the
version of your Django installation:

django-admin version
If your output prints something like 1.4.5, then
everything is okay. Otherwise check your
PATH settings or look for error messages.
django-admin is the main Python script
for
Django
administrative
tasks.
Run
django-admin without any command-line parameters
to display usage information and a list of all the
available commands.
Although at the time of writing this article the latest
Django version is 1.6.1, Debian uses an older version.

02

The Django web server

03

Creating your project and


your application

Django has its own lite web server that is


used during development for testing the application
so that you do not need to set up Apache just to test
your Django site.
The test web server allows one connection
at a time and has no security. By default, it
listens to localhost (its IP is 127.0.0.1) only. The
default port number that the test server listens
to is 8000. If you want to change the server port
number, pass it as a command-line argument
(python manage.py runserver 8580). If you
want to change the IP address of the server,
pass it along with the port (python manage.py
runserver
192.168.1.100:8080). If you make
changes to the project, you do not need to restart
it; just hit reload on your web browser and it will
restart itself. Instead of keeping a log le, it prints
descriptive error messages and other information on
screen that are very useful during development.

Execute the django-admin command with specic


parameters to start a new project. Each project can
contain many applications, so the project itself is like a
big container of different applications.
You can create a new project, called LUD, by
executing the following command:

$ django-admin startproject LUD


If you want to create a new application called
photoBlog, you have to go to the LUD projects
directory that is automatically created and has
the same name as the project and execute the
following command:

$ python manage.py startapp photoBlog


The manage.py script is automatically created
for every Django project and is a wrapper around

the django-admin Python script. You will see many


les ending with .pyc. These are byte code les
automatically created by the Python interpreter
and are executed by the Python virtual machine.
The rst thing to do is add the application
name to the list of the INSTALLED_APPS in the
LUD/settings.py le. If you do not do this, you will
not be able to use the application.

04

Directory structure and


les of your project

Django automatically generates certain les


and directories when you create a new project
or a new application. As far as the project
goes, the LUD directory contains a le called
__init__.py that makes Python treat the directory

as a Python package; a le named settings.py,


which is the projects conguration le (a very
important le that you will need to edit multiple
times); a le called urls.py that is the Master Table
of Contents le for the project; and a le named
wsgi.py that contains a simple SGI application
used for deploying Django with WSGI app servers.
You are going to need the wsgi.py le during
deployment phase.
Inside the photoBlog directory there exists a
le called models.py where you create the data
models of the application, a le named test.py
that has the unit tests, and a le called views.py
that includes the functions that handle data and
various other things. There is also an __init__.py
le that has the same purpose as before.

Linux & Open Source Genius Guide 137

Developer guide

05

Making changes to the


LUD/settings.py le

As you already know, the rst thing to do after


creating a new application is adding it to the list of
INSTALLED_APPS. For the photoBlog app, you just
have to add the following line:

photoBlog,
Next, add valid values to the ADMINS eld.
Then, change the TIME_ZONE value to match your
location. Later on in this article, you will learn about
the other changes you need to make to it in order to
support the Django administration interface.
The next step will show you how to use a
database with your Django project.

06

Connecting to your database

Django can communicate natively with


PostgreSQL, MySQL, SQLite and Oracle, but you
can also nd third-party drivers for other DBMSs.
When a request for a URL page comes, Django uses
a series of regular expressions to match the URL to
one of the dened views. Django has also its own
authentication system.
For reasons of simplicity, your advisor is using
an SQLite database, which is more than enough
for medium-size projects. The main advantages of
SQLite are that it does not need a server process
to run and that it is a single le database, meaning
that one single le has all necessary database data.
If the database le (photoBlog.sqlite) does not
exist, it will be automatically generated the rst
time you synchronise the database (using the
syncdb command). If you want to use PostgreSQL,
MySQL or Oracle, you should create a new database
and a user with a password at this point.
Make the required changes and run the
python manage.py syncdb command. The
great thing is that you dont need to write any SQL
commands to communicate with the database.

07

About the project

You will be able to see our example Django


site live at blog.highiso.net your advisors
photography blog. Having a live site to test a web
framework such as Django gives you a great
opportunity to learn new software better, faster and
deeper. Your only two concerns should be security
and having an online Linux server to host your site.
You should also have some system administration
knowledge.
Django sites are fast, easy to maintain and
contain only the necessary components as
opposed to content management systems, which
are huge pieces of software.
Each Django app you create is autonomous
and therefore can be easily used in other projects.
Keep that in mind when you are designing a new

138 Linux & Open Source Genius Guide

Django uses a series of regular


expressions to match the URL to one
of the defined views
application and try to make it as generic and selfsufcient as possible.

08

Using models

Using Django's ORM (object-relational


mapper) is one of the project goals. ORM allows all
Python classes that were dened inside models.py to
access the selected database without having to deal
with the database directly.
You will need to edit photoBlog/models.py
using your favourite text editor to create a new
class called photoBlogPost. Each photoBlogPost
class will contain a myID eld, a Title, a Body, the
Publication Date, the Number of Likes and a Photo
that can either be a binary object or a link to an image
le, depending on its type. Django prefers having
static resources served by your web server, so a le
link will be used. Django automatically adds a eld
called id that will be the primary key of the table.
After creating your models, you should run python
manage.py syncdb to tell Django to create the tables
inside the SQLite3 database le.
It is possible to change your models afterwards,
but it is good to be as accurate as possible during
design time to avoid potential problems in the future.

09

Dynamic templates

Django supports dynamic templates, which


are a way of presenting data dynamically. Its template

system comes with built-in template tags and lters,


but you can also create your own.
The templates used in this project will be located
inside the templates directory and you can reference
them in Django code. You should add the following
line using the full path inside the LUD/settings.py
le to make Django recognise the directory with the
template les:

/home/mtsouk/docs/article/working/djangoBlog.
LUD/code/LUD/templates
When your server goes live and is moved to a
different hard disk location, you will need to make
more changes, but this is sufcient for development
purposes.
You can have multiple template paths if needed. To
make your app to use templates, you should add the
following two lines inside your photoBlog/views.py file:

from django.template.loader import get_


template
from django.template import Context

10

About Django views

Django views are functions that describe


how to display information on screen when you visit
a URL. You dene different functions that describe
how different pages will be displayed. A view takes
an HTTP request and returns an HTTP response.
The rst argument of every view should always be an
HTTP request.

Linux & Open Source

Genius Guide

You will have to edit le photoBlog/views.py


to add views to your Django website. Every
view should correspond to a URL dened in the
LUD/urls.py le.

11

Project URLs

When you use Django for the rst time, you


will have to dene many URL patterns. The more
you use Django, the more you are going to reuse
previously dened patterns.
URL patterns are processed in order; as soon as a
match is found, it stops any additional matching. The
idea of this is to put specic expressions before any
generic expressions.
URLs are dened inside the le LUD/urls.py.
You can dene additional URLs elsewhere, but
everything starts at LUD/urls.py.
URLs usually use regular expressions to match
a URL given by the user on a web browser. The
following line will match a URL that has the word
welcome after the address and execute the
function welcome, located inside Python module
photoBlog, and its Python code which is in the
le views.py:

url(r'^welcome/$', 'photoBlog.views.welcome'),

12

Uploading and loading photos

The rst thing to do is declare the full


path of the directory that will hold user-uploaded
les using MEDIA_ROOT. This will happen inside
LUD/settings.py.

MEDIA_ROOT = '/home/mtsouk/docs/article/
working/djangoBlog.LUD/code/LUD/photos/'
The MEDIA_URL variable is set to uploaded_photos
so all images are stored inside directory /code/
LUD/photos/uploaded_photos.
Timestamps will be used for making the lename
unique (it is crucial that lenames are unique) using
the following Python module:

from time import time


A new method called get_le_name()
generates a new unique lename that is used by
the Django system. The unique lename is then
stored inside the database as a string. Djangos
ImageField does the rest of the work for you!
All les are stored locally. Later on, when your
production web server is serving the images, you
will have to create an Apache alias that matches
the string stored in the database.

13

Creating the views for the project

The welcome function now uses dynamic


templates, so it is renamed to welcome_template.
Renaming the function to include the word
template is not necessary, but it helps us to
remember how it is implemented.
A function named about_template will be created,
which will contain static HTML code and will also use
dynamic templates. Another function named list_
template is also implemented. This function reads
all the available blog posts from the database and
displays them on screen as a list.

14

The admin interface

Djangos admin interface allows you to work


on your model data using a webpage instead of the
command line.
The rst thing to do is to alter the
LUD/settings.py le. You should uncomment the line
that says django.contrib.admin, which is the Django
module that allows us to use the admin interface.
Next, you should tell Django about the URL of the
admin interface, which will happen inside the LUD/
urls.py le. By just conguring a URL, you can use
the admin interface application! A good technique
is making the URL to the admin interface a kind
of second-level security by naming it something
difcult to guess.
The last task to do is creating a le named admin.
py inside the directory of your app (photoBlog). This
le will have these contents:

from django.contrib import admin


from photoBlog.models import photoBlogPost
admin.site.register(photoBlogPost)

A veiw takes an
HTTP request and
returns an HTTP
response

Linux & Open Source Genius Guide 139

Developer guide

15

The beta version of the website!

What is currently missing and is going to


be added is a navigation menu for the website.
First, you need to add the following in the
LUD/settings.py le:

# For Supporting the Navigation Menu


TEMPLATE_CONTEXT_PROCESSORS = (
'django.contrib.auth.context_processors.auth',
'django.core.context_processors.debug',
'django.core.context_processors.i18n',
'django.core.context_processors.request',
'django.core.context_processors.static',
'django.contrib.messages.context_processors.
messages',
)
The URLs inside LUD/urls.py must have a slightly
different format so as to support the navigation menu:

url(r'^welcome/$', 'photoBlog.views.
welcome_template', name='welcome'),
url(r'^about/$', 'photoBlog.views.about_
template', name='about'),
url(r'^list/$', 'photoBlog.views.list_
template', name='list'),
The names are used in the template le templates/
navigation.html, as in Fig 01.
Please note that if you know how to write CSS code,
you can make the look of the Django site better. This
article is about Django programming, not website
design and CSS.
In order to add a new blog post, you will have to
use the admin panel. Alternatively, you can create
a Django form, but we think that the admin panel
is more than enough. Nevertheless, have in mind
that the admin panel is not for end users.
What is still missing is going live, but you are going
to learn more about it in a while!

16

screen, write it to a le or to a network socket.


By default, the Django logging configuration emails
all ERROR or CRITICAL messages as long as the DEBUG
value is set to False, which is not happening during
development. When DEBUG is True, all messages
caught by the logger are displayed on screen.

17

Implementing the Number


of Likes feature

First , add the following line to LUD/urls:

url(r'^node/like/(?P<myID>\d+)/$', 'photoBlog.
views.like_blogPost'),
Then, you need to implement the like_blogPost
method inside photoBlog/views.py:

def like_blogPost(request, myID):


if myID:
a = photoBlogPost.objects.get(id=myID)

count = a.nLikes
count += 1
a.nLikes = count
a.save()
return HttpResponseRedirect ('/node/%s/' %
myID)
The like_blogPost method updates an SQLite3
database record simply and elegantly. Last, you
should add the following to the templates/display_
blogpost.html le:

<p><a href="/node/like/{{myID}}">I like it!</


a></p>
The aforementioned HTML code calls a new URL
that has the form of /node/like/<ID>. This URL is just
used for calling the like_blogPost method and then
returns to the original photo blog post.

{% load base_extras %}

Fig 01

<div id="topbar">
<div class="navbar">
<div class="navbar-side navbar-left"></div>
<div class="navbar-content">
|| <a href="{% url welcome %}">Welcome</a> ||
<a href="{% url list %}"><b>List Posts</b></a> ||
<a href="{% url about %}">About</a> ||
</div>
<div class="navbar-side navbar-right"></div>
</div>
</div>
<VirtualHost *:80>
ServerAdmin mactsouk@gmail.com
ServerName blog.highiso.net
DocumentRoot /srv/www/blog.highiso.net/public_html/

Fig 02

Django logging

Django supports logging, which makes


debugging and administration easier. There are five
different log levels: CRITICAL, ERROR, WARNING, INFO
and DEBUG. You are already using the DEBUG log level
during development, but it is not recommended for
your production server.
To change the current logging setup, you will have to
edit LUD/settings.py.
Next, you need to add the following two lines of
code in the le you want to enable logging:

import logging
logger = logging.getLogger(__name__)
A logger is the entry point into the logging
system. Each log message that is written to the
logger is a Log Record. Each log record also has
a log level indicating the severity of that specic
message. Each logger sends its log records to a
Handler. The Handler determines what happens
to each message and can display a message on

140 Linux & Open Source Genius Guide

<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
WSGIScriptAlias / /srv/www/blog.highiso.net/public_html/LUD/wsgi.py
Alias /static/ /srv/www/blog.highiso.net/public_html/static/
Alias /uploaded_photos/ /srv/www/blog.highiso.net/public_html/static/uploaded_photos/
ErrorLog /srv/www/blog.highiso.net/logs/error.log
CustomLog /srv/www/blog.highiso.net/logs/access.log combined
</VirtualHost>
[Mon Jan 06
[Mon Jan 06
[Mon Jan 06
[Mon Jan 06
Python/2.7.3

Fig 03
18:07:49 2014] [notice] SIGHUP received. Attempting to restart
18:07:49 2014] [warn] mod_wsgi: Compiled for Python/2.7.2+.
18:07:49 2014] [warn] mod_wsgi: Runtime using Python/2.7.3.
18:07:49 2014] [notice] Apache/2.2.22 (Debian) PHP/5.4.4-14+deb7u7 mod_wsgi/3.3
configured -- resuming normal operations

Linux & Open Source

Genius Guide

18

Setting up Apache

The primary deployment platform for


Django is WSGI, which is the Python standard
for web servers and applications; therefore you
are going to need to have mod_wsgi enabled
on your web server. Run apache2ctl
-t -D
DUMP_MODULES as root to see if it is already installed
and enabled.
You will need to set up the DNS record for the
website domain. You adcvisor did this using the web
interface of Linode. The name of our example site is
http://blog.highiso.net/.
The conguration for creating the Apache virtual
host for the website is as in Fig 02.
After successfully setting up Apache and WSGI, the
Django site will no longer be in development mode, so
do not forget to change the DEBUG = True setting in
LUD/settings.py to DEBUG = False.
You will also need to revise the directory paths
dened inside LUD/settings.py if you are going to
move the development Django site to a new hard disk
location.
It is very important to change the
ALLOWED_HOSTS value to the following for
supporting connections from every internet host:

ALLOWED_HOSTS = ['*']
For the admin site to work properly on
the production server, you will need to
run the following command after setting
STATIC_ROOT and STATIC_URL:

# python manage.py collectstatic


Last, you need to use the full path of the SQLite3
database inside le LUD/settings.py.
If everything is working with Apache and WSGI,
you will see log messages similar to Fig 03 inside
/var/log/apache2/error.log.
This part of the tutorial cannot cover every possible
difculty that may come up, because even small
differences in program versions may create small
problems you will have to observe the log les and
correct the errors that appear.

19

Examining database
tables and data

It is very interesting to look at the tables of the


SQLite3 database. The site is based on 11 tables.
The most important table is called photoBlog_
photoblogpost. You can easily understand that
the rst part of its name is the name of the app
(photoBlog) and the second part is the name of
the model in lower-case letters (photoblogpost) as
dened inside the photoBlog/models.py le.

20

Last words

You should denitely study the project code,


run it using the Django test web server and try to make
small changes to the project.

The primary deployment platform for


Django is WSGI the Python standard
for web servers and applications
The presented Django application allows you to
quickly create and publish a photo blog from scratch,
in less than two hours, provided that you are familiar
with Django and Python. It needs some CSS code, but
it is fully functional.

21

Other things to try


when using Django

If you want to see the SQL code used for creating


one of your database tables, use the following
Django command:

$ python manage.py sql photoBlog

If you want to enhance the site, you can


implement keywords, user comments and search
functionality. You should also try to implement
Django forms.
When trying new things, always run the Django
web server to test your changes or nd errors in
your code. You can always check the live version of
the example Django blog at blog.highiso.net and
see how it progresses over time. Head over the the
Django website for answers to any questions.
Getting used to using Django will take some time,
but it will denitely be time well spent!

Linux & Open Source Genius Guide 141

Developer guide

Create a real-time CPU meter


with Node.js and Duino
Advisor

Sean M Tracey is a
creative technologist at a
leading digital agency. He
spends a lot of his time living
inside of Node.js, Python
and Arduino

Resources
Node.js v0.10.17:
nodejs.org

NPM v1.3.8:
www.npmjs.org/doc/cli/npm-install.html

Duino v0.0.9:
github.com/ecto/duino

Forever.js v0.10.11:
github.com/nodejitsu/forever

Arduino Uno/Duemilanove
5 LEDs
Potentiometer
13 prototyping wires
Your favourite text editor
Arduino IDE 1.0.5

Get a Node.js-powered Arduino to


display the current processor usage of a
computer in real-time
In this tutorial well be using Node.js and
the Duino library to get real-time CPU data
and display it with a series of LEDs. Well
also look into adjusting the brightness of
those LEDs with a potentiometer, and running the app
as a background process with Forever.js.
The normal process of programming an Arduino
is you write a program (or sketch, as its called by
enthusiasts), compile it, upload to your microcontroller
and then let it run. If you dont get the result youre
expecting, you amend the sketch, recompile and
re-upload it. Though robust, this method is a tad
cumbersome and long and thats before we even get
into topics like wiring up shields, relays and extra kit.
This is where Duino comes in. Rather than going
through all of that, we upload a sketch that allows
the Arduino to become a slave, of sorts, to another
computer an endpoint for data and instructions.
This is done through a serial port connection and, in
this instance, a Node.js library. The benets of this
approach are numerous, but primary among them
is that we can use the calculating ability of a far more
powerful computer to crunch problems and send only
the result to the Arduino to enact a function or effect.

01

Prepare the project

The rst thing we need to do to get this ball


rolling is install Duino.js so that we can upload its
software to our Arduino in order for the two to be able
to communicate. Lets start by cloning the GIT repo of
Duino to our computer.

git clone https://github.com/ecto/duino.git


You should now have a folder called duino on
your system. Open the directory and nd the src
folder. Found in this folder is a le called du.ino,
which is the software that will let Duino talk to our
microcontroller. Open this sketch in the Arduino IDE
and click Upload. The sketch will upload to your
Arduino, so that were now ready to start talking to it.
On your system, create a new project folder. You
can name it anything you like, and then enter it.

142 Linux & Open Source Genius Guide

From the command line execute the following:

npm install duino


This will create a folder called node_modules, which
will include the Duino software and all of its listed
dependencies.
Were now ready to start writing the program for
the Arduino.

02

Prepare the app

Using your favourite text editor, create a new


le app.js and insert the following:

var arduino = require(duino),


board = new arduino.Board({debug : false}),
spawn = require(child_process).exec,
pins = [3, 5, 6, 9, 10],
//These pin numbers may vary across
different versions of Arduino
brightness = 255;
Here were including the Duino library and using
it to create an instance of Board the function
that provides the methods for interacting with our
Arduino. Because Node.js doesnt have easy/direct
access to our systems CPU details, our app will
use the Node.js child_process module to execute
the shell programs vmstat and tail to get access
to information about user CPU usage. This will be
covered in detail in a few steps.
The pins array contains the numbers of the pins
well be using in this sketch. If you look at your
Arduino, youll notice that all of these pins are
capable of reading and writing both digital (HIGHLOW) and analog (0-255) data.
A little later well be adjusting the brightness of
our LEDs by adjusting a potentiometer. In order to
affect this change across the LEDs, we need to be
able to control the amount of power that goes to
them, so well be writing analog values with PWM
(which is what the ~ symbolises on the Arduino
board) to reect the potentiometers value.

Linux & Open Source

Genius Guide

How does it
all work?

Full code
avaliable online:
http://bit.
ly/1gH6W0S

The Duino software is basically an


abstracted version of Firmata, with
an API for Node.js. At its core, the
Arduino becomes an endpoint, always
listening for instructions from another
machine on what to read or write
through the Arduinos serial port. The
beauty of serial data is that, if you
wanted, you could go away and write
your own library in another language
C++, Python, Ruby, etc and so long as
they send the commands the Arduino is
expecting, each language will be able to
control it with minimum effort.

03

Wire the Arduino

04

Get the CPU data with


Node.js child processes

Before we proceed any further with


coding, now would be a good time to wire up
and then plug in the Arduinos. In the diagram
included, each of our LEDs is powered by its
own pin, but are all connected to the ground
pin via the ground bus on the breadboard.
The potentiometer is not connected to our
digital pins, but to one of the free analog pins, it
doesnt matter which one, but this tutorial uses A0 as
its the rst free pin.
In order to get a reasonably stable reading from
the potentiometer, you must connect it to the 5V
power pin. This is done through the power bus or
directly, as shown in the diagram. Once all of that is
connected, you can plug in your Arduino. Nothing will
happen right now as were yet to start controlling it.

Before we can start writing information to the board,


we need to have information to send it, so lets get
the CPU data. Earlier, it was mentioned that Node.
js doesnt have direct access to CPU information.
Depending on the version/avour of Linux youre
running, the following may vary slightly. This tutorial
was tested in Debian 7.2 but if your system runs
vmstat and tail, it should be ne. If we were to check
the CPU usage at every possible instance that we
could, our program would begin to tax the CPU, which
would affect the readings. Irony abounds! If we read
the CPU usage values every half-second or so, well
get a useful value and a reasonably good measure of
whats happening inside the box without causing a
measurable impact ourselves. Well achieve this by
wrapping the following code in a setInterval().

QTo get a stable reading from the potentiometer, it must be connected to the 5V power pin

}, 500);
setInterval(function(){
spawn(echo $[100-$(vmstat|tail -1|awk
{print $15})], function(error, stdout,
stderr){
var percentage = parseInt (stdout);
displayUsage(percentage);
});

Using the Node.js child_process module,


we can dispatch Shell commands (among
other things) and have the standard output
passed to callback function. Because weve
used a little command-line magic to shorten
the output of vmstat, what will be passed
through to the callback in this instance is
the current user CPU usage value expressed
as a percentage out of 100. Perfect. So lets
do something with that.

Linux & Open Source Genius Guide 143

Developer guide

05

Display CPU usage

Lets create the function displayUsage()


that we just passed the percentage to outside
of our interval loop. This part of our process isnt
particularly time sensitive.

function displayUsage(percentage){
var ledCount = ((percentage / 100) * 100)
/ (10 * (10 / pins.length));
}
Right now, all display usage will do is calculate

an integer between zero and four. Why is


this useful? If you look to the circuit youve
wired, youll see that we have ve LEDs. This
is a deliberate choice; wiring ten LEDs and
controlling them can become complicated and
slow in short order, not to mention that we dont
have ten pins capable of sending PWM values
on the Uno. The formula given will take the
percentage expressed by vmstat and calculate
exactly how many LEDs should be lit given
that percentage. In this instance, each LED
symbolises a 20 per cent block of CPU usage.

If all ve LEDs are lit, were running at 80 - 100


per cent, three LEDs and its somewhere between
60 to 80 per cent. Now that we understand what
ledCount does, lets use it.

06

Alternative ways of
controlling LEDs

Add to displayUsage() the following...

function displayUsage(percentage){
var ledCount = ((percentage / 100) * 100) /
(10 * (10 / pins.length));
for(var z = 0; z < pins.length; z ++){
if(z < ledCount){
board.analogWrite(pins[z],brightness);
} else {
board.analogWrite(pins[z], 0);
}
}
}
Remember the array of pin numbers we
created earlier on? Here were looping
through them and checking if the pin selected
from the array represents a value low enough
that it can be turned on. We can do this in
a variety of ways. If, for example, we only
wanted to toggle the pins on and off then
we could do something like this

var leds = [],


pins = [3, 5, 6, 9, 10];
for(var x = 0; x < pins.length; x += 1){
leds.push(
new arduino.Led({
board: board,
pin: pins[x]
})
);
}
function displayUsage(percentage){
var ledCount = ((percentage / 100) * 100) /
(10 * (10 / pins.length));
for(var z = 0; z < pins.length; z ++){

QNode aims to offer users an easy way to build scalable network programs

144 Linux & Open Source Genius Guide

if(z < ledCount){


leds[z].on();

Linux & Open Source

Genius Guide

} else {
leds[z].off();
}
}
}
...But those methods, while really useful, dont
allow us to set the values being sent. They
are HIGH or LOW because we want to be able
to adjust the brightness of our LEDs based
on user input, so were going to address the
board directly with board.analogWrite(pins[z],
brightness). The analogWrite method lets
us send an analog value between 0 and 255,
0 being off (or the analog equivalent of
LOW) and 255 being on full power (or, the analog
equivalent of HIGH).

07

Adjust the brightness


If you run app.js with

If we were to check the CPU


usage every possible instance
that we could, our program would
begin to tax the CPU, which would
affect our readings
of the dial. With a little bit of maths, we can
turn that value into a percentage of 255. If we
set the brightness as we have done below, it
will now turn the brightness of your LEDs up and
down as we turn it.

potentiometer.on(read, function
(err,
value) {
value = (value / 1024) * 100;
brightness = Math.round((value / 100) *
255);

node app.js
});
in the command line, assuming everything
is wired correctly, you will now have a fully
functioning CPU meter. To try it out, why not
open a couple of taxing programs at the same
time to see how it affects the measure? As
more of the CPU time is used, the number
of LEDs lit up should increase, and we could
leave it there. But, that does mean that weve
only covered writing values. If we want to read
values, how do we do that?
Head back over to your project le. Immediately
after the variables where we set the board
information, create a new variable called
potentiometer. This will be the object that
we use to listen for events and access data
from the potentiometer.

var potentiometer = new arduino.Sensor({


board: board,
pin: A0
});
You will notice that when we create the
arduino.Sensor object, we pass through an options
object with a board and pin key. Feasibly, this
means we can control more than one Arduino at a
time, but were not going to look at that this time.
Pin, of course, is the analog pin well read data from.
As well as an initiator, potentiometer is an
event emitter. If we listen for a read event on
potentiometer, well get a value somewhere
between 0 and 1024, depending on the position

08

Run it Forever... .js

And thats our real-time CPU app. Weve


managed to quickly put together something
pretty funky to use with our system but
right now were running app.js manually in the CLI.
CLI is awesome, but what good is a CPU meter that
needs a window open to run? This is where Forever.
js comes in.
Forever is, at its core, a daemon that lets your
Node.js scripts run forever in the background.
Forever will try its best to keep your Node.js app
running it even handles app crashes and restarts.
Installation is simple:

More benefits
The key benet of using something like
Duino is that you can ofoad a lot of the
problem crunching to a beeer machine.
Perhaps even more useful is the ability
for the controlling computer to navigate
the internet on behalf of the Arduino.
Arduino was never actually designed
with the internet in mind, by letting
our computer deal with navigating the
network and sending the result we
dont have to worry about shields, the
amount of bytes available or dropping
connections. It just works.

info:
Forever processes running
data:
uid command
script
forever pid logfile
uptime
data:
[0] fwGv /usr/local/bin/ node
app.js 96489 96490 /Users/ seantracey/.
forever/fwGv.log 0:0:1:13.707
If you want to stop a process, you can enter forever
stop [PROCESS INDEX].

$ npm install forever -g


Notice the -g ag at the end. This is a global install,
meaning that if any Node.js app needed to call
forever on this system, it wouldnt need to be saved
in the node_modules folder. After thats completed
installing we can now enter...

forever start [PATH TO OUR NODE.JS + DUINO


FILE]
...and the CPU meter is now a background process.
If you want to check the apps you have running with
forever, you can enter <code>forever list</code> and
youll get an output that looks something like this:

09

Brief recount

Voila! So concludes our whirlwind tour


of Arduino and Node.js. Weve covered a lot
in a short space of time. We now know how to
read information from an analog source (our
potentiometer), writing analog values to various
pins, the difference between digital and analog
signals, why weve used analog signals for this
tutorial, executing child processes within a Node.js
app to enhance our Nodes abilities and making an
app run forever.
This should be enough to get you excited about
what Arduinos can do; now its time to start
going out and making awesome bits and bobs
you can do so much with just a few wires and a
microcontroller!

Linux & Open Source Genius Guide 145

Developer guide

Build tic-tac-toe with Kivy


Ease into the workings of Kivy by creating this pen-and-paper
classic in just over 100 lines of Python...
Advisor
Alexander Taylor is a
graduate student in Physics,
but with a little time for fun with
Kivy on the side. He makes Kivy
screencasts and tutorials at
http://inclem.net

Kivy is a highly cross-platform graphical


framework for Python, designed for the
creation of innovative user interfaces like
multitouch apps. Its applications can run
not only on the traditional desktop platforms of Linux,
OS X and Windows, but also Android and iOS, plus
devices like the Raspberry Pi.
That means you can develop cross-platform
apps using Python libraries such as Requests,
SQLAlchemy or even NumPy. You can even access
native mobile APIs straight from Python using some
of Kivys sister projects. Another great feature is the
Cython-optimised OpenGL graphics pipeline, allowing
advanced GPU effects even though the basic Python
API is very simple.
Kivy is a set of Python/Cython modules that
can easily be installed via pip, but youll need a
few dependencies. It uses Pygame as a rendering
backend (though its API is not exposed), Cython for
compilation of the speedy graphics compiler internals,
and GStreamer for multimedia. These should all be
available through your distros repositories, or via pip
where applicable.
With these dependencies satised, you should
be able install Kivy with the normal pip incantation.
The current version is 1.8.0, and the same codebase
supports both python2 and python3. The code in this
tutorial is also version-agnostic, running in python2.7
and python3.3.

pip install kivy


If you have any problems with pip, you can use easy_
instal via easy_install kivy.
There are also packages or repositories available for
several popular distros. You can find more information
on Kivys website.
A kivy application is started by instantiating and
running an App class. This is what initialises our pps
window, interfaces with the OS, and provides an entry

146 Linux & Open Source Genius Guide

point for the creation of our GUI. We can start by making


the simplest Kivy app possible:

from kivy.app import App


class TicTacToeApp(App):
pass
if __name__ == __main__:
TicTacToeApp().run()
You can already run this, your app will start up and youll
get a plain black window. Exciting!
We can build our own GUI out of Kivy widgets.
Each is a simple graphics element with some specific
behaviour of its own ranging from standard GUI
functionality (eg the Button, Label or TextInput), to
those that impose positioning on their child widgets
(eg the BoxLayout, FloatLayout or GridLayout), to those
abstracting a more involved task like interacting with
hardware (eg the FileChooser, Camera or VideoPlayer).
Most importantly, Kivys widgets are designed to be
easily combined - rather than including a widget for
every need imaginable, widgets are kept simple but are
easy to join to invent new interfaces. Well see some of
that in this tutorial.
Since Hello World! is basically compulsory in any
programming tutorial, lets get it over with by using a
simple Label widget to display the text:

from kivy.uix.label import Label


Well display the Label by returning it as our apps root
widget. Every app has a single root widget, the top level
of its widget tree, and it will automatically be sized to fill
the window.
Well see later how to construct a full GUI by adding
more widgets for this one, but for now its enough to set

QThe classic Hello World! in Kivy GUI form,

using the built-in Label widget


the root widget by adding a new method to the App:

def build(self):
return Label(text=Hello World!,
font_size=100,
color=0, 1, 0, 1)) # (r, g, b, a)
The build method is called when the App is run, and
whatever widget is returned automatically becomes
the root widget of that App. In our case thats a Label,
and weve set several properties - the text, font_size
and color. All widgets have different properties
controlling aspects of their behaviour, which can be
dynamically updated to alter their appearance later,
though here we set them just once upon instantiation.
Note that these properties are not just Python
attributes but instead Kivy properties. These are
accessed like normal attributes but provide extra
functionality by hooking into Kivys event system. Well
see examples of creating properties shortly, and you
should do the same if you want to use your variables
with Kivys event or binding functionality.
Thats all you need to show some simple text, so run
the program again to check that this does work. You can
experiment with the parameters if its unclear what any
of them are doing.

Applications
can also run on
Android, iOS, plus 01
Raspberry Pi

Our own widget: tic-tac-toe

Since Kivy doesnt have a tic-tac-toe widget,


well have to make our own! Its natural to create a new
widget class to contain this behaviour:

Linux & Open Source

Genius Guide

Kivy comes
with all the tools
needed to use kv
language
these events to call our own functions when a user
presses the button, and can set the buttons text
property to display the X or O. We also created a new
Kivy property for our widget, coords well show how
this is useful later on. Its almost identical to making a
normal Python attribute by writing self.coords = [0, 0]
in GridEntry.__init__. As with the TicTacToeGrid, well
style our new class with kv language, but this time we
get to see a more interesting feature.

<GridEntry>:
font_size: self.height

Q
 A tic-tac-toe grid now accepting input, adding a O or X alternately

name based on our App class. That is, we should place


the following in a file named tictactoe.kv:

from kivy.uix.gridlayout import GridLayout


class TicTacToeGrid(GridLayout):
pass
Now this obviously doesnt do anything yet, except
that it inherits all the behaviour of the Kivy GridLayout
widget - that is, well need to tell it how many columns
to have, but then it will automatically arrange any child
widgets to fit nicely with as many rows as necessary.
Tic-tac-toe requires three columns and nine children.
Here we introduce the Kivy language (kv), a special
domain-specific language for making rules describing
Kivy widget trees. Its very simple but removes a lot of
necessary boilerplate for manipulating the GUI with
Python code - as a loose analogy you might think of
it as the HTML/CSS to Pythons JavaScript. Python
gives us the dynamic power to do anything, but all that
power gets in the way if we just want to declare the
basic structure of our GUI. Note that you never need kv
language, you can always do the same thing in Python
alone, but the rest of the example may show why Kivy
programmers usually like to use kv.
Kivy comes with all the tools needed to use kv
language; the simplest way is to write it in a file with a

<TicTacToeGrid>:
cols: 3 # Number of columns
This is the basic syntax of kv language; for each
widget type we may write a rule defining its behaviour,
including setting its properties and adding child
widgets. This example demonstrates the former,
creating a rule for the TicTacToeGrid widget by
declaring that every TicTacToeGrid instantiated should
have its cols property set to 3.
Well use some more kv language features later, but
for now lets go back to Python to create the buttons that
will be the entries in our tic-tac-toe grid.

from kivy.uix.button import Button


from kivy.properties import ListProperty
class GridEntry(Button):
coords = ListProperty([0, 0])
This inherits from Kivys Button widget, which
interacts with mouse or touch input, dispatching
events when interactions toggle it. We can hook into

As before, this syntax defines a rule for how a


GridEntry widget should be constructed, this time
setting the font_size property that controls the size of
the text in the buttons label. The extra magic is that kv
language automatically detects that weve referenced
the Buttons own height and will create a binding to
update this relationship when a GridEntry widgets
height changes, its font_size will change so the text
fits perfectly. We could have made these bindings
straight from Python (another usage of the bind
method used later on), but thats rarely as convenient
as referencing the property we want to bind to.
Lets now populate our TicTacToeGrid with
GridEntry widgets (Fig.01). This introduces a few
new concepts: When we instantiated our GridEntry
widgets, we were able to set their coords property by
simply passing it in as a kwarg. This is a minor feature
that is automatically handled by Kivy properties.
We used the bind method to call the grids
button_pressed method whenever the `GridEntry`
widget dispatches an on_release event. This is
automatically handled by its Button superclass,
and will occur whenever a user presses, then
releases a GridEntry button. We could also
bind to on_press, which is dispatched when the
button is first clicked, or to any Kivy property of
the button, which is dispatched dynamically whenever
the property is modified.
We added each GridEntry widget to our Grid
via the add_widget method. That means each
one is a child widget of the TicTacToeGrid, and so it
will display them and knows it should automatically
arrange them into a grid with the number of columns
we set earlier.

Linux & Open Source Genius Guide 147

Developer guide

Now all we have to do is replace our root widget


(returned from App.build) with a TicTacToeGrid and
we can see what our app looks like.

def build(self):
return TicTacToeGrid()
# Replaces the previous label
With this complete you can run your main Python file
again and enjoy your new program. All being well, the
single Label is replaced by a grid of nine buttons, each
of which you can click (it will automatically change
colour) and release (youll see the printed output
information from our binding). We could customise
the appearance by modifying other properties of the
Button, but for now well leave them as they are.

02

Has anyone won yet?

Well want to keep track of the state of the


board to check if anyone has won, which we can do
with a couple more Kivy properties:

from kivy.properties import


NumericProperty)

(ListProperty,

class TicTacToeGrid(GridLayout):
status = ListProperty([0, 0, 0,
0, 0, 0,
0, 0, 0])
current_player = NumericProperty(1)
This adds an internal status list representing who has
played where, and a number to represent the current
player (1 for O, -1 for X). By placing these numbers in
our status list, well know if somebody wins because
the sum of a row, column or diagonal will be +-3. Now
we can update our graphical grid when a move is
played (Fig. 02).
You can run your app again to see exactly what
this did, and youll nd that clicking each button
now places an O or X as well as a coloured
background depending on whose turn it is to
play. Not only that, but you can only play one
move in each button thanks to our status
array keeping track of existing moves.
This is enough to play the game but theres one vital
element missing... a big pop-up telling you when youve
won! Before we can do that, we need to add some code
to check if the game is over.
Kivy properties have another useful feature
here, whenever they change they automatically
call an on_propertyname method if it exists
and dispatch a corresponding event in Kivys
event system. That makes it very easy to write
code that will run when a property changes,
both in Python and kv language. In our case
we can use it to check the status list every time

148 Linux & Open Source Genius Guide

QThe game with final additions, making the grid square and extending the interface

it is updated, doing something special if a player has


filled a column, row or diagonal.

def on_status(self, instance, new_value):


status = new_value
# Sum each row, column and diagonal.
# Could be shorter, but lets be extra
# clear whats going on
sums = [sum(status[0:3]), # rows
sum(status[3:6]),
sum(status[6:9]),
sum(status[0::3]), # columns
sum(status[1::3]),
sum(status[2::3]),
sum(status[::4]), # diagonals
sum(status[2:-2:2])]
# Sums can only be +-3 if one player
# filled the whole line
if 3 in sums:
print(Os win!)
elif -3 in sums:
print(Xs win!)

elif 0 not in self.status: # Grid full


print(Draw!)
This covers the basic detection of a won or drawn
board, but it only prints the result to stdout. At this
stage we probably want to reset the board so that the
players can try again, along with displaying a graphical
indicator of the result (Fig. 03).
Finally, we can modify the `on_status` method
to both reset the board and display the winner
in a ModalView widget.

from kivy.uix.modalview import ModalView


This is a pop-up widget that draws itself on top of
everything else rather than as part of the normal
widget tree. It also automatically closes when the user
clicks or taps outside it.

winner = None
if -3 in sums:
winner = Xs win!
elif 3 in sums:
winner = Os win!

Linux & Open Source

Genius Guide

elif 0 not in self.status:


winner = Draw...nobody wins!
if winner:
popup = ModalView(size_hint=0.75, 0.5))
victory_label = Label(text=winner,
font_size=50)
popup.add_widget(victory_label)
popup.bind(on_dismiss=self.reset)
popup.open()
This mostly uses the same ideas we already covered,
adding the Label widget to the ModalView then
letting the ModalView take care of drawing itself
and its children on top of everything else. We also
use another binding; this time to on_dismiss, which
is an event dispatched by the ModalView when it is
closed. Finally, we made use of the size_hint property
common to all widgets, which in this case is used to
set the ModalView size proportional to the window
while a ModalView is open you can resize the window
to see it dynamically resize, always maintaining these
proportions. This is another trick made possible by
a binding with the size_hint Kivy property, this time
managed internally by Kivy.
Thats it, a finished program! We can now not only
play tic-tac-toe, but our program automatically tells
us when somebody has won, and resets the board so
we can play again. Simply run your program and enjoy
hours of fun!

03

Time to experiment

This has been a quick tour through some of


Kivys features, but hopefully it demonstrates how to
think about building a Kivy application. Our programs
are built from individual Kivy widgets, interacting by
having Python code run when their properties change
(eg our on_status method) or when they dispatch
events (eg Button on_release). We also briefly saw
kv language and experienced how it can automatically
create bindings between properties.
Find the full program at the above URL, which you
can reference to check youve followed everything
correctly. Weve also added an extra widget, the
Interface, with a structure coded entirely in kv
language that demonstrates how to add child widgets
this way. You can test it by uncommenting the return
Interface() line in TicTacToeGrid.build. It doesnt
do anything fundamentally different to what we
already covered, but it does make extensive use of
kv languages binding ability to automatically update
a label showing the current player, and to resize the
TicTacToeGrid so that it is always square to fit within its
parent. Play with all these settings to see exactly how
it fits together, or try swapping out the different widget
types to see how other widgets behave.

Try swapping out the different widget types


to see how other widgets behave
class TicTacToeGrid(GridLayout):
def __init__(self, *args, **kwargs):
super(TicTacToeGrid, self).__init__(*args, **kwargs)
for row in range(3):
for column in range(3):
grid_entry = GridEntry(
coords=(row, column))
grid_entry.bind(on_release=self.button_pressed)
self.add_widget(grid_entry)

Fig 01

Full program
avaliable online:
http://bit.
ly/1gH6W0S

def button_pressed(self, instance):


# Print output just to see whats going on
print({} button clicked!.format(instance.coords))

def button_pressed(self, button):


# Create player symbol and colour lookups
player = {1: O, -1: X}
colours = {1: (1, 0, 0, 1), -1: (0, 1, 0, 1)} # (r, g, b, a)

Fig 02

row, column = button.coords # The pressed button is automatically


# passed as an argument
# Convert 2D grid coordinates to 1D status index
status_index = 3*row + column
already_played = self.status[status_index]
# If nobody has played here yet, make a new move
if not already_played:
self.status[status_index] = self.current_player
button.text = {1: O, -1: X}[self.current_player]
button.background_color = colours[self.current_player]
self.current_player *= -1 # Switch current player

# Note the *args parameter! Its important later when we make a binding
# to reset, which automatically passes an argument that we dont care about
def reset(self, *args):
self.status = [0 for _ in range(9)]

Fig 03

# self.children is a list containing all child widgets


for child in self.children:
child.text =
child.background_color = (1, 1, 1, 1)
self.current_player = 1

Linux & Open Source Genius Guide 149

Developer guide
Installing Gulp

Get started with


Gulp.js
Explore the many benets of using the
Gulp.js task runner in your web projects
to help streamline your workow and
optimise your code

01

Install
Node

01

Install Node

02

Install Gulp

03

Save as dependency

04

Ready to gulp

Before you can begin exploring Gulp.js,


you will rst need to satisfy the most important
requirement, which is to install Node. Simply head
over to nodejs.org and download and install the
application onto your development machine. This is all
a very quick and painless process, which will take no
time at all.

With Node installed, you now have


access to the Node Package Manager (npm),
which we will use to install Gulp as a global
package onto our system. With your Terminal
or command prompt window open, use the
git
repository
(github.com/gulpjs/gulp)
to
install Gulp on your system.

Now you will need to create a package.


json le within your project directory. A skeleton
le can be created for you using the npm init
command from the Terminal. Once created,
you will need to install Gulp as a dev
dependency for your project, which will update the
package.json le for you.

The chances are quite high that during


any one web project, you have dealt
with or written more than one CSS or
JavaScript le. These have network
overheads when we end up delivering them to the
end user and can cause latency and loading issues
if they are too large. They may also not be as well
optimised as they could be, perhaps leading to
console errors or slow performance in general.
Enter Gulp.js, a free open source JavaScript
task runner powered by Node.js, which has quickly
gained traction in the web development community
as a fast, powerful alternative to existing task
runners on the market. But what are the benets of
creating and implementing a task runner tool into
your workow?
Using a task runner you can make the most of CSS
preprocessors such as SASS or LESS to streamline
how you write stylesheets, as well as coffeescript for
your JavaScript les. The task runner can compile
these into the desired formats for nal delivery. It

150 Linux & Open Source Genius Guide

can then iterate over your directories and combine


preselected les into one larger le, minifying it in
the process, while still keeping an unminied version
as well.
The additional benet of a task runner is
that it will watch all of your directories and les
and only run selected tasks when a le or
structure has changed. This removes the
necessity to manually run any task each time
an update has been made.
Lets also add in the ability to run code
validation and tests against your code,
automatically run as you save the le and
now you should start to see the real power
behind implementing a task runner procedure into
your web projects.
If you have some prior exposure to JavaScript, you
can get started with Gulp.js in a matter of moments
and add it into your projects quickly. Lets have a look
at how easy it is to work with and create your own
task denitions.

After the Node Package Manager has


obtained and installed Gulp as a dev dependency, the
reference to the dependency is also automatically
added to the package.json le as a requirement. Any
user or development machine with shared access
to this json le can install the same dependencies
quickly and easily.

Using a task
runner you
can make the
most of CSS
preprocessors
to streamline
how you write
stylesheets

Linux & Open Source

Genius Guide

Install and
maintain
dependencies
05

Install chosen plug-ins

As with any Gulp project, you will rst


be required to install your plug-ins as dev
dependencies to have them automatically added
to the package.json le. You are also able to install
multiple plug-ins in one command just make
sure to use the --save-dev command. Install the
gulp-load-plugins plug-in as well, which will help
manage the references when we code.
QTasks should be written in a clean manner, so that code is easy to maintain and manage

npm install gulp-concat gulp-jshint gulpbump gulp-load-plugins--save-dev

06

Reading package contents

The top of Gulple.js requires the core


Gulp library as usual. Directly below this, instead
of calling in each plug-in individually we can load
them dynamically by using just the gulp-loadplugins library. This will read the dependencies in
the package.json le and inject each of them for us.

var gulp = require(gulp);


var gulpLoadPlugins = require(gulp-loadplugins);
var plugins = gulpLoadPlugins();

07

page at github.com/jackfranklin/gulp-load-plugins.

gulpLoadPlugins({
pattern: gulp-*,
config: package.json,

scope: [devDependencies],
replaceString: gulp-,
camelize: true,
lazy: true,
});

Adjust references

The dynamically loaded plug-ins have


been assigned and required into the project
via the gulp-load-plugins extension. They will now
be available for use in your Gulple.js by referencing
them from the plug-ins object variable. For example,
any reference to concat() would become plugins.
concat().

gulp.task(lint, function() {
return gulp.src(js/*.js)
.pipe(plugins.jshint())
.pipe(plugins.jshint.reporter(default));
});

08

Thanks to the high engagement


level of the community, more
plug-ins are being added on a
continuous basis

Further options

This plug-in has a number of


conguration options which you can pass into the
gulpLoadPlugins() method. These include which
le to read, which keys in the selected le are read
to obtain the plug-in name, and whether or not the
plug-ins are lazy loaded. For more information,
make sure you check out the ofcial repository

Set up a project
One thing to be mindful of is to try and not
overload your Gulp conguration le in any
way. This could be through the structure
of your task denitions or the general
formatting of the code within.
Your tasks should be written in a clean
manner that is simple to understand and
maintain or manage should you need to.
Your Gulple may potentially be distributed
amongst a team of developers working on
the same project. Should any one individual
wish to amend or revise the le, it shouldnt
take too much time to do so.
There are a vast amount of plug-ins
available to download, implement and use
within your projects. Thanks to the high
engagement level of the development
community, more are being added on a

continuous basis. Gulp focuses on allowing


one plug-in to handle one task and to
handle it well. Although rather unlikely, it
is possible to saturate your Gulple with
too many plug-ins and tasks that are not
necessarily of any relevance to your project.
Consider the structure of your team (solo
or a larger group of people). Think about
the requirements for the application itself.
Do you need minication? The live reload
is useful but does it really t in to your
workow at the moment?
Think about the bare minimum tasks you
need in order to achieve your optimum Gulp
workow, and build upon that as and when
necessary. Start with the less is more
principle. Task runners are here to make
your life easier and more productive.

Linux & Open Source Genius Guide 151

Developer guide
Apply Gulp to 12
a project
09

Create gulple.js

When you run Gulp via the command line,


it will look for the existence of the gulple.js le.
This le informs the task runner which plug-ins to
load and denes the available tasks to perform
as well as setting the order of any chained tasks.
Create Gulple.js in the project root and ensure it
requires Gulp at the very top.

var gulp = require(gulp);

10

Reference plug-ins

Any plug-ins that have been installed


as dev dependencies and are to be used
within the Gulp process will need to be dened
and made accessible. To do so, simply require
each plug-in by name and assign to a variable
that can be called and used throughout the
conguration.

var
var
var
var
var

jshint = require(gulp-jshint);
concat = require(gulp-concat);
uglify = require(gulp-uglify);
rename = require(gulp-rename);
header = require(gulp-header);

11

Code linting

The rst task will check any JavaScript


code for errors by passing it through the default
linting engine. Simply register the task by
providing a name and the associating function.
Here the source directory is set and all les
within it will be piped to the jshint() method
for parsing and validation.

gulp.task(lint, function()
{
return gulp.src(js/*.js)
.pipe(jshint())

.pipe(jshint.reporter(default));
});

Minication
and concatenation

To streamline network resources we should also


make sure that any JavaScript les are combined
into one le and then a minied version of each is
also created.
Once more, set the source directory from which
to pull the les, which are then concatenated and
injected with a header value before being placed
into the destination directory.

gulp.task(scripts, function() {
var headerValue = Evaluated by
gulp.\n;
return gulp.src(js/*.js)
.pipe(concat(combined.js))
.pipe(header(headerValue))
.pipe(gulp.dest(dist))
.pipe(rename(combined.min.js))
.pipe(uglify())
.pipe(header(headerValue))
.pipe(gulp.dest(dist));
});

13

Always watching over


your les

One major benet of any task runner is having


the ability to watch a directory of les and then to
have the ability to react to any changes that are
made to them.
Here another task is created to manage this very
simple but powerful aspect of the workow.
Once
a
change
is
detected,
Gulp
will pass those les through to the lint and scripts
methods for processing.

gulp.task(watch, function() {
gulp.watch(js/*.js, [lint, scripts]);
});

14

Default tasks

When starting up Gulp from the command


line you can optionally choose to pass in the
name of the task to run. If the name is omitted,
Gulp will instead run a default task if one is
available.
You can easily ask Gulp to run a series of tasks
in order as part of the default task.

gulp.task(default, [lint, scripts,


watch]);

Write your own


plug-in
Gulp is highly
extensible and caters
for the majority of your
needs, but if you need
to build something
specic you can write
your own plug-in
Hopefully you should be able to see the
benet of implementing a task runner
into your development workow. The
short amount of time it takes to build up
your task functions and combinations by
far outweighs the time it would take to
manually run some of the processes and
features they provide.
With so many plug-ins available to
enhance and build your task resources
up into a manageable and benecial
workow, you will be hard pressed to
be at a loss for the majority of functions.
Having said this, Gulp is still very open
for you to create your own extensions
and plug-ins in order to enhance
your build les and give back to the
community at the same time.
Writing a plug-in for Gulp is a
matter of writing JavaScript that
can interact with and process streams
and data passed through them as well
as learning about the custom vinyl le
object.
There may be a learning curve for
many developers to get up and running
with their own plug-in. There are also
some fairly strict guidelines to follow,
and the code must pass tests before it
is accepted. More information on the
process and guidelines can be found
online here:
github.com/gulpjs/gulp/tree/master/
docs/writing-a-plugin.

Gulp is very open for you to create your own plug-ins to


enhance your build files
152 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

15

Plug-in
repository

Resources
Here are a selection
of plug-ins that will
add power to your
Gulple tasks
GULP-BUMP
www.npmjs.org/package/gulp-bump
Whether you are working within a team
of developers, committing open source
code for public consumption or building
a task runner for your own projects, this
plug-in allows you to update and bump
version numbers within any Semver
JSON les following any major, minor or
patch changes to the code.
GULP-SASS
www.npmjs.org/package/gulp-sass
This aptly named plug-in will convert
your Syntactically Awesome Style
Sheets into processed productionready CSS les. Combined with the
watch task, your les can be interpreted
as soon as they have been updated or
amended. This includes success and
error callback methods that can be built
into the workow process.

Gulp plug-ins
15

Plug-in repository

While Gulp doesnt have a plug-in


repository, a client-side application has been
created that reads from the Node Package
Manager library and lters plug-ins that are
available for Gulp. This is a fantastic resource
to see exactly what you can implement into
your task workow.

16

Details and documentation

Each plug-in page within the NPM repository


holds detailed documentation about the plug-in in
question, the installation procedure and then some
code examples for usage and implementation.

17

No limits

There are no limits on how many plug-ins


you can install into your Gulp process, saving them
into your package.json le or in a single request
from the Terminal and the npm install command.
Tailor your workow task processes to suit your
own preferences and your project requirements.

18

Update plug-ins

Plug-ins can evolve as new features and


bug xes are added. You can update your installed
dependencies easily using the Node Package
Manager via the command line. Simply enter npm
update from the command line and keep your plugins up to date.

GULP-IMAGEMIN
w w w. n p m j s . o r g /p a c k a g e /g u l p imagemin
Images and digital assets can often
be too large network resource hogs,
and the cause of many loading
issues for applications. This handy
plug-in will minify your PNG, JPEG
and GIF images to a more respectable
le size and help alleviate some of these
issues for your production servers.
GULP-LOAD-PLUGINS
www.npmjs.org/package/gulp-loadplugins
This fantastic plug-in removes the
requirement to specically dene
and require each plug-in individually
into your Gulple. The plug-in can
read the package.json le associated
with the project and the scope of
dependencies
contained
within
and
automatically
assign
them
to a global scope on your behalf.

Linux & Open Source Genius Guide 153

Developer guide
All the examples
presented in this
article are tested for
Vim version 7.3.547
on Ubuntu 13.04

Just type :help and


Vim displays some
valuable info for
newbies, such as
navigation keys and
keyboard shortcuts
to enable the
mouse in xterm or
GUI etc

Type :help iccf


and Vim displays
information
(including weblinks)
related to donations
for the needy in
Uganda as well as for
the development of
the Vim editor

Master Vim in easy steps


Advisor

Himanshu Arora is a

software programmer, open


source enthusiast and Linux
researcher. His articles have been
featured on IBM developerWorks,
Computerworld and in Linux
Journal. He (along with some
other Linux freaks) blogs at
mylinuxbook.com

Resources
Vim:
www.vim.org/download.php

154 Linux & Open Source Genius Guide

Take a step ahead, learn some advanced


features and get more productive with the
Vim editor
Thanks to a steep learning curve, many
people either avoid using Vim or use it on
a very basic level. If youre guilty of either of
these then youre really missing out.
Vim, rst released in November 1991, is a
command-line-based text editor that was developed
by Bram Moolenaar as an improved Vi clone for
Amiga platform. Besides including all the features
of the popular Vi editor, it also contains a lot of new
features and so derives its name from Vi improved.
As well as being a text editor, Vim can be used as an
integrated development environment by software
programmers. Vim provides a wide range of features,
but as it is a command-line utility it has a steep
learning curve. Please note that Vim is not a word
processor, which means that you cannot do layoutrelated changes with it.
While you will get to know about the real strengths
of the Vim editor once you master it, on the whole

some of the prominent pros include fast text editing


(because of little to almost no use of the mouse) and
easy execution from anywhere on the command
line. There are also many GUI front-ends (like gVim)
available for Vim.
Although it is command-line based, Vim has a
loyal following and is extensively used by system
administrators and software programmers. Though
it is hard to learn and understand all the features that
Vim provides, in order to reap its real benets it is
essential to master Vim for your day-to-day work.
Through this tutorial you will learn how to work
with multiple les, how to indent and fold code, how
to search and replace keywords, how to split the
Vim window horizontally/vertically, and many other
related features provided by Vim.
Note: This tutorial assumes that you have some
basic knowledge (opening, editing, saving and closing
a le) of the Vim editor.

Linux & Open Source

Genius Guide

01

Work with multiple les

While most regular users use Vim to edit


one le at a time, it can also be used to edit multiple
les in parallel. To open multiple les with the
Vim editor, just provide names of all these les as
arguments to the vim command. The rst le is
displayed by default. Type :n to switch to the next
le, :prev (or :N) to switch to previous le, :bf to
switch to rst le, :bl to switch to last le and :bw
to close the le.

02

Work with multiple les

Sometimes it is required to open a


new or an existing le while editing an already
opened set of les. Well, this can easily be done
by typing :e followed by the lename (along with
a complete path if its not located in the current
working directory).

03

Get list of currently opened les in


Vim

To get the list of les currently opened in the Vim


editor, just type :ls. This will display list of opened
les along with other information like the current

line number of the cursor in a le, le in current


active window (represented by %a), le in previous
active window (represented by #) etc. Switch
between current and previous active windows by
typing :b#.

04

Use wildcard characters

Vim also supports wildcard characters


(like *, ? etc) while opening multiple les at a
time. This is especially useful in cases where
multiple log les begin with a common set of
characters like the current date (2013-08-05---.log), or where multiple programming source
les have same language extension such as
.c or .cpp.

05

Copy-paste across the les

Another frequent requirement is to copy


from one le and paste into another. For this, you
need to split the Vim window to open both les. This
can be done either by using the -o option with the
vim command (vim -o <file1-name> <file2name>); or by opening a le with vim <file1name>, then opening the second le in split window

mode from within the active Vim window with :split


<le2-name>. To switch cursor control between the
two windows, press Ctrl+w (a couple of times).
Copy-paste operation across the les can
be done as follows: open les in split mode (as
explained above); copy the required text from one
le; press Ctrl+w to switch the control to the other
le; then paste the copied text.
You can also split a Vim window vertically. All
you need to type is :vsp <le-name> from within an
active window.

Pros include fast


text editing and
easy execution
from the
command line

Linux & Open Source Genius Guide 155

Developer guide

While most regular users use Vim to


edit one file at a time, it can also be
used to edit multiple files in parallel
06
Close all les

To close all the opened les in one go, add


a to regular close options like :q, :wq or :q!. So, to
close all the les in a single shot, do something like
:qa or :wqa or :qa!.

07

Close les without closing Vim

08

Fold blocks of code

Normally, when the last open le is closed,


the Vim application also exits. But, if it is required
to close the last open le without closing the
application, then just type :bd. This closes the
current le and opens an empty buffer.

Another frequent requirement which


software programmers usually have is to fold

156 Linux & Open Source Genius Guide

some blocks of code in order to save window space.


This helps them to focus on the required code
blocks. Vim does support block folds. For this, the
rst step is to bring cursor inside the braces of the
function denition. For example, bring the cursor to
the rst line of the function body. Once there, just
type va}zf. As you type this, you will observe that
va is used to activate visual mode in Vim, while }
selects the complete block including curly braces
and zf enables the fold.
To open the fold, press zo; to close it, press
zc. Make sure that the cursor is within the block
while performing fold open and close. Typing va}
zf creates a fold (that is invisible). zo and zc work
only when a fold is created. An existing fold can be
deleted by typing zd.

Linux & Open Source

Genius Guide

indenting for C code, use :set autoindent and :set


cindent. Now, try writing some C code and you will
see that Vim automatically produces indentation as
you write it.
Sometimes, depending upon the existing
settings of your Vim editor, lines might get
indented to a number of spaces other than
four spaces or any other default value that
you desire. As a solution to this, you can
set the number of spaces to indent using
:set shiftwidth=<number-of-spaces>. For example,
use :set shiftwidth=4 to indent lines by four
spaces.

09

Manual indenting

Another
important
feature
that
Vim provides is code indentation. Software
programmers are required to indent their code
and Vim provides easy steps for code indentation.
To indent a single line of code, you can use >> or <<
to indent the line to the right or left respectively.
Multiple lines can also be indented using [numberof-lines]>> or [number-of-lines]<<; for example, 10>>
or 10<<. This is a better technique than indenting
each line, but you still need to count the number
of lines to indent. This could prove frustrating,
especially if the number of lines is very large.
Fortunately, there is another trick: to select the
lines by working with Vim in visual mode. In order
to do this, you rst need to bring Vim into visual
mode. This (as mentioned previously) can be done
using va. Once in visual mode, use the arrow keys to
select all the lines that you want to indent in one go.
After this, all you need to do is press > or < and the
whole selected area will be indented accordingly.

For those
programmers
who want
everything
automated, Vim
also provides
auto-indenting
10
Automatic indentation

For those programmers who want


everything automated, Vim also provides an autoindenting feature. For example, to enable auto-

11

Search words

12

Replace words

To search for a word in the active window,


simply type /[search keyword]. This will enable
Vims search mode and nd the rst occurrence of
the searched keyword. Use n (just press n) in order
to search more occurrences of the keyword. Note
that you cannot do a backward search here.
If you need to do a backward search then just
type :?[search keyword]. This will enable Vims
backward search mode. Again, use n to continue
the search, but backwards this time.
Sometimes the keyword is present in the
text/source le and it is desired to nd other
occurrences of that word in the current text/source
le. In this case, just put the cursor below that word
and press * (ie Shift+8). This way, you can search
for a word without even typing it.

Vim also supports search and


replace operations. To execute a basic
replace operation, just type :%s/[keywordto-search]/[keyword-to-replace-with]
to
replace keyword-to-search with keyword-toreplace-with.
This trick is good if you want to replace all the
occurrences of a searched keyword, but what if it
is desired to replace only selected occurrences?
Well, there is a way through which you can run a
search and replace operation in interactive mode.
This means that Vim will conrm before replacing
the keyword each time. This can be done through
:%s/[keyword-to-search]/[keyword-to-replacewith]/c.
The rst occurrence is always highlighted and
Vim displays a prompt (in green, at the bottom)
where you can input your choice of whether you
want to replace this occurrence or not. Here,
y = Yes, replace this occurrence; n = No, skip
this occurrence; a = Replace this and all other
occurrences; q = Quit the command; l = Replace
this match and then quit the command.
Use Ctrl+e (^E) and Ctrl+y (^Y) to scroll the
window up and down in this mode. Also, use i and
I (along with /c, for example /ci or /cI) for caseinsensitive and case-sensitive search.

Linux & Open Source Genius Guide 157

Essential software
The best distros and projects
160 Presentation software
Choose the best slideshow software as we look
over a selection of the best Linux presentation
apps on offer

164 Music players


We find out the best way to organise and play
your music library by testing the four best
music players on Linux

168 Cloud storage services


Discover which service is the best and easiest
way for Linux users to store, share and manage
data in the cloud

172 Integrated development


We pit four of the best IDEs against each other
to find out how Linux users can streamline and
enhance their workflow

176 Ultimate IRC client


Keep up to date with your community as we
find out which is the best IRC client

180 System recovery


We compare four live-booting system rescue
distros so that you know which one to turn to in
a time of need

184 Video editors


We discover which video editor tool is the best
on Linux for creating and editing clips and films
for prosumers and professionals alike

188 Photo managers


Sort and edit your photos, from holiday snaps
to hardware shots, as we find out the best
photo-managing tool on Linux

Weve sorted out


the cream of the
crop in terms of
essential Linux
software for you

158 Linux & Open Source Genius Guide

Linux & Open Source

Genius Guide

Linux & Open Source Genius Guide 159

Essential software

Presentation software
Choose the best slideshow applications

Calligra Stage
Calligra is very impressive, although
it lacks the same kind of community
support and popularity as LibreOffice.
Calligra Stage is the presentation software
bundled with the suite, and is a standard desktop
presentation application in the same vein as
PowerPoint and Impress.
Starting up Stage, youll notice one thing to begin
with clutter. The main interface is just lled with
functions that dont quite t the space unless youre
using a 1080p or higher monitor. Even then, theres a
lot on the screen to deal with, leaving little room for
the preview of the actual presentation. The interface
is somewhat context sensitive at least, changing
out some of the boxes depending on whether youre
editing images, text or animations.
Some of the functions arent straightforward to
use, though, even if you can nd them in the interface.
The pointer function can link between text boxes and
titles, but it requires the use of anchor points on each

Successor to KOfce and KPresenter,


and direct rival to Impress

object, and it takes a little experimentation to gure


out how that works. This is not so good for a piece of
software you may not be using regularly and if you
do need to use it regularly, it may take more time to
learn than youre willing to give it.
Slide transitions are included in Stage; however,
animation for different objects on the slides is not
supported, even if theyre imported from another
slideshow application. This doesnt seem to affect the
imported presentations so much, though, which we
have had problems with in LibreOfce, and at most
itll require a little rearranging to make sure everything
is visible.
Presentations created with Stage look absolutely
superb, with a wide selection of fonts to choose
from, a nice array of image and line tools, and loads
of slide transitions to use. It would be a bit better if
the interface wasnt so nicky and cluttered, though,
and theres not much you can do to x the latter bar
buying a bigger screen.

Q There are a few graphical themes to choose

from, but theyre not all that great

SCORES
Ease of use

The bizarre nature of some


features makes it a little
tricky to create your slides

It will import many common


formats fairly well, but
Compatibility animation is not supported

Q The interface is very cluttered on non-1080p screens, but there are a lot of features to choose from

160 Linux & Open Source Genius Guide

7
7

Aesthetics

Presentations can
be made to look
good enough for a
business environment

Features

A wide variety of features


for images, text boxes
and ow charts, but
no animations

Overall

Calligra Stage is perfectly


functional, but there are
some interface problems
that are difficult to
wrestle with

Linux & Open Source

Genius Guide

EWIPE

A very simple
presentation
application that
does everything
in blocks

When we rst looked at EWIPE, we


were intrigued. Running on a Python
base, EWIPE presents itself as a fairly
lightweight program that can be used
to make presentations and CVs, while also being
able to save as a HTML le. While this all sounds
promising, the entire application is let down by a
number of issues that make it, frankly, unusable as
a way to create slideshows.
Firstly, the interface and slideshow only
use a 640x480 window. While this may be an
inconvenience for the main user interface if youre
creating the presentation, during the actual
presenting its unacceptable to only have a small
portion of the screen in use especially if the
display in question is 1920x1080.
Its a bit of a shame, though, as the method of
constructing your presentation is quite interesting.
Blocks are created in sequence for what you wish
to be placed on the slide. You can have three or so
different title types, text boxes, images and even

Q Theres only a limited workspace in which to design your slideshow

GIF animations. While it doesnt allow you to alter


the horizontal position or allow for two things to be
side by side, its a good way to focus on the ow of
the presentation.
This is unfortunately when another problem
comes in text and images will run off the edge
of the displayed slide, as theres no automatic
image scaling or any way to scale an image within
the tool whatsoever. This means youll need to
change image size in an external application, but
even then youre still wrestling to t items into such
a small area.

Even if you decided to create the basic


presentation in EWIPE to then use elsewhere,
theres no way to do that. EWIPE saves to HTML
and its own le format, one that isnt recognised
by Stage or Impress. In the same way, its unable
to import any other kind of slideshow le, not even
its own HTML les, and GIFs are the only images it
supports as well.
EWIPE is a huge disappointment and not worth
your time. The interface is a bit weird, but there are
some good ideas; however, everything else about
the software is unusable and unprofessional.

SCORES
Ease of use

A simple interface that


is sometimes just a little
too simplistic

It only works with its own


slideshow les, and there
Compatibility is no import option

Q The presentation itself is also limited, and does not have a full-screen option or smart scaling

6
0

Aesthetics

It looks like it was


designed in the early
Nineties, and you cant
make it full screen

Features

The most advanced


feature in EWIPE is its
ability to play animated
GIFs, something even
LibreOfce cannot do

Overall

We like the way you build


up the presentation with
blocks, but it doesnt
make up for the lack of full
screen and scaling

Linux & Open Source Genius Guide 161

Essential software

LibreOfce Impress
The slideshow
creator from
our ofce suite
champion is it as
good as the rest of
the suite?
LibreOffice is just an excellent office suite,
and a true open source success story.
One of the strengths of LO is its
compatibility with other software,
especially with Microsofts versions. While Writer
has some of the best compatibility of the lot,
Impress is still pretty great at importing a variety
of different presentation le types with very few Q The layout is clean and simple, giving you access to all the necessary functions
issues none that cant be tweaked to work with
Impress at least.
of course, you can add, remove or reposition
It is an amazing piece of software, well worthy of
The software comes with just about every
functions as you see t. Priority is given to making the LibreOfce name and a place in the full suite.
function youd see in other slideshow creators, sure you can see the preview of the slides, and the Presentations look great when displayed, while it
meaning you wont need to switch between
screen is never overly packed with functions and imports other le types reasonably well and is just
multiple ones. Whether youre doing a very
customisation. This is in stark contrast to the way nice to use.
professional business presentation or something
Calligra Stage is laid out, with everything placed
The only real issue with the software is that
a bit more informal, all the necessary graphs
on the screen at once with very little room to you cant quickly or easily publish presentations
and animations and sounds and interactions are
manoeuvre around them. Due to Impresss layout, online, although such a function will probably
readily available to use in Impress.
we were able to quickly and efciently put together be coming sometime in the future. For now,
The interface to create your presentations is
a presentation without having to hunt down though, Impress is most denitely the best ofine
extremely neat and logically laid out by default anything or wrestle with the interface at all.
presentation application.

SCORES
Ease of use

Every important function


is plainly laid out in an
easy-to-access fashion

While there are some minor


issues with various file
types, they can usually be
Compatibility fixed in the software

Q Impress is the only one that supports slide transitions and object animations

162 Linux & Open Source Genius Guide

9
8

Aesthetics

Impress will perfectly


display your presentation
the way you want it

10

Features

From graphs to
animations and everything
in between, Impress has
just about everything

Overall

LibreOffice is just an
amazing suite, and the
Impress package doesnt
let it down

Linux & Open Source

Genius Guide

Slidier
An online
presentation creator
and sharer that uses
HTML-like code
Although its not open source, even if you
are using it through Chromium, Google
Docs has a very serviceable presentation
tool. It may be a little basic, but it does just
enough for a presentation to look as professional as
anything Stage or Impress can make. Slidier is in
some ways the open source equivalent to this aspect
of Google Docs, allowing you to create and share your
spreadsheets online without needing any specic
software installed. In fact, the Slidier version is
better in that regard, as you dont need an account
for it, and sharing the link is a bit easier than the
Google method.
Unfortunately, thats just about the only thing it
has going for it against Docs. Slidier is an incredibly
simple tool that has you build your presentation as
a piece of code, one with very basic formatting to
signify titles or a break to the next slide. The most
complicated thing is adding an image, which uses
the img tags from standard HTML, while most other

Q Its a very simple way to create a slideshow, almost like editing a wiki page

functions involve a dash, a hashtag or a slash to yet professional, and play on just about any browser.
signify what they represent. Some of these are also Every presentation you make is saved with two URLs
accessible from preset buttons on the interface; one thats read-only, and another that allows you to
however, some functions need to be hunted down in further edit the slideshow.
If you need to create a simple presentation very
the About section.
The presentations you make also have some limits quickly, Slidier may just be what youre looking
due to this construction method. Like EWIPE, you for. However, there just arent enough functions to
can only control the position of items vertically on consider it as a replacement for Impress or Calligra,
the screen. You cannot change the alignment of text especially if you plan to use graphs or need some
or the position of images, or even have two items form of animation. For now, if youre happy using an
side by side. If you are able to play within these rules, ofine app or Google Docs, theres really no reason
though, the presentations you can create are simple to switch.

SCORES

Ease of use

While there are some


interface buttons, youll
have to learn the different
code bits to take full
advantage

Theres no way to import


or export a presentation
although for the latter
it would be quicker
Compatibility to recreate

Q There are some interesting features in Slidifier, but it does lack some important ones

7
0

Aesthetics

It generally looks very


nice, but you cant position
objects horizontally on
the page, or with much
precision vertically

Features

A very basic set of


features, but no
slide transitions or
other animations

Overall

There are some very good


ideas in Slidifier, but in the
face of Google Docs and
Impress, thats not enough

Linux & Open Source Genius Guide 163

Essential software

Music players
We nd out the best way to organise and play your music

Rhythmbox
There was a bit of a kerfuffle a year or
so back with Rhythmbox. Soon after
Ubuntu One started streaming music,
Canonical opted to switch to Banshee for
the next release of Ubuntu. These kinds of changes
happen everyday in Linux. However, what was more
surprising was the instant backtracking by Canonical,
reinstating Rhythmbox as the default music player in
the following release of Ubuntu returning with better
music streaming options.
Rhythmbox is a great default music player for
a number of reasons. It has a straightforward
interface with nice, big, standard buttons that
stand out so anyone can use it. The entire music
library is accessible down the side, along with a
handful of basic streaming services other than
Ubuntu One, and a dedicated Radio and Podcast list.
Playlists are split up from this main Library, although
these only include playlists youve created and
automatic ones such as recently added or played
music, with no smart recognition.

The Ubuntu default is a Linux


mainstay, but how does it compare?

Playback is absolutely ne, with les, folders and


selections being easily added to the now playing
queue thanks to great integration with the standard
le manager. Theres also good integration with
notication areas, particularly with GNOME-based
volume controls, allowing you to reduce the number
of icons and control music and volume from one place.
So pretty basic but good, then. Unfortunately, thats
about it for Rhythmbox. Theres not a huge amount of
customisation available for the interface either for
the layout or the way it generally works. Compared
to Clementine, which has a huge selection of music
streaming services available, access to Last.fm, Libre.
fm and Ubuntu Music is just not enough for Rhythmbox.
It needs more.
Overall, Rhythmbox is a great music player. Music is
very well organised and easy to search through, with
the multi-pane library window aiding in this. Creating
playlists and play queues on the go is nice and easy,
and playback is easy to control. Smart playlists are
noticeably absent, though, as are the aforementioned

range of online streaming services. If youre using it,


perhaps consider an upgrade.

Q The interface is simple yet fully featured

SCORES

Q Theres not a whole lot of customisation in Rhythmbox

164 Linux & Open Source Genius Guide

Playback

Pause, skip and more via


notifcation bar, media
buttons or the interface

Interface

A nice straightforward
interface with good
OS integration

Great music ltering and


playlist handling with a
Management good search function

7
9
8

Online

Very few online services


can be streamed, other
than normal radio
and Last.fm

Overall

Rhythmbox is definitely
good, but its in sore need
of a big update to make it
more relevant in a world of
online music

Linux & Open Source

Genius Guide

Audacious
Lightweight yet with
a surprising number
of features
Audacious is the only media player in
this group test that can be described
as lightweight and in fact like many
lightweight apps, its made that way
in mind. Still though, its a very popular music
player and music manager, thanks in part to a good
selection of features that make it more than just a
basic music player.
Audaciouss interface is simple, with a single
pane to list all available media, listed by album by
default. Music can be easily added, with a standard
folder being used to check regularly for any more
content for Audacious to add to your library of
media. Interestingly, playlists and play queues are
handled in separate tabs from the main library,
keeping the interface a little cleaner and allowing
you to have a specic layout of your favourite
playlists without needing all of them on show.
The search function is a bit odd, though. Instead
of an omnipresent search eld on the interface,
a separate column needs to be opened up from
the menus. The results from the search are
not displayed very well, with what seems to be
a random mash of authors, albums and songs
included in the results. This highlights a small

Q The layout of media can make it tricky to find anything

problem with Audacious: while the interface is quite


neat and minimalist, it makes looking for the music
you want to play a bit of a chore. Youre much better
off creating playlists in other music players to then
use in Audacious, or simply add them from the le
manager. Theres at least some decent integration
with the desktop environment once youve got
everything playing, though.
However, due to its lightweight nature, there are
no smart playlists, and there is no way to connect to

online services. This is its major downfall compared


to the rest of the applications in this test, as with
ever growing music libraries in multiple locations,
your music may not be all available on your PC or
laptop, or even your home network.
At the very least, Audacious is very customisable.
From the behaviour of what it plays at startup to
how it handles playlists and such, theres an eye
for making it convenient or keeping it lightweight. At
least in that regard, it performs very well.

SCORES

Playback

Plays media perfectly,


but lacks auto-generated
playlists due to
being lightweight

Interface

A simple and interesting


tab-based interface,
although search could
be better

Due to the simple


interface, actually looking
Management for your media is tricky

Q The search function is not the best, but Audacious is highly customisable

Online

No online services offered


whatsoever

Overall

Audacious is fine as
a lightweight music
player, but for proper
media management and
integration youll need a
bit more

6
0

Linux & Open Source Genius Guide 165

Essential software

Clementine
A Linux favourite,
how is the latest
Clementine player?
Clementine is based on the KDE
music player, Amarok, but with a few
improvements and a much better
interface. Its quickly become a very
popular media player and the latest version,
1.2, has arrived with a whole host of great new
features. These are sure to attract new users
while appeasing die-hard fans who still want to
use their favourite media player.
First of all, Clementine now has access to a lot
more music streaming services than before, with
new additions such as Dropbox and Ubuntu One
joining the already impressive list of existing ones.
These include Google Drive, Spotify, SoundCloud,
Last.fm and Grooveshark. You can easily search
within the free services using the built-in
Clementine search functions, and you can log in
to do the same with the account-driven services
such as Spotify and the cloud storage ones. These
settings are easily found in the preferences menu
under a different section to the vast wealth of
customisation options that Clementine offers.
Through these options you can change just
about every way Clementine behaves, from simple

Q Clementine has everything but the Kitchen Sink. It even has the Hypno Toad

things like how it might fade between tracks, to


tweaking the transcoding settings or even setting
a Wii Remote as a remote control device. New in
Clementine 1.2 is the ability to use an Android
device as a remote, a feature which has been a
long time coming. However, instead of using a
basic HTTP interface, it uses a special app to
make it work.
The playback is fantastic, with a special
Clementine icon ticking down to the end of the
song, and showing a play symbol so you know its

actually going. While you can control Clementine


from here, you can also control it from the usual
volume control icons if youre using the right
desktop environment.
Clementine basically has it all, then. Its smart
playlist feature, the dynamic random mix, isnt
quite as good as some online equivalents, but its
a lot better than any of the other players in this
test. It also has the greatest selection of online
services it connects to, is the most customisable
and makes nding your music easy.

SCORES

Playback

Makes playback as easy


as it can be, short of
dedicated buttons on the
notication area

Interface

The interface contains a


lot, but does the best it can
for the amount of features

Easy to navigate and nd


media, although some of
the online services could
Management work better

Q The Android app offers better control than some HTTP interfaces

166 Linux & Open Source Genius Guide

Online

Connects to everything
you would probably want
to use bar Pandora and
Google Music

Overall

An amazing piece of
software that lets you
do just about anything
youd want to do with all of
your music

9
10

Linux & Open Source

Genius Guide

Banshee
It may not be
as popular as
Clementine, but is
Banshee better?
Banshee was the one-time default audio
player for Ubuntu, replacing and then
being usurped by Rhythmbox. Due to
this, you would be forgiven in thinking
that they are incredibly similar applications and
in some regards they certainly are. They both
employ a similar three-pane layout for your media,
and they both include a column down the side
for navigating your media, videos, podcasts and
online services. At the core, they also both run off
GStreamer, which is a great media back-end and
allows the two to play just about anything with the
right codecs installed.
The interface for Banshee is nice and easy to use,
and very responsive. Search is instant, bringing up
results as you type, and the way results are listed is
conducive to nding the tracks, album or artist youre
looking for. The album pane on the main interface
has thumbnails of the album art instead of a list
although the grid effect can be disabled if you wish. It
all works very well and, like all the others, integrates
just ne with the desktop environments that allow
for playback options via volume controls.
Customisation wise, theres not a whole lot more
than Rhythmbox. You cant even set a specic

Q The Banshee interface is very nice

interval or time for the music library to update.


These kind of features are sorely missing, especially
compared to Clementine and Audacious which have
a whole host of different features and options that
can help you streamline the experience. At the very
least, theres a fairly rich plug-in system and you can
turn off some of the features of Banshee you dont
wish this way, making it much more lightweight than
it is by standard. Its through these extensions that
the online services are included in Banshee like

Rhythmbox, though, theres only a handful like Last.


fm and Amazon. There are a few other, communitybuilt extensions, but none to challenge the features
of Clementine.
So overall, Banshee is pretty good. While its
easy to compare it to Rhythmbox, its generally
a little better, with superior plug-in support that
enables it to be more lightweight if you wish, and a
slightly cleaner and informative interface. Its still no
Clementine, though.

SCORES

Playback

Good playback
options, but lacks
its own dedicated
notication icon

Interface

An easy-to-use interface
that is laid out in a
logical manner

For local content, its a


great way to keep track of
Management all your media of any type

Q The extensions give Banshee a lot of its features, and turning these off is the main way to

customise the software

Online

Limited online
options, and its only
minimally extendable

Overall

Banshee is a great media


player that wed be very
happy to use if we didnt
have access to any
online services

7
9
10
4

Linux & Open Source Genius Guide 167

Essential software

Cloud storage services


The best cloud services to store, share and manage data

Ubuntu One
It was a fairly big deal a few years ago
when Canonical announced Ubuntu
One 1GB of storage for all users of Ubuntu,
to be used as they wish to save and sync
their les to and from the cloud. Since then, Ubuntu
One has upgraded to 5GB of free storage and is
available to anyone, not just those using Ubuntu. As
well as offering 5GB for free, you can also add 20GB
of storage for $2.99 a month or $29.99 a year and
this stacks so you can continue to add 20GB to your
account, and add a further $2.99 or $29.99 to your bill
in the process.
The Ubuntu One client is fairly simple to use,
requiring little more than your Ubuntu One login
details and then some basic information on what you
wish to keep synced. As well as its own Ubuntu One
folder that has les that are automatically synced

Canonicals own cloud storage


works on more than just Ubuntu

across, you can choose to back up and/or sync


individual folders, with quick suggestions of those in
your home directory.
You can access your les from the web, other
computers with the client on, or even from an Android
client. If you have any music stored on it, you can pay
a little more for a streaming service, allowing you to
stream your music from Ubuntu One over the web
without having to download any of it rst. This is a
fairly unique feature, although it will cost you an extra
$3.99 a month or $39.99 a year.
Syncing is fairly quick and it noties you of any
changes made. There is no tray icon, though, so youll
have to hunt down the client to nd out if theres any
syncing going on. If youre running Ubuntu, it also
integrates into context menus, allowing you to rightclick les and send them to the cloud a nice feature.

The client doesnt allow for much customisation,


though, only speed limitations and whether or not it
should launch at boot time. Its easy to set up and let
run in the background, but Ubuntu One doesnt quite
have all the tweaks wed want.

Q Choose extra folders to back up to the cloud

SCORES

Space

5GB free is basically


a standard right now,
although 20GB
increments stack up
for heavy users

Integration

Notications are a bit


overbearing and theres
no tray icon, but the
context menu entries
are good

Features

As well as a standard sync


folder, you can stream
your own music and back
up other folders

Rather disappointingly,
only the very basic
customisations are
Customisation available in Ubuntu One

Q Straightforward access to files though the online portal

168 Linux & Open Source Genius Guide

Overall

Ubuntu One is a decent,


basic service that will
happily sync your files
without requiring much
direction. It should do
more, though

Linux & Open Source

Genius Guide

Copy
The newcomer
with a lot of
promise
Copy is so named as a little bit of an injoke ofcially, of course, its because
it copies your les to the cloud, like all
cloud storage services do. However, it
also copies other cloud storage services in some of
the way it looks and operates, giving its name a bit
of a double entendre. This is no reason to look past
it, though, as Copy has quickly become a very good
cloud storage solution.
First of all, you get 15GB of storage free, the
most out of any of the services were reviewing in
this feature. It also has the cheapest upgrade path,
offering 250GB for only $9.99 a month/$99.99 a year,
or 500GB for $14.99 a month/$149.99 a year. This is
one of the best prices available on any service and
while you would think there must be a catch, it isnt
quite what youd expect.
The client software for Copy is supplied in three
binaries, although you only need to use one of them
at a time, or ever. The main one is the graphical client,
allowing you to log in and set up your computer to
sync les to your copy account. Similar to Dropbox,

Q This is about all you get with the Copy client

Copy works by syncing a single folder between


devices and the cloud, and does not include
the folder backup function of Ubuntu One or
SpiderOak. There is, quite frankly, not a lot of
customisation available for the client side of Copy;
however, it does expose the transfer rate, some le
management options and allows you to set it as a
startup service for your system without having to
properly install it.
The other two binaries include a command-line
interface and an online API app. The command line

benets from already having done the graphical


setup, mainly because it doesnt automatically allow
you to log in, but its good for headless servers that
need the cloud storage.
Copy has denitely had a great start, offering a
competent service for a great price. However, this
kind of service comes at a cost to the company and
while 15GB free is a great incentive, it may quickly
become a double-edged sword if you become reliant
on a service that doesnt take off in a landscape
saturated with cloud storage suppliers.

SCORES

Q The website is laid out very nicely at least

Space

A huge amount of free


storage compared to
rivals, along with very
cheap upgrades

Integration

The binary doesnt


properly install, but allows
the client to launch into
the notication bar

Features

Only really offers one


folder syncing client side,
although you can undelete
les from it

You can only change the


speed limits and whether
or not the service starts at
Customisation boot time

Overall

Copy is very good, but


its lacking some of the
features of the other cloud
storage clients, and it may
be too early to jump on
board with it

Linux & Open Source Genius Guide 169

Essential software

Dropbox
No introductions
necessary for the
king of cloud storage
Dropbox is synonymous with cloud
storage; in fact, when describing other
cloud services to less tech-savvy people,
the conversation usually includes
the phrase its like Dropbox. Popularity doesnt
necessarily mean quality, though, which is why were
comparing it with several other services in this test
to see if it truly deserves its reverence.
Here are the facts then Dropbox is a crossplatform service offering 2GB of free data, with
small increments available for referrals and such,
which is upgradeable for a price. This starts at $9.99
a month for 100GB, which seems to be somewhat of
a standard in the cloud storage space and is cheaper
than Ubuntu Ones 20GB add-on scheme. Dropboxs
Linux client is fairly simple compared to the other
platforms it works on; however, it offers roughly the
same level of functionality without some of the bells
and whistles those versions have.
The client only allows for syncing of the main
Dropbox folder, rather than letting you select
different folders to also sync or back up. However,
you can tell Dropbox to selectively sync specic
folders from the main Dropbox directory, just in

Q Selectively sync specific folders in Dropbox

case there are les that only need to be accessed


by specic devices, or just via the web interface. You
can also share folders between other users, allowing
you to collaborate on work, or just quickly and easily
transfer les between each other. If you also have
another client on the same network, it will allow you
to transfer les over the LAN automatically not a
huge feature, but good nonetheless.
The web interface for Dropbox is also one of
the best, allowing you to easily navigate, edit and
change settings on les on your account and in any

shared folders that you have ownership of. Theres


also a pretty advanced undelete function that lets
you browse the last few weeks of les you have
deleted and restore them as long as you have space
in your account.
The good thing about Dropbox is that its available
on all desktop and mobile platforms, and it works
pretty much seamlessly between all of them. While
it doesnt offer all that much space for free, it does
have reasonably priced upgrade options and is one
of the most stable services available.

SCORES

Space

Not a lot of storage


space given away for
free, but very reasonable
upgrade fees

Integration

Integrates throughout a
network and has a useful
notication icon

Features

A good set of syncing


features and sharing,
but no extra folder
backup options

Theres actually not a


huge amount to change in
Dropbox, although there
Customisation are proxy settings

Q Have more control over your files via the web interface

170 Linux & Open Source Genius Guide

Overall

Dropbox is very good and


deserves its reputation.
Its solid, reasonably
priced and works very well
and very quickly

Linux & Open Source

Genius Guide

SpiderOak
A veteran in Linux
cloud storage
One of the rst commercial cloud
storage services to make its way to
Linux, SpiderOak has been around
for a few years. This means a couple
of things right off the bat rstly, it has the
same 2GB free, $9.99 for 100GB storage deal as
Dropbox. Secondly, it also has probably the best
and most mature client program out of any in
this test, which well shortly go into more detail
about. One of the benets of using SpiderOak is
that the rm behind it is extremely condent in the
security and privacy of your data on its server. It
operates what it calls a Zero Knowledge scheme,
where the server doesnt actually know what you
have stored on it because its encrypted. This is a
good idea and should guard against most types
of intrusion.
As mentioned, the client for SpiderOak is
extremely good on Linux. The main interface
supplies a lot of information on the current status
of backup and syncing, what the current queue
of uploading or downloading les is, a log of any
changes made etc. Like Ubuntu One, you can
select other folders to back up, as well as having
a SpiderOak Hive folder that syncs between all
devices. You can also heavily customise the way the

Q You get complete control over the client and how it syncs

client works such as telling it to only back up les


of specic size range, age range or excluding specic
keywords; setting a backup and syncing schedule in
case you dont want it to do it all the time; and it also
includes LAN sync like Dropbox.
While the website end isnt as advanced as
Dropboxs, there is an Android app which allows you
to access your les more easily on the go. One of
the neat functions included with SpiderOak, though,

is the ability to sync two folders between devices,


or on the same device, without having to use the
dedicated syncing folder. This is particularly useful
for syncing between folders on the same device if
you need some kind of a backup, or multiple users
are working on the same system.
Overall, SpiderOak is an extremely mature cloud
storage service and unlike other solutions, it hasnt
skimped on the Linux support.

SCORES

Space

Like Dropbox it only offers


2GB of free storage,
but there are decently
priced upgrades

Integration

SpiderOak is able to
integrate well with a
number of setups

Features

It offers the highest


number of features
weve seen for a cloud
storage service

Just about every aspect of


the client is customisable
with useful functions that
Customisation can be enabled or disabled

Q The web interface is basic but usable

Overall

SpiderOak is probably
the best commercially
available storage solution
on Linux, and we dont say
that lightly

Linux & Open Source Genius Guide 171

Essential software

Integrated development
The best IDEs to streamline and enhance your workow

Code::Blocks
A relatively new IDE, Code::Blocks
was released in early 2008 after three
years of release candidates. Although a
C++ IDE by default, it has plug-in support
to extend its functionality to allow for building and
testing of other languages. However, for pure coding,
Code::Blocks can be used to write scripts in various
languages thanks to support for syntax of a variety of
different languages. However, you cannot test these
without add-ons.
Unfortunately, the plug-in scene for Code::Blocks
is fairly immature and there isnt even proper support
for Python and other interpreted programming
languages. Plug-ins are found via the Code::Blocks
wiki and are not particularly well laid out at the
moment. Plug-ins are installed manually, which
would be ne normally, but the other IDEs in the test
have some interesting ways of managing plug-ins
that make this method seem somewhat archaic.

A more open IDE originally for C++,


now able to do more

While you wont be able to properly debug test


code, the interface for Code::Blocks is still very nice
for actually writing your code. Projects are handled via
proprietary projects les which link to specic folders.
Projects dont need to be created with a specic
language in mind and, like some text editors, it will
smartly know to highlight the syntax for different
languages. Browsing doesnt always work for all
functions and classes; however, it will allow you to
perform simple code editing to test elsewhere if you
specically like the interface.
Code::Blocks is also quite customisable, allowing
you to change the look and behaviour of the editor, the
way the compiler works and even editing the startup
script for the application itself.
Code::Blocks is a nice piece of software, although
due its immaturity it doesnt quite have the best plugin support of the other IDEs in this test especially if
you want to use it for multi-language programming.

Its a bit lighter than Eclipse at least, and the interface


is well laid out, but it may still be a few years before it
becomes preferable to the others.

Q There are a few first-party plug-ins available,

but not many third-party

SCORES

Q The interface is nice and aids workflow

172 Linux & Open Source Genius Guide

Installation

Available in repos, but


binaries and source
code is easily accessible
elsewhere

Workflow

Code::Blocks interface
is generally good for
workow, although
it doesnt support all
languages properly

Features

A great selection of
features that can all be
heavily customised

Plug-in
support

Very limited right now,


meaning that not all
languages are
properly supported

Overall

Code::Blocks has a great


interface, and some great
ideas on how to code and
manage projects, but the
plug-in selection needs to
be improved

Linux & Open Source

Genius Guide

Geany
A fully featured
IDE thats a little
more lightweight
than most
Geany is sometimes mistaken for a
gloried text editor. In reality, its a full
IDE, albeit a slightly more lightweight IDE
than most.
Geany has multi-language support out of the
box, with easy options to create les that use
different languages as part of the main interface.
Projects are kept in specic folders, like Eclipse,
although there is a le to go with them that Geany
can read to manage the projects. Multiple projects
can be viewed and edited at a time, all in different
languages. There are also debuggers and builders
for the various languages that support them, and
you can run and test stuff like Python that doesnt
need compiling as such.
The rest of the interface for Geany is clean and
well labelled. Code is automatically highlighted
with the correct syntax and theres a smart tab in
the left column that allows you to track and view
the different variables, functions and classes in the
projects and code. Navigation through the code is
fairly simple via context-sensitive menus and nice
options within the rest of the interface, and the

Q Geanys interface is smart and easily breaks down code for workflow purposes

usual code-editing tools such as commenting out a


selection or indenting are all present.
Sadly, Geanys plug-in support is pretty dire.
Some plug-ins extend the basic functionality for
a few of the languages, but theres not the kind of
depth as Netbeans or Eclipse in the sheer number of
available plug-ins.
There are a lot of ways to customise Geany with
the standard tools and menus, though. Almost every
part of the interface and workow is editable, with

ways to even change the characters required for


autocomplete suggestions.
Geany is without a doubt a very smart IDE, with a
low barrier to getting your project started straight
away. It does lack some of the features found in
some of the bigger IDEs, such as more advanced
unit testing and debugging, although it will let you
browse compile errors. Geanys plug-in selection
is also pretty poor, so it may not be extensible for
specic functions.

SCORES

Q The native languages are supported quite well with compilers and such

Installation

Readily available in
most repos and requires
minimal dependencies

Workflow

Workow is superb for


the variety of different
languages included

9
9

Features

No proper debugger,
but a great selection
of build and project
management functions

Plug-in
support

Although it doesnt quite


need it like the others,
there are very few
plug-ins available

Overall

Geany is a great,
lightweight IDE with
good native support for a
variety of languages, but
theres not much room
for expansion

Linux & Open Source Genius Guide 173

Essential software

Eclipse
Eclipse is an
industry standard
so how does it
fare against
more communityrun efforts?
Eclipse is one of the most popular
development suites around, and at first
glance its easy to see why. While created
mainly for Java development, Eclipse is
highly customisable through plug-ins. Thanks to
its popularity and community, this has resulted in
a great selection of add-ons that enables Eclipse to
work with just about any language. These plug-ins
allow for more than that, with a marketplace full of
interface and behavioural modications alongside
the language elements.
Eclipse has great project management tools as
well, with a smart tabbed interface, and plug-inspecic menu entries for starting projects in
different IDEs if needed. Projects are located in plain
folders in the location of your choice, allowing for
easy access of source code if you need it, instead of
being inside a proprietary le. The function search
ability works well and the interface has plenty of tips

Q Plug-in support for Eclipse is top-notch

and warnings for anything that might be inefcient


in terms of the code. There are plenty of editing tools
too, such as simple indent or dedent options.
The debug suite in Eclipse is fully featured, with
various ways to run, check and unit-test code,
although this depends on the plug-ins to some
degree. The tools are there, though, and most of the
major plug-ins seem to use them.
The plug-ins are handled by a repository system,
which lets you keep any add-ons up to date. While
there arent a huge amount available by default, its
easy enough to add more to the plug-in manager

and you can even select which extensions to install


from each of the repositories.
Eclipse is customisable in other regards, with
an expansive properties and settings menu that
lets you edit a huge amount of the native behaviour
of the IDE, from the way patches are viewed to
little things like key bindings and other shortcuts.
Eclipse is a fairly big suite of packages, though, and
a large resource hog. It does have probably the best
extensibility of all these IDEs, however, meaning it
also probably has the most to offer those who work
on a lot of differing projects.

SCORES

Q Code navigation and highlighting aids your workflow

174 Linux & Open Source Genius Guide

Installation

Eclipse requires a lot of


dependencies; however,
its available in most
major repos

Workflow

Smart interface design


that lets you easily
navigate projects
and code

Features

A great set of features by


default, although mainly
for handling Java

Plug-in
support

The best support around,


with a great repo system
offering a wealth of
add-ons

10

Overall

Eclipse is popular for a


reason its got a great
selection of features that
are easily extensible to
suit almost any need

Linux & Open Source

Genius Guide

Netbeans
The Oracledeveloped Java
dev platform
that multitasks
Netbeans started life as a humble
student project to make a Java IDE in
Prague. Within a few years, a company
had been set up around the IDE, and then
had been bought by Sun Microsystems. The rest
is history and after over 15 years of development,
Netbeans is on version 7.3.
Netbeans has a very similar feature-set to Eclipse,
with debuggers, compilers, unit testers and the
like, although by default Netbeans has support for
PHP and HTML as well as its Java-specic features.
Netbeans also has a huge database of plug-ins
to call from, allowing you to extend its features to
just about any programming language. These plugins are nicely arranged on the Netbeans website,
with simple search functions and a fairly easy way
to install the plug-ins to the system. Theres no
repository function for the plug-ins like Eclipse, but
the database displays enough information to know if
youre getting the latest version of the plug-in.
The interface is a little confusing not very
well streamlined and aesthetically a little dated.

Q Netbeans has a nice welcome screen to start new projects

While the splash screen is nice for starting new


projects, the rest of the interface is cluttered and
some functions are hidden away in weird menu
options. The code view is not particularly well
handled, with syntax highlighting a little sparse. The
function navigator is fairly nice, though, with the
sidebar allowing you to track the hierarchy a little
better than some.
Customisation in Netbeans is handled with a
nice-looking options menu, but there just isnt quite

the same level of customisation as Eclipse or the


others. However, for similar customisation tasks, its
a little better laid out than Eclipse.
Overall, Netbeans seems like its a little behind
the times, with an interface that needs a small
overhaul and a better selection of customisation
options. Its by no means a bad IDE, with a great suite
of debugging and testing options to use on code. On
the other hand, Eclipse is equal or better in almost
every regard.

SCORES

Q The interface is not particularly well laid out

Installation

Packages are readily


available for Netbeans
and it requires a
Java install

Workflow

The interface is a little


cluttered and convoluted,
but there are some decent
options to be found

Features

A similar feature-set to
Eclipse, but not as much
customisation

Plug-in
support

A fairly large selection


of plug-ins are available
for it within an easy-tosearch database

Overall

Netbeans just misses the


mark with a slightly dated
interface, but its made up
for somewhat with a good
selection of plug-ins

Linux & Open Source Genius Guide 175

Essential software

Ultimate IRC client


Keep up to date with your community

Xchat

A lightweight and customisable


IRC client with a basic interface

IRC client Xchat is currently found preinstalled on a lot of Linux distributions,


coming with both a graphical and
command-line interface. The graphical
interface is GTK+ based and incorporates a fairly
minimal and lightweight design, but not so much as
you would feel like there are missing features from
the interface.
Upon startup, a simple connection interface
is presented, enabling you to connect to one of
many precongured servers, or quickly connect
to another not on the list. Every major IRC server,
and more, is available here, although due to this it
can be a bit tricky to navigate to the one you want.
Its not easy to trim the list either, with many to edit
and delete.
Its also on this screen that you can start editing
your identity details, with different priorities
to different nicknames, with some generic
suggestions based on your distros username. This

is limited to ve, but should be more than enough


for most servers. You can set default channels to
autoconnect to in the servers, which works well
with the tabbed interface. Startup is quick and
the handy notication bar icon is omnipresent for
multi-desktop workows.
Theres a lot to customise in Xchat as well, from
the way notications behave, to the graphical
display and font choices. The notications are
quite varied, with different visual options and
sound options to make sure you know when very
specic actions have occurred. This works well,
although it makes out that there are some default
sounds already available for the client, whereas
you actually need to add your own.
As well as including logs, Xchat is a very solid
IRC client, something that should be set as the
minimum standard to compare others. Lightweight,
easy to use and quick to customise with a wealth
of options, Xchat allows you to congure it to work

in just the way you want it, while still maintaining


a great, basic interface. Although it may not be the
best IRC client, its denitely a great choice to be
the default in many distros.

Q There are fairly advanced server editing

options

SCORES
Installation

Available in a large number


of distributions; requires
GTK, though

Ease of use

The interface is easy to


understand, with a lot
of options contained in
one place

Highly customisable, but


not the most customisable
Customisation in this test

Q Customisation goes deep into the workings of Xchat

176 Linux & Open Source Genius Guide

Features

Feature rich for such a


lightweight application,
with a lot more than the
basics covered

Overall

A fine default, and a


great IRC chat client. The
inclusion of a commandline interface as well
means you dont need any
extra packages for that

8
8

Linux & Open Source

Genius Guide

Irssi
A functional
command-line
client with a limited
feature set
Irssi is the sole command-line-only IRC
client in this section, and is the preferred
option for a lot of Linux users that forgo
the graphical IRC offerings available.
Even though its a CLI application, it claims to be the
client of the future, and because of this it does have
the highest compatibility among the clients in the
test and wont require any extra graphical packages
such as GTK+ or Qt.
Being a command-line tool means the interface
is extremely limited. Theres no JED-style window
emulation here, requiring a completely manual
setup of Irssi to just about everything. The manual to
make it work is only available on the website which,
ironically, you will need to access via a graphical
web browser. From there you can learn how to
connect to servers and channels, how to start
creating multi-window setups and editing other
display preferences. There is actually not a whole

Q The command line limits the amount you can do

lot to edit, again limited by the fact its on the


command line.
Irssi works, though, and is probably the best
command-line IRC client around. The extra windows
stuff, while not bringing it up to the same standard
as the graphical clients in this test, is fairly novel for
a command-line-interface IRC client and allows you
to have a little more control over it. You can edit les
to keep these windows, allowing them to be part of

the default startup, which will then autoconnect to


channels and such.
While Irssi denitely has its unique and positive
attributes, its just not up to par with Xchat or
Konversation for the sheer amount of features,
customisation and general ease of use. Of course,
if youre only using a command line, we would
recommend this in an instant; however, if youre in a
desktop environment, the GUI clients offer more.

SCORES

Q Ironically, youll need to look up the website in a graphical browser to make sense of Irssi

Installation

Irssi is available in the


repo for most distros and
doesnt require many
dependencies

Ease of use

The setup needs to be


done manually and does
not offer any help unless
you read the website

While theres a decent


amount to do for a
command-line app,
theres still very little
customisation compared
Customisation to the others on test

Features

There are very few


features compared to
what you get with the
other clients

Overall

Good for a command-line


tool, but just not good
enough to abandon one of
the much better graphical
tools available

Linux & Open Source Genius Guide 177

Essential software

Konversation
Take full control of
your graphical IRC
experience with this
KDE app
Konversation, while a KDE app, is not
directly tied to that desktop environment.
Its development is independent from the
DE and it has its own release schedule
because of this. Its used in some KDE-based distros
as a default, although it doesnt quite have the
ubiquity of Xchat from earlier in this test. This means
that Qt is required for Konversation; however, the
necessary dependencies wont cause a problem in
the vast majority of cases.
First impressions of Konversation are that its a
little more complex than Xchat. The menu system
and startup screens offer a fair few more options,
with some of these displayed in separate windows,
such as a more advanced nickname selector, away
messages etc. However, theres only one server in
the list to begin with, a complete opposite to Xchat.
While this means a little more setup on your rst go,
it does result in a lot less clutter in the server listing.
The same automatic connection settings can be
made, with the ability to add multiple channels to
any server.
The amount of features and customisation
in Konversation is vast. Theres not a single part

Q Set up many nicknames and change their priority easily

of the interface or behaviour that you cannot


edit in some way from showing up any and all
information available from the IRC server, to setting
different fonts for various parts of the interface. Its
incredibly rich and while the application itself is
possibly the heaviest of the selection, its still very
quick and not particularly weighed down by all the
extra features.
Even the download manager is highly editable.
As well as the basics such as location, theres the

ability to edit the buffer, time-out and the fallback


interface. Its very well thought out as well, with all
the default settings for even the most niche options
set to give a great experience without any extra
ddling.
Konversation is basically the complete package,
allowing you to have full control over your graphical
IRC experience. It doesnt include a command-line
version, like Xchat does; however, its denitely the
very best you can use on a graphical desktop.

SCORES
Installation

Available by default on
some distros, get it with Qt
on others

Ease of use

A little more complex than


Xchat, but overall easy to
use with great defaults

Every part of the


application is
customisable through the
Customisation included menus

Q The amount of options for customising is staggering

178 Linux & Open Source Genius Guide

9
8
10

Features

Everything you could think


of, except a command-line
interface alternative

Overall

An amazing client and allround piece of software,


taking an IRC client to its
logical conclusion

Linux & Open Source

Genius Guide

Smuxi
Smuxi is a simpleto-use application
but its interface is
too simplied it
doesnt quite stand
up to its rival IRC
clients
One of the things we werent too keen
with in Irssi was how limited it is as
a command-line application, and
how much extra manual setup you
have to perform to get it to anywhere near the
functionality of the other clients weve tested.
Even then, there was only so close you could get to
that level of functionality.
Smuxi is inspired by Irssi and is a graphical
client with a twist. As well as being a standalone
graphical client, you can also have it as a front-end
client for a back-end IRC server. The back-end can
be connected at all times, with multiple front-end
clients connecting to it at once, allowing for the
client system to use up very few resources. The
standalone mode is still very lightweight, though,
and can be used just ne in that mode. The interface
is a fair bit simpler than the offerings from Xchat

Q The interface is more simple than the others

and Konversation, with no welcome window to


connect to servers, and some precongured
autoconnections to a couple of channels to begin
with. A quick connect menu is present in the corner,
allowing for just that. This limited main window is
indicative of the rest of the application, which has
very limited customisation options compared to
the others in this test, and nowhere near the same
amount of features.
The interface is at least good, though the tabbed
servers and channels works well, it connects and
runs relatively quickly and, as mentioned before,

its lightweight. Using Smuxi is ne enough and


it handles the very basics of what youd like to do.
Theres also simple download functionality, and
logging; however, it has the weird habit of requiring
you to exit the window and then quitting the entire
application from its icon on the notication bar.
Overall, Smuxi really isnt as good as Xchat or
Konversation, especially without the same level of
customisation the other two enjoy. The front-end
separation is a nice addition, and could be useful for
some, but it doesnt make up for the interface youll
be using being so simple.

SCORES

Installation

Its available in most


distros repositories
and is otherwise easy
to nd

Ease of use

Smuxi is a very simple


program that is equally as
simple to use

Limited customisation
for the interface
Customisation and behaviour

Q You can set servers to connect to at startup

8
9
6

Features

Aside from the very basic


interface features, the
front-end separation
is great

Overall

Smuxi isnt much more


lightweight than the
others, so its lack of
features makes it difficult
to recommend

Linux & Open Source Genius Guide 179

Essential software

System recovery
Live-booting system rescue distros you should turn to
the best system recovery tool
Parted Magic Not
but offers other useful tools
Parted Magic gets its name from
two of the partition editors featured
in the software suite, which is
an indicator of its focus. Theres
a lot of partition editing software on the disc, both
in graphical and command-line form, and there are
even tools and packages to help you diagnose the
health of your hard drives. Shufed in with these
apps, though, are a few great tools for other system
recovery and maintenance tasks.
For starters, ddrescue is included with the other
partition and editing tools. This data recovery
tool is very powerful for xing le corruption and
suchlike, with ways to compare it to previous
versions, even if they too are corrupt. Parted
Magic lists all available tools on its website, with
categories and a small description for each, allowing
you to look up the relevant package for the task
you wish to accomplish. While this is great, theres

otherwise very poor documentation on how to


perform various maintenance and recovery tasks
on the website, something that we will discuss in our
SystemRescueCD review.
The ISO itself is pretty tiny at just over 300MB, and
comes with multiple boot options at startup for 32and 64-bit kernels, alternate graphical modes, and
whether you want to load into RAM or not. This extra
bit of choice gives it a lot more compatibility on older
or slower systems, and in general its very fast to boot
and load.
In addition, it has some other maintenance
features, such as some rsync-based utilities for
backups, Clonezilla for disc cloning, and even an antivirus through ClamAV to scan your drives. Theres also
access to the internet through Firefox so that you
can look up any information or tutorials for the tasks
youre aiming to do.
Parted Magic is a good recovery distro, even if its

a bit more focused on hard drive partitioning. Theres


not much in the way of extra support for if you have
it installed to a USB stick or something; however,
there isnt much customisation youd need to do to it
between uses.

Q Theres plenty of choice at boot time

SCORES
Quick to boot, with some
advanced boot options
Boot

Q Parted Magic is full of graphical tools

180 Linux & Open Source Genius Guide

Recovery

Using ddrescue, you can


recover and x les in a
number of ways

Ease of use

A lot of graphical tools,


although not much
documentation on the
website

Extras

Backups, disc cloning,


system health and
partitioning tools are also
available

Overall

A great live distro to


have to hand if you need
to maintain a system,
but not the best for
recovery

Linux & Open Source

Genius Guide

Finnix 107
The smallest and
fastest system
recovery distro but
can be tricky to use
A Debian-based distro, Finnix is the
smallest and fastest one in this test. 107
comes in at about 120MB for both x86
and PPC architectures, yet is apparently
able to help you with all your system recovery
needs. It manages this by not including a desktop
environment, something all the others have, and
this also allows it to be incredibly quick due to its
command-line-only nature.
Before you get to the command line, though,
theres a selection of boot options that seems to be
the standard with these rescue discs, with a choice
of 32- or 64-bit. There are also a few extra options to
boot into FreeDOS, use the Smart Boot Manager and
run an iPXE so you can easily install other distros
over the network to the current system. There are
other boot proles as well, for those who want more
control over how Finnix runs.
It loads very quickly into the command-line
interface, proudly announcing how many packages
are installed. You can browse whats available
using dpkg -get-selections to get an idea of whats

Q Boot from the disc or boot from the network

available. However, theres noticeably less choice


than the other distros. Its still a decent selection,
though, allowing the system administrators its
aimed towards to perform le system recovery,
GRUB xing, le transfers, backups, cron jobs and
so on. Its just a little too obscure for other users to
use, and there arent many great guides for those
that need it either.

Its not a bad distro in any sense of the word,


though: if you know what youre doing, and you need
some very specic tasks done, Finnix may be your
best friend. Its extra network-boot functionality
is very useful, and its size and speed make it ideal
for a lot of system maintenance jobs. As already
mentioned, the downside is that it can be difcult
for some to use.

SCORES
Great a lot of booting
options, including ones
outside of Linux
Boot
Decent recovery apps, but
not a huge selection

Ease of use

Difcult for those not used


to working entirely in the
command line

7
3

Extras

There are some great


extras in terms of net
booting, but otherwise its
limited

Overall

A great recovery distro for


people who know exactly
what theyre doing, but
not so great for those who
suddenly find themselves
needing to use it

Recovery

Q Finnixs documentation is very light

Linux & Open Source Genius Guide 181

Essential software

Knoppix
An all-purpose
portable livebooting distro but
not our rst choice
This all-purpose live distro does have
sheer exibility as a mobile OS. With
multiple versions of every kind of app
youd need on the DVD edition, and a great
selection of core packages on the CD version,
Knoppix is a jack-of-all-trades as a Linux livebooting distro. When it comes to system recovery,
though, there are some very specic features that
need to be included, and Knoppix is clearly not the
most focused distro.
Here are the things it can do, at least. Obviously,
as a live-booting distro, you dont need to be able to
properly boot into your operating system to access
les if theres a problem booting. You can also x
GRUB using the command line if you know what
to do and of course with ample browsers in the
software list, you can easily look that up. There
arent any data recovery or forensic tools, though, so
for corrupt data youll need to look elsewhere.
Knoppix is a great portable distro. Its easy to
put onto a USB stick permanently and can then
have programs and packages added to it from
there. By default, it does come with other system

Q Theres a lot of software on the Knoppix disc

maintenance tools, such as partition editors and


backup utilities, and even the same virus scanner as
Parted Magic.
Theres no specic disc cloning utility other than
the standard dd, though, which may not be as fully
featured as some would like.
Knoppix also comes in rather a large size
compared to the other distros in this test, with the
CD clocking in at 700MB, and the DVD a full 4GB. It is

fairly fast at least, but can take a while to download if


you need it in a hurry.
Knoppix can be useful in a pinch, but its not really
designed for this kind of system recovery work.
While its a great portable distro and you should
probably have a USB stick of it to hand somewhere
if youre being that prepared though, you should
most denitely have one of the other distros to
hand as well.

SCORES
Highly customisable boot
sequence using cheat
sheet
Boot

Recovery

Ease of use

You can browse and copy


les, but theres not much
in the way of data recovery
The remaining packages
are easy enough to use
It has pretty much
everything else youd ever
need, at least

Extras

Q Once installed, you can extend the functionality of Knoppix

182 Linux & Open Source Genius Guide

Overall

As a system recovery
distro, Knoppix doesnt
do so well. All the other
distros here are far
better

9
5
8
10

Linux & Open Source

Genius Guide

SystemRescueCD
An essential
portable distro you
should keep to hand
On rst impressions, SystemRescueCD
seems to tick all the boxes wed want it to
tick as a recovery distro. The live-booting
OS comes in at a fairly small 370MB
and immediately offers up a huge variety of boot
options, with the choice of 32- and 64-bit, languages,
alternate boot methods etc. This then quickly loads
into a command-line interface for the interest of
speed, and it either lets you continue doing work on
the command line or gives you information on how to
boot into a graphical environment. This environment
is fast and gives you some extra graphical tools to do
your recovery tasks.
All in all, thats a great start. Its got plenty
of applications to help you with data recovery,
including ddrescue that we mentioned earlier, which
work mostly on the command line. However, the
SystemRescueCD website has some great guides
to get you going on that. This manual is set as the
homepage of the included web browser (Midori, to
keep it fast and within Xfce) so as long as you have
an internet connection, you can easily access it.
There are other maintenance apps as well,

Q The online manuals offer great guides

including partition editors and backup utilities,


making it multifunctional as a recovery and
maintenance distro. You can install it to a USB stick
from inside the distro with its own tool, or do it via a
Linux distro with a script included with the ISO. You
can even build it yourself, customising it along the
way. Its a well-executed concept that does just
about everything wed want a distro of this kind to

do. You can quickly work from the command line


if you need to, or boot into a graphical environment
if thats how you prefer to work, and being able to
easily clone itself from within the distro is incredibly
useful as well.
Having a stick or a CD lying around with
SystemRescueCD on it could easily save your
computer one day.

SCORES
A huge number of boot
options available through
a straightforward menu
Boot

Q Theres a wealth of boot options for different systems

10

Recovery

Multiple recovery tools


that can be used in
various ways

Ease of use

A lot of commandline tools, but theyre


thoroughly explained in
the manual

Extras

Other system
maintenance tools
round out the selection
of apps

Overall

SystemRescueCD is
a fantastic tool for
performing recovery
tasks on your system,
or doing some general
maintenance

Linux & Open Source Genius Guide 183

Essential software

Video editors
The best tools for creating and editing clips and lms

PiTiVi

A simple to use video-editing software, but tricky to


make more complex edits

PiTiVi is a tool that the developers say is


we tried to place audio or visual effects onto the clips,
easy enough for anyone to use, and rst
either not adding them properly or then creating
impressions of this video editing software
graphical glitches from the mouse cursor until the
are that it has a very straightforward
software was closed. Effects have very limited
interface that limits the on-screen functionality in
customisation as well, and are applied to the entire
favour of being clean and understandable. The basics
track rather than just a short selection.
are present: timeline, some cutting and selection
Its quite a shame, as there are some nice, albeit
tools, a preview pane, with options and such lling up
slightly random features available in PiTiVi. Firstly,
the rest of the space.
you can change any clip into a sequence of images
Sadly, this slightly simplistic presentation limits great if you want to create an animated GIF of
the amount of functionality of PiTiVi, making more
something, or edit specic frames.
than just simple cuts a chore or impossible in general.
Theres also a feature to try to join clips using
Sections cannot be separated from the main clip their audio as a guide, and while this requires you to
instead, the latter can only be cut, copied or deleted
install an extra Python component, its an interesting
and then placed somewhere else. Theres only one
addition that can help string together either brokenvideo or audio track, making it difcult to mix together
up clips or dialogue scenes.
audio without editing it in a separate audio editor,
Generally, though, PiTiVi is too simplistic for its
which sort of defeats the point. Its also a little buggy, own good, even if it has a fairly decent import and
with the interface regularly getting confused when
export ability thanks to GStreamer.

Otherwise, the problems with the effects make it


difcult to use for anything other than the simplest
editing tasks.

Q PiTiVi has a fairly simple interface

SCORES

Q There are a fair number of rendering options

184 Linux & Open Source Genius Guide

Installation

PiTiVi is available in most


major repos, and also
requires GStreamer

Ease of use

A little bit too simple,


making it not as useful
as the other editors in
this test

Features

There are a few effects,


and the image sequence
tool is nice, but its lacking
in general

Codec
support

Thanks to a GStreamer
base, it has decent
decoding for a variety of
video codecs

Overall

A bit more basic than


we had hoped for, and a
little buggy, but you can
make some decent videos
with PiTiVi

Linux & Open Source

Genius Guide

OpenShot
An allencompassing
video-editing tool
Built upon its own media framework,
OpenShot is a fairly all-encompassing
package that has similar codec support
to FFmpeg. Its one of the newer entries
in this group test, with its rst release just over four
years old now, and its denitely one of the better
editing suites weve come across. Like PiTiVi, the
interface is nice and straightforward; however, unlike
PiTiVi, this interface makes it easier to access a much
deeper library of features and editing options.
The timeline consists of a track hierarchy, with
higher tracks generally having dominance over
the lower tracks in terms of what is previewed
and encoded. Theres no differentiation between
music and video tracks, but its smart enough to
know not to cut off video if an audio track is placed
higher than a video track. These tracks can be
moved between by the use of transitions, which are
visually represented very nicely with an arrow to
determine the direction of play, bridging the tracks
together to give a nice sense of the ow of the video
itself. There are plenty of transitions available,
ranging from dissolves, wipes and fades to either
emulating a specic style or even creating some
impromptu special effects. There are some visual
and audio effects available as well, which are added

The layout is clean yet hides a lot of depth

to the entirety of the clip in the timeline rather than


a selected area. In this case youll need to split up
tracks to break up the effects. You can split off the
audio from a video track as well if you so wish.
There are a fair number of rendering options, with
plenty of presets and more advanced customisation
available for bitrate, le format, aspect ratio,
quality and more, so you can create your perfect
video le. This goes hand in hand with OpenShots

compatibility with a wide range of codecs: its able to


import from MKV containers as well as lots of other
le formats and codec types.
OpenShots slightly more advanced workow
over PiTiVi is a fantastic yet still easy-to-understand
addition, which allows people to easily make much
more advanced video clips and movies. Overall,
OpenShot manages to combine ease of use with a
good feature set.

SCORES

Q There are a great selection of effects and transitions that are fully customisable

Installation

Widely available in
repos, and has its own
media framework

Ease of use

One of the easiest


editors to use in this test,
without dumbing down
the features

Features

A great selection of
video editing tools and
tricks, as well as effects
and transitions

Codec
support

We found few issues


importing video, and there
were many export options

Overall

A prosumer-level video
editor that is only slightly
more difficult to use than
your basic video editor

Linux & Open Source Genius Guide 185

Essential software

Kdenlive
A video editor
suitable for
prosumers
Whereas OpenShot is a straightforward
video editor with a deceptive amount
of depth and large number of features,
Kdenlive could prove a little more
intimidating. With a more utilitarian interface
and workow, Kdenlive is deceptive in its design,
however, and generally just as easy to use as
OpenShot. The KDE video editor has been updated
in recent years to work a lot better on other
desktop environments and is also based on the
popular FFmpeg media framework, giving it a lot of
compatibility with various containers, formats and
codecs.
Like OpenShot, Kdenlive works on a hierarchy of
separate tracks, with higher tracks having priority,
and transitions being used to go between the
clips up and down the tree. However, in Kdenlive
there is a differentiation between video and audio
tracks, allowing you to perform some audiocentric manipulation to the tracks and clips in
your timeline. Effects and transitions are generally
easier to get to as well, with the right-click menu
bringing up a list of both to access. These effects
can also be heavily customised to your exact liking.
Theres a whole host of other video editing
features in Kdenlive to use as well, with a
stabilisation algorithm and the ability to create

The multi-track interface is very effective

noise tracks, countdowns and clicks among many


others its a pretty advanced piece of software.
Theres also the ability to create a video DVD, taking
already DVD/MPEG2 rendered material either
from Kdenlive or another DVD to create your own.
Rendering itself is fairly quick, with a queue so you
can have multiple things ready to go at once, or
even transcode other les to use elsewhere.

Kdenlive really is a very complete package,


with a great amount of attention to detail to a lot
of its core and extended features that make sure
it does just about everything youd want a video
editor to do.
With everything highly customisable, including
the presets and encoding proles, Kdenlive is a
fantastic prosumer-level video editor.

SCORES
Installation

Built on FFmpeg and


easily obtainable

Ease of use

There is some sacrice


to usability for the sake
of features

Features

Q Video exporting is highly customisable

186 Linux & Open Source Genius Guide

A wealth of options to edit


together any video project

Codec
support

Due to the FFmpeg


core, its video support
extends far

Overall

With a little more to offer


than most, and decent
compromise on the
interface for it, Kdenlive is
one of the best Linux video
editors around

8
8
10
9

Linux & Open Source

Genius Guide

Cinelerra
An out-dated videoediting tool for
professionals
Cinelerra is an interesting program.
Having previously won awards for its video
editing ability, it was a bit confusing to nd
that it was not readily available in all major
distros. Its promoted as an incredible advanced
video editor, with a lot of the tools used in more
professional scenarios, including a compositing
engine and the ability to work in both RGBA and YUV
colour spaces. The website offers links to binaries
and PPAs so you can install it on your system, albeit
only for a small selection of distros.
The interface for Cinelerra really needs an
overhaul. With a lot of low-resolution artwork and
bad colour choice, it can be a little jarring at rst.
At the very least, its comprised of many smaller
windows for the timeline, compositor, preview,
options etc, with a couple more windows you can
add and arrange as you please. The timeline has
the same layout as weve seen in OpenShot and
Kdenlive, and you can also now select sections of
track to manipulate. There are of course effects
and transitions; however, there doesnt seem to
be quite the amount that comes with some other
editors in this test.
Perhaps due to its status as more professional
software, it is unable to import a fair few le formats
we tried to throw at it, such as MKV, MOV and some
AVIs. These are not uncommon le formats and
while it will handle more RAW style formats, thats

The layout is configurable and complex

not useful to everyone. This same limitation is there


for encoding as well. The software also seems very
slow, chugging along trying to preview video, and
would grind to a halt when encoding a fairly simple
video that hadnt caused much issue to the other
video editors. Cinelerra has only received minor
updates in the past few years, and this lack of

attention to its display and import features shows.


While it may have been a great professional tool
in the past, its far past its prime now, meaning
more professional video makers will have to look
elsewhere, and others will have to stick to the
prosumer-level applications also featured in this
group test.

SCORES
Installation

The software is not well


maintained, and not
readily available

Ease of use

Cinelerra forgoes usability


in favour of offering a
professional experience

Features

Huge variety of features,


although there isnt much
depth in some of them

Codec
support

Q There are limited presets, but everything is customisable

Overall

Very few le types can be


imported and exported
Cinelerra may once have
been very good, but right
now its slow and unwieldy

6
5
10
3

Linux & Open Source Genius Guide 187

Essential software

Photo managers
Discover the very best photo-managing tool on Linux

Shotwell

This popular photo-managing tool covers all


the basics, but you might want more

Shotwell is the one application on


our test that is used by default in the
likes of Ubuntu or Fedora. Its popular
in GNOME-based distributions and
other GTK desktop environments, although of
course it works ne in KDE and the like. Its a
straightforward photo manager that requires little
conguration and is fairly lightweight to begin with,
much like the other Yorba applications.
When we say the interface is straightforward,
what we mean is that theres only a simple
thumbnail view of your pictures. While you can
increase the size of the thumbnails in this view, this
of course reduces the amount of images you can
see at once, and is not a true preview like you would
get with something like a lmstrip view that
kind of view is great for quickly scanning through
pictures for either yourself or showing off to others,
so its absence is a bit odd. The interface does
have its perks, though, allowing you do a few batch

operations such as tagging, placing in events, and


using the enhance button to do some autocorrection
on lighting and colouring it wont instantly make
things look amazing, but its subtle enough that it
doesnt overdo anything.
Events are treated like albums in the interface and
you can only assign one to a photo, while multiple tags
can be assigned. This is enough to differentiate their
usage, as you might have hundreds of tags but only a
few events.
Image editing in Shotwell is also very basic,
allowing you normal things such as cropping and
rotating, along with red-eye removal and a manual
colour balancing tool. Thats about it, though, and if
you want to do some other effects or manipulations
youll need to re up the GIMP.
Shotwell is a very serviceable photo manager,
doing the semi-basics well enough so that while it
might be easier to use other apps in this test, youll
be able to get the job done. It does come with a

great function that allows you to publish photos


online, though this includes posting to Facebook
or Flickr accounts, and is a good way to sort out all
your photos before uploading them.

Q Theres only a thumbnail view for photos

SCORES

Q Publishing online is a great feature

188 Linux & Open Source Genius Guide

Installation

A lot of dependencies
required; however, its
a default application on
some distros

Ease of use

A lmstrip view would be


nice, but otherwise its
very straightforward

Editing

The most basic tools to


crop and rotate images,
along with a magic
enhance button

Features

Quite basic; however, the


online publishing feature
stands out among all the
other photo managers

Overall

A decent photo manager


that can do all the basic
tasks youd want from
such an application, but
not much more

Linux & Open Source

Genius Guide

gThumb
A straight-forward
photo manager to
rival Shotwell
We were actually a little surprised by
gThumb, especially as its a GNOMEmade application. With the recent move
to simplifying all parts of GNOME, we were
wondering whether its popularity was misplaced;
however, it looks like all the changes to the GNOME
Shell havent affected the way gThumb works.
Its not to say that gThumb a super-advanced,
though it features a fairly straightforward
interface that wed want from a photo manager. By
default, its a thumbnail view, displaying the Picture
folder in the home directory. Instead of importing
photos from specic albums, you merely navigate
through the directory structure. This means that
instead of having specic albums to create, youll
need to organise those photos yourself in a le
manager or terminal.
Clicking on the Edit le button in the top right
opens up the image editor, and a sort of limited
lmstrip view as well. You can click between
photos in the stream on the bottom to edit them
individually, although you cant move between them
with arrow keys. The editor is a little more advanced
than what you get with Shotwell, allowing for more
manual colour and balance corrections, as well

gThumb lets you navigate the directory structure

as anti-blurring, desaturation and negative lters.


There isnt any red-eye removal, though, and no
paintbrushes or anything to do that, meaning youll
need to open GIMP for those kind of operations.
You also cant do batch autocorrection for the
images, but you can do batch tagging. You can drag
a box over the images to select them, or press Ctrl/
Shift with click, and youll be able to add, remove or
assign tags in batches.
You can edit information in batches as well;
however, youll likely want to do that individually.

Images that have been edited in this way get a date


added to them, letting you know when they were last
modied by gThumb.
Like Shotwell, gThumb also has online publishing
tools, with similar services such as Facebook,
Twitter, Flickr etc. Its nice that these tools exist, as
these services dont have touch-up options, allowing
for mass uploading of holiday photos and the like.
gThumb is a great little application and while it
doesnt quite have everything we want, its denitely
a step up from Shotwell.

SCORES

Q Plenty of photo information can be changed in gThumb

Installation

Easy to obtain, but


also requires a few
GNOME packages

Ease of use

Generally very easy to use;


however, the edit/lmstrip
view is confusing

Editing

Decent amount of editing


tools, although no
red-eye removal

8
8
7

Features

While not particularly


feature-heavy, its got
enough to properly
manage your images

Overall

A very strong showing


from gThumb, a photo
manager with plenty of
features, although it could
do with some batch image
touch-up options

Linux & Open Source Genius Guide 189

Essential software

F-Spot
A little out-dated
tool but with some
interesting features
F-Spot was replaced by Shotwell in
Ubuntu a few years ago. However, the
application has stuck around, albeit
without an update in over two years.
This simple image editor actually comes with an
interesting variety of features, although theyre not all
that useful in the long run.
The rst time you start it up, youll need to
import photos from a directory. This is a pretty
straightforward process, although it initially looks
like youll need to change settings on each individual
image in selected folders. It quickly imports into
the library, displaying a thumbnail view, some basic
metadata, a histogram and a sort of timeline for
basic ltering of your photos. The timeline breaks
are used to show a graph of photos taken per month
along the top, allowing you to select each month
from there.
You can also lter by tags and such in the main
view, as well as adding extra tags and dates to the
lters to nd exactly what you need.
The thumbnail view is quite basic, and you can
increase the size of the thumbnails in the view.
Double-clicking a picture enters the edit mode,

The timeline can help filter some images

which you can also select from the top bar, and this
includes a lmstrip view that you can key between
with arrow keys on the keyboard. The editor is a little
simplistic, with some basic autocorrection tools,
although they seem to do very little to the photos we
tried them on.
The main browsing interface can be a little
confusing as well, especially once you start ltering
images. Its not always clear how youre supposed
to get back to the standard view if you want to, and

you need to completely reset the lters to remove


any. Tagging is a little weird as well, since you need
to create the tags before you can even apply them
to anything, even if you have the images you need to
tag already selected.
While there are some good parts to F-Spot, in
general its not as complete or as easy to use as we
would have liked. The interface has a few ease-ofuse issues that are not encountered with the other
photo managers, too.

SCORES

Q The editor and file strip view are quite nice, but could do more

190 Linux & Open Source Genius Guide

Installation

Generally easy to get,


although its a little older
now and requires older
dependencies

Ease of use

Basic navigation is
great, but can get a little
counter-productive with
more advanced searches

Editing

Has a small selection of


autocorrection buttons,
but they do very little

Features

A decent variety of
features, but not all of
them work quite as well as
they should

Overall

F-Spot is showing its age


and is just not as good as
the others in this test. A
newer update could fix the
problems, but that may
never come

Linux & Open Source

Genius Guide

digiKam
A fullycomprehensive
photo manager
The KDE-developed image organiser has
recently released the digiKam Software
Collection version 4.0.0 which has lots of
new features. DigiKam has received a few
upgrades over the past few years since it hit version
2.0 and became one of open sources premier photo
managers. It comes with a full range of functions,
including the standard preview and lmstrip view
were looking for, batch tag and data editing, editing
software, and even a light-table function.
Although built for KDE, digiKam works just ne
with GTK-based desktop environments and has a
more neutral design aesthetic than KDE, allowing
it to not stick out on other desktops. It can also be
found in all major package managers thanks to
its popularity and ties to KDE, although options to
compile from source via a tarball or GitHub are just
as available.
Initial startup involves a quick but thorough setup
wizard, going through basics such as photo library
locations, the manner of importing RAW les and a
few more options. While a lot of people can mostly
ignore these and have a decent default selection,
they are useful for people with more specic needs.

digiKam has a selection of different views

Any changes you make during this wizard can of


course be altered in the main programs settings and
such, and theres a lot of extra customisation you
can do to digiKam as well.
The main window allows you to view your photo
library in a stream, with thumbnails and previews,
as well as by their geotags if they have any. Its here
you can edit tags, geotags etc, as well as looking
up metadata and colour maps for the images.
The interface aids in this, allowing you to select
multiple images with the plus symbol that appears
as you hover over thumbnails, although Ctrl and
click also still works.

Selecting an image then allows you to enter the


image-editor window or the light-table view. In the
image editor, there are a lot of functions for adding
effects and general image manipulation, although
its mostly hidden away in drop-down menus,
making it a bit tricky to use. The light table is a great
feature, allowing you to compare and contrast a
couple of photos in case you need to select from
multiple images of the same angle and subject.
Its a really comprehensive piece of software,
offering just about everything wed want from a
photo manager and more, especially for those that
need it in a professional capacity.

SCORES

Q Light table allows you to compare photos to find which is best

Installation

Requires some extra


packages from KDE, but
easy to install otherwise

Ease of use

The main window is easy


enough to use, although
the image editor is a
little confusing

Editing

Not quite at the level


of GIMP, but more than
good enough for a
photo manager

Features

digiKam is overowing
with a variety of features
to make it easier and
better to use

Overall

A fantastic overall
package that offers
specialised tools to
organise, analyse and edit
your photo library

Linux & Open Source Genius Guide 191

tri Spe
al ci
of al
fe
r

Enjoyed
this book?

Exclusive offer for new

Try
3 issues
for just

* This offer entitles new UK Direct Debit subscribers to receive their first three issues for 5. After these issues, subscribers will then pay 25.15 every
six issues. Subscribers can cancel this subscription at any time. New subscriptions will start from the next available issue. Offer code ZGGZIN must
be quoted to receive this special subscriptions price. Direct Debit guarantee available on request.
** This is a US subscription offer. The USA issue rate is based on an annual subscription price of 65 for 13 issues, which is equivalent to $102 at the
time of writing compared with the newsstand price of $16.99 for 13 issues, being $220.87. Your subscription will start from the next available issue.

The magazine for


the GNU generation
Written for you

Linux User is the only magazine dedicated to


advanced users, developers & IT professionals

About
the
mag

In-depth guides & features


Written by grass-roots developers & industry experts

4.5GB DVD every issue

Four of the hottest distros feature every month insert the disc, reboot your PC & test them all!

subscribers to

Try three issues for 5 in the UK*


or just $7.85 per issue in the USA**
(saving 54% off the newsstand price)
For amazing offers please visit

www.imaginesubs.co.uk/lud
Quote code ZGGZIN
Or telephone UK 0844 249 0282 overseas +44 (0) 1795 418 661

Pass our knowledge


off as your own

TM

Know more with world-leading features and tutorials on everything


from WordPress to Photoshop

BUY YOUR COPY TODAY

Print edition available at www.imagineshop.co.uk


Digital edition available at www.greatdigitalmags.com
Available on the following platforms

facebook.com/ImagineBookazines

twitter.com/Books_Imagine

VOLUME 5

Anda mungkin juga menyukai