QuickTime™ and a

decompressor
are needed to see this picture.

Tight Integration of Machine Vision

& Robotics with LabVIEW
for automated assembly,
packaging and testing

Ignazio E M Piacentini
ignazio.piacentini@imaginglab.it
www.imaginglab.it

Giovedi 8 Ottobre 2009, Museo del Patrimonio Industriale, Bologna

Martedi 13 Ottobre, Technotown – Villa Torlonia, Roma

 QuickTime™ and a
decompressor
are needed to see this picture.

Agenda

Basic concepts and requirements

Few application examples

Standards in Machine Vision / lack of standards
in robotics

Direct communication from LabVIEW to the
robots: introducing a robotics library / a robotics
toolkit in LabVIEW

Questions and answers

Tight integration of vision & robotics 2

 QuickTime™ and a
decompressor
are needed to see this picture.

How and why did we
We have a strong background in

start integrating vision machine vision, we saw the
potential of adding vision to robots,
and robotics? we design integrated solutions
‘vision-first’

We strongly believe that there is
an increasing number of
applications suitable for fast, small
payload robots

We purchased our first robot
(anthropomorphic) in 2004:
 We liked everything except the
programming aspects!

We designed and developed our
software library / toolkit in LabVIEW
initially for our own internal use
(Mitsubishi), we have done the
same for the Denso robot (using b-
Cap in strict collaboration with
Denso Europe /Japan), work is
progressing fast with Kuka, other
brands will follow
Tight integration of vision & robotics 3
 QuickTime™ and a
decompressor
are needed to see this picture.

Customers and applications

Our customers are small / medium companies, often with strong
mechanical background …or large companies / institutions

From neutrino detectors, to semiconductors, to packaging, to
cosmetics, …

Automated assembly, testing, packaging, …

Some of the most interesting applications are covered by NDAs and
cannot be shown in this presentation

We will see a limited number of video-clips

All projects are based on LabVIEW + IMAQ + ImagingLab
robotics library

The robotics library for LabVIEW has been deployed on
approx 50 different robotics / vision installations

Tight integration of vision & robotics 4

 QuickTime™ and a
decompressor
are needed to see this picture.

The essence of vision-robotics integration:

the equivalent of LabVIEW 3-icon demo

Acquire the
image

Extract the
information

Download
info to
robot &
execute

Tight integration of vision & robotics 5

 QuickTime™ and a
decompressor
are needed to see this picture.

Flexibility, configurability, usability

We can be very clever designer, use the best available

technology, develop the best software code, but the ultimate
verdict is with the end user / customer, or equally important
with the operator of the automated machine

Robotics and vision need to be considered as a whole single
entity

A clever hardware design must be complemented by an even
smarter software design and implementation

A good level of ‘end user autonomy’ must be designed into the
vision-robotics system from the very beginning

Tight integration of vision & robotics 6

 QuickTime™ and a
decompressor
are needed to see this picture.

Machine vision, GPM, and ….
We will not enter into the details
of machine vision but ..

Geometric and conventional
pattern matching need to be
complemented by other checks

Parts may have different
orientations / views

It is necessary to discriminate
recognizable / pickable / conformal
(QC) parts

Some inherent complexity can be

removed with a ‘smart’ user
interface

The end user must be able to set
pass / fail criteria, vision and
robotics origin and zeros

Auto-calibration tools are a must
Tight integration of vision & robotics 7
 QuickTime™ and a
decompressor
are needed to see this picture.

The vision platform: from panel PC to Smart

Cameras

The hardware selection is dictated

by the application needs:
 Conventional PC
 Industrial Panel PC
 PXI-based industrial PC (Win-based or
RT OS)
 Compact Vision System / Embedded
Vision System (Real Time OS)
 Smart Camera (Real Time OS)

The software platform remains the
same!!

See Smart Cam videoclip

Tight integration of vision & robotics 8

 QuickTime™ and a
decompressor
are needed to see this picture.

A quick intro to 3D imaging

3D imaging has a great potential in robotics applications

Available technologies: laser scanning, stereo-imaging, time of flight

ImagingLab is currently working on 3D and has recently released a
LabVIEW wrapper for the Ranger series of SICK-IVP cameras

Example: random bin-picking (as opposed to palletizing)

Tight integration of vision & robotics 9

 QuickTime™ and a
decompressor
are needed to see this picture.

From 2D to 3D imaging: detecting flaws

Detecting flaws on free-form objects are not very obviously detected.

Maybe this one is clearly a flaw ...

Courtesy of AQSense

But this one isn't so clear !!!

Especially using “traditional” 2D vision

3rd December 2008 Tight integration of vision & robotics

 QuickTime™ and a
decompressor
are needed to see this picture.

From 2D to 3D imaging: comparing with model

If the scanned surfaces could be compared with their respective models,
the “defects” would be made apparent.

Above Tolerance
Courtesy of AQSense

Zero Difference

Below Tolerance

3rd December 2008 Tight integration of vision & robotics

 QuickTime™ and a
decompressor
are needed to see this picture.

From 2D to 3D imaging: alignment, calibration, …

IMPORTANT ISSUEs: geometric calibrartion in 3D

space, merge of multiple images, alignment with
reference model
Courtesy of AQSense

Generally, scanned objects are not

Scanned object with Model

aligned with their respective models

However, if a comparison must be done,

both surfaces should be aligned, up to Before Alignment
a specified accuracy.

Reference or2008
Model Point Cloud After Alignment
3rd December Tight integration of vision & robotics
 QuickTime™ and a
decompressor
are needed to see this picture.

A complex application: testing DIMM memory

modules @ EEMS (ex Texas Instruments) (1’ 35’’
movie)

Fully automated functional testing

of DIMM memory modules

64 interchangeable motherboards/
256 memory slots

2 robots with common working
space (collision avoidance)

Full auto-guide / auto-learn / on-
board camera

Total testing cycle time
compatible with production line

Repetitive project

Tight integration of vision & robotics 13

 QuickTime™ and a
decompressor
are needed to see this picture.

‘Smart’ pick and place for the cosmetic

packaging

Fast picking from
conveyor and accurate
placement (< 0.1 mm)

Many different sizes and
shapes require flexible
imaging configuration tool

Most repetitive project
using inverted Denso
SCARA (22 robots, more
to come ….)

Triggered the
development of the direct
LabVIEW > b-Cap library

Co-design with partner
company

Tight integration of vision & robotics 14

 QuickTime™ and a
decompressor
are needed to see this picture.

‘Smart’ pick and place for the cosmetic packaging /2

Early testing of
precision conveyor
tracking (accurate
on-the-fly deposition)

Assembly of multi-
colour face-powder
trousse

Two robots / three
vision systems

Fast development of
new machines

Tight integration of vision & robotics 15

 QuickTime™ and a
decompressor
are needed to see this picture.

EST Projet (Educare per la Scienza e la Tecnologia):

teaching vision & robotics at the ‘Leonardo da Vinci’
Science Museum in Milan

Un progetto, anzi due,

per il
Museo della Scienza di
Milano

Two ‘machines’ sponsored by NI,

Mitsubishi, IL

Recognising shape and colour,
assembling and disassembling

Interactive

Taking care of an audience of
variable age / designing for zero
or little maintenance

Tight integration of vision & robotics 16

 QuickTime™ and a
decompressor
are needed to see this picture.

Conveyor tracking in pharmaceutical

packaging

Dual line with 2 vision
systems / 4 DENSO
SCARA + 2 FANUC

Large selection of
different items

Curved transparent /
translucid plastic

Multiple pattern
matching / multiple
object views

……

Tight integration of vision & robotics 17

 QuickTime™ and a
decompressor
are needed to see this picture.

Why integrators need to use different

brands of robot

Standards exist for cameras, at least at communication /
driver level: (old analogue standards at signal level, CCIR,
RS170, etc.), IEEE 1394 Firewire, CameraLink, GigE, …

Robots of different brands do not share a common
programming language

Integrators / end user are often faced with the need to ‘mix &
match’ different robot brands
 Based on offer range and characteristics
 Choice imposed by end customer

Functionally all robots are similar (performances, reliability,etc.

not withstanding)

An ‘abstraction layer’ can be built in LabVIEW

Tight integration of vision & robotics 18

 QuickTime™ and a
decompressor
are needed to see this picture.

Three different text-based programming examples

Kuka code example
1 INI
Denso code example

2 PTP HOME Vel= 100 % DEFAULT

'!TITLE "Test Pick&Place" Mitsubishi code example
PROGRAM TEST_PICKPLACE 10 SERVO ON
3 GRIPPER_OPEN () ; Open the gripper
TAKEARM 20 OVRD 100
4 PTP P_PICK_UP Vel= 100 % P_PICK_UP Tool[1] Base[0] ; Move 100 mm above the pick position.
MOTOR ON 30 HOPEN 1 ' Open the gripper.
5 LIN_REL {Z -100} ; Move to the pick position.
CALL GRIPPER OPEN 40 DLY 0.5 ' Open the gripper
6 GRIPPER_CLOSE () ; Close the gripper
APPROACH P,P_PICK,100, S=100 ' Move50
100 mm P_PICK,100
above the pick position.
MOV ' Move 100 mm above the pick position.

7 LIN_REL {Z +100} ; Move 100 mm MOVE

above the pick position.
L, P_PICK, S=50 60 OVRD 50 ' Move to the pick
position.
8 PTP P_PLACE_UP Vel= 100 % P_PLACE_UP Tool[1] Base[0] ; Move 100 mm aboveP_PICK
70 MVS the place position. ' Move to the pick position.
CALL GRIPPER_CLOSE ' Close the gripper
9 LIN_REL {Z -100} ; Move to the place position. 80 HCLOSE 1 ' Close the gripper.
DEPART P,100, S=100 ' Move 100 mm above
the pick position.
10 GRIPPER OPEN () ; Open the gripper 90 DLY 0.5
APPROACH P,P_PLACE,100, S=100 ' Move 100 mm above the place position.
100 OVRD 100
11 LIN_REL {Z +100} ; Move 100 mm above the place position.
MOVE L, P_PLACE, S=50 ' Move to the place
position. 110 MVS,100 ' Move 100 mm above the pick position.
12 PTP HOME Vel= 100 % DEFAULT
CALL GRIPPER_OPEN 120 MOV P_PLACE,100
' Open the gripper ' Move 100 mm above the place position.

DEPART P,100, S=100 130 OVRD 50' Move 100 mm above

the place position.
140 MVS P_PLACE ' Move to the place position.
MOTOR OFF
150 HOPEN 1 ' Open the gripper.
GIVEARM
160 DLY 0.5
END
170 OVRD 100

180 SERVO OFF

Tight integration of vision & robotics 19
 QuickTime™ and a
decompressor
are needed to see this picture.

Three different communication / programming examples:

Mitsubishi set-up

Client
Server

1;1;MOTOR ON

•String assembly
•Motor-on icon
•Generation of a ‘readable’ string
•String sent to the controller (TCP-IP, serial, …)
•The controller ‘parses’ the string and performs the command
Tight integration of vision & robotics 20
 QuickTime™ and a
decompressor
are needed to see this picture.

Three different communication / programming examples:

Denso set-up

Client
Server

Data packet
001010011001010101001
010101010101010000111
101010100010101010101
01001

•Data packet assembly

•Motor-on icon
•Generation of data packet
•Data packet sent to the controller (TCP-IP over Ethernet)
•The controller ‘parses’ the data packet and performs the command
Tight integration of vision & robotics 21
 QuickTime™ and a
decompressor
are needed to see this picture.

Three different communication / programming examples:

Kuka set-up

Client
Server

•VI Server technology (NI)

•Motor-on icon
•Instance of motor icon both at client and server level
•‘Call’ to the LabVIEW program resident on the server
•The controller performs the command
Tight integration of vision & robotics 22
 QuickTime™ and a
decompressor
are needed to see this picture.

A common LabVIEW ‘world’ for robotics, vision, feeders

and HMI

A simplified diagram of a whole application

Select vision recognition algorithm Enable MV quality check Pick & place coordinates

Type of feeder Type of robot Select vision-based

feeder management
Tight integration of vision & robotics 23
 QuickTime™ and a
decompressor
are needed to see this picture.

Building an ‘abstraction layer’: how to handle

different brands of robots

An ‘action’ or command is essentially identical for different brands of
robots

A high level command (or program) common to ‘all’ robots is
translated via a robot-specific plug-in

Command High level LabVIEW robot protocol

Denso Mitsubishi Kuka Staubli XYZ

Protocol Protocol Protocol Protocol Protocol
…

Robot Controller
Tight integration of vision & robotics 24
 QuickTime™ and a
decompressor
are needed to see this picture.

Official release and commercial

availability of LAbVIEW robotics
libraries ….

Denso Robotics Library has been released February
2009 by ImagingLab and will be by NI web services
(ni.com): official announcement on the 25 November
2009 at IREX 2009, Tokio, and SPS-IPC, Nuremberg

Mitsubishi Robotics Library is available now from
ImagingLab

Kuka Robotics Library will be released Q1-2010 (beta
version available now on request)

Other library, special needs, please ask …..

Tight integration of vision & robotics 25

 QuickTime™ and a
decompressor
are needed to see this picture.

Questions &
answers …

Tight integration of vision & robotics 26