OpenCV for iOS

ICVS 2013

Alexander Shishkov
alexander.shishkov@itseez.com
Agenda
1. OpenCV for iOS
2. Hello, World! application
3. OpenCV camera example
Why bother?
Its fun
Its trendy
It can make you rich
Its impressive way to
show some technology
OpenCV for iOS
Started with GSoC 2011 project
Firstly released 2012-07-04 with OpenCV 2.4.2
Thanks to C++ ! Objective C interoperability
native OpenCV for iOS just works (excluding
some parts of highgui module):
working with camera
reading/writing video files
namedWindow, imshow,
Changelog
2.4.2
the first release supporting iOS
2.4.3
added support of armv7s and iOS 6.x
iOS tutorials
multithreading based on GCD (parallel_for_):
denoising
optical flow
CLAHE
cascade detector
background/foreground subtraction
distance transform
generalized Hough transform

Changelog
2.4.6
fixed bug with incorrect video saving with CvVideoCamera
added rotateVideo flag to the CvVideoCamera class
added functions for conversion between
UIImage and cv::Mat
iOS specific part
Camera stack
#import <opencv2/highgui/cap_ios.h>
CvAbstractCamera, CvPhotoCamera, CvVideoCamera
Conversion functions
#import <opencv2/highgui/ios.h>
UIImageToMat, MatToUIImage

iOS specific part (2)
Supported architectures: armv7 and armv7s
excluding iPhone 2G/3G, iPod 1G/2G
Supported operation systems: iOS 5.x, 6.x
No ARM/iOS-specific acceleration yet "
(planned for OpenCV 3.0)
opencv2.framework for simpler integration
iOS Anatomy
Apps
Media (CoreImage, CoreGraphics, OpenGL ES,
AVFoundation)
Cocoa Touch (UIKit etc.)
Core Services (Databases, iCloud, GDC, )
Core OS (Unix, POSIX, )
OpenCV+iOS Anatomy
Apps
Media (CoreImage, CoreGraphics, OpenGL ES,
AVFoundation)
Cocoa Touch (UIKit etc.)
Core Services (Databases, iCloud, GDC, )
Core OS (Unix, POSIX, )
OpenCV for iOS is just another framework
Framework layout
OpenCV Framework
Prebuilt framework
from opencv.org

From source code
cd ~/<my_working_directory>
git clone https://github.com/Itseez/opencv.git
git checkout b 2.4 origin/2.4
python opencv/platforms/ios/build_framework.py ios
Agenda
1. OpenCV for iOS
2. Hello, World! application
3. OpenCV camera example
iOS Development
Device should be provisioned i.e. registered to be
used with the particular developer certificate

iOS Simulator:
+ +
iOS Device: + + + $99 +
Google it: Start Developing iOS Apps Today

iOS Simulator iOS Device
Cost Free* $99 per year
Prerequisites OSX 10.7 + Xcode
4.3.x or later
+ iOS-5 capable
device
Target CPU x86 (SSE, no
Neon)
ARM (Neon, no
SSE)
Speed fast slower
RAM lots of limited
OpenGL + +
Photo Library, Still
Photo processing
+ +
Camera - +
Accelerometer - +
Debugging,
Logging
+ +
Performance
evaluation
+/- +
Objective C
Objective C = Smalltalk + C
Apples take: Objective C++ (can use classes, STL etc.)
Since iOS 5 includes ARC (automatic
reference counting)
Dynamic methods are called by name,
not by indices in vtable
Has very rich class library with dedicated
root object NSObject
.h extension for headers, .m & .mm
extension for implementation
Hello world
#import <Foundation/Foundation.h>
int main(int argc, char *argv[])
{
//printf("\n\nHello, world!\n");
@autoreleasepool {
//NSLog(@"\n\nHello, world!);
NSLog([[NSString stringWithUTF8String:"\n\nHello"]
stringByAppendingString:@", world!"]);
return 0;
}

Its a superset of C
Since OSX 10.7 & iOS 5 memory is automatically managed
#import == #include with automatic guards.
[recipient messageWithArg1:A andArg2:B!];

Cocoa(Touch) key concept: MVC
View is created visually as a
Storyboard element
For each View component (image view,
button, slider) we add corresponding
IBOutlet-marked property in the
Controller class
For each action we add IBAction-marked
method in the Controller
Model-View-Controller
(For simple demos Model
is a part of Controller class.)
View
Controller
Model
update
update
user action notify
Class declaration
#import <UIKit/UIKit.h>
@interface ViewController : UIViewController {
UIImage* image;
}
@property (weak, nonatomic) IBOutlet UIImageView *imageView;
@property (weak, nonatomic) IBOutlet UIBarButtonItem *loadButton;
-(IBAction)effectButtonPressed:(id)sender;
@end
@interface @end delimits class declaration
there must be one and only one base class (UIViewController in our sample)
data members are declared inside {}.
@property declares a property, IBOutlet qualifier denotes outlets.
No need to declare overloaded methods, only the newly added
Actions have IBAction return type
Class implementation
#import ViewController.h
@implementation ViewController
@synthesize imageView;
- (void)viewDidLoad {
[super viewDidLoad]; // call the superclass method
NSString* filename = [[NSBundle mainBundle]
pathForResource:@"helloworld" ofType:@"png"];
UIImage *image = [UIImage imageWithContentsOfFile:filename];
if( image != nil ) imageView.image = image;
}

@end
@implementation @end delimits class implementation
@synthesize automatically generates correct code for reading and writing properties
viewDidLoad is the place to put additional initialization. Xcode generates stub for it
Some methods, including viewDidLoad, require to call the original superclass method
To show the image, just assign it to the image property of UIImageViewer
Lets now add OpenCV!
Hello world
1. Download framework from OpenCV website
2. Add framework to your project
3. Change project language to Objective C++.
4. Add #import <opencv2/opencv.hpp> to the *.pch file.
UIImage ! cv::Mat
UIImage* MatToUIImage(const cv::Mat& m) {

}

void UIImageToMat(const UIImage* image, cv::Mat& m) {

}
These are not super-fast functions,
minimize such conversions
#import <opencv2/highgui/ios.h>
Hello world
Add OpenCV code

if( image != nil ) {
cv::Mat m, gray;
UIImageToMat(image, m);
cv::cvtColor(m, gray, CV_RGBA2GRAY);
cv::GaussianBlur(gray, gray, cv::Size(5, 5), 1.2, 1.2);
cv::Canny(gray, gray, 0, 50);
m = cv::Scalar::all(255);
m.setTo(cv::Scalar(0, 128, 255, 255), gray);
imageView.contentMode = UIViewContentModeScaleAspectFit;
imageView.image = MatToUIImage(m);
}
Live demo #1
25
Agenda
1. OpenCV for iOS
2. Hello, World! application
3. OpenCV camera example
Adding Camera input
Camera is not supported by Simulator "
Setting up camera, retrieving frames,
displaying them, handling rotations etc.
takes a lot of code. Thats why OpenCV
encapsulates this logic.
Its built using delegate pattern. The
delegate will be our ViewController.
Camera interface
Update ViewController interface part
#import <UIKit/UIKit.h>
#import <opencv2/highgui/cap_ios.h> //or ios.h
using namespace cv;

@interface ViewController : UIViewController <CvVideoCameraDelegate>
{
UIImage* image;
CvVideoCamera* videoCamera;
}
@property (nonatomic, retain) CvVideoCamera* videoCamera;


Protocols and delegates are not exclusive things in standard Cocoa
components; well-designed components may introduce custom protocols
and use delegates. See cap_ios.h for details.
ARC works with pointers to your own classes as well (e.g. with
CvVideoCamera*)
Camera interface
Setup camera in ViewController viewDidLoad method
- (void)viewDidLoad
{
[super viewDidLoad];
self.videoCamera = [[CvVideoCamera alloc] initWithParentView:imageView];
self.videoCamera.delegate = self;
self.videoCamera.defaultAVCaptureDevicePosition =
AVCaptureDevicePositionFront;
self.videoCamera.defaultAVCaptureSessionPreset =
AVCaptureSessionPreset352x288;
self.videoCamera.defaultAVCaptureVideoOrientation =
AVCaptureVideoOrientationPortrait;
self.videoCamera.defaultFPS = 30;
}
CvVideoCamera class is super-easy to use. Leave the default
resolution, fps etc. or customize them as needed
Using lowest possible resolution and reasonable frame rate can save a
lot of power and make apps more responsive
Camera interface
Modify button callback
- (IBAction)captureButtonPressed:(id)sender {
[self.videoCamera start];
}

- (void)processImage:(Mat&)image;
{
// Do some OpenCV stuff with the image
Mat image_copy;
cvtColor(image, image_copy, CV_BGRA2BGR);

// Invert image
bitwise_not(image_copy, image_copy);
cvtColor(image_copy, image, CV_BGR2BGRA);
}
iOS performance tips
Do expensive initializations once
(e.g. in NSViewControllers viewDidLoad
method)
Use smaller resolution and FPS
Plan your work with memory, avoid redundant
memory allocations and copies (reuse buffers)
Profile your apps using Xcode profiler
But avoid premature optimization
iOS performance tips (2)
Use Accelerate.framework
Use OpenGL and CoreGraphics for rendering,
CoreImage for image processing
Use Grand Dispatch Central (implements data-
level and task-level parallelism)
Use NEON SIMD intrinsics & OpenGL ES shaders
Live demo #2
33
Multi-threading


Backends:
Intel Threading Building Blocks (Intel TBB)
Microsoft Concurrency Runtime
OpenMP
Grand Central Dispatch (GCD)
C=
void parallel_for_(const Range& range, const ParallelLoopBody& body);

class LoopBody : public cv::ParallelLoopBody
{
public:
void LoopBody::operator() (const cv::Range& range) const
{}
}
Future plans
Improve documentation and tutorials
Add iOS samples
Publish few applications in App Store
Support iOS 7
Resources
http://developer.apple.com
(including excellent introductory guides to
Objective-C, Cocoa and iOS programming)
http://docs.opencv.org/master/doc/tutorials/ios
(iOS tutorials for the latest OpenCV version)
http://code.opencv.org/svn/gsoc2012/ios/trunk
(our GSoC 2012 iOS project repository)
http://answers.opencv.org
(Q&A forum for OpenCV)
Resources
Coming soon
OpenCV for iOS minibook
Chapters:
Getting Started with iOS
Displaying Image from Resources
Linking OpenCV to iOS Project
Detecting Faces with Cascade Classifier
Printing Postcard
Working with Images in Gallery
Applying Retro Effect
Taking Photos From Camera

Thank you!

Any questions?
alexander.shishkov@itseez.com
admin@opencv.org