Touchscreen Support in BREW® and UI

Widgets
Sean Rogers, Staff Engineer/Mgr
Qualcomm Incorporated
 Introduction

2
Good afternoon
• My name is Sean Rogers
• My company is Qualcomm
• M di
My division
i i iis Q
Qualcomm
l CDMA T
Technologies
h l i
• My job is BREW
• My project is Widgets
• My favorite color is Blue

3
 Overview

4
Overview
• Touchscreens 101
• Software architecture
• BREW Touch
T h Intro
I t
• BREW Touch Support
• BREW UI Widgets Intro
• BREW UI Widgets Basic Touch Support
• BREW UI Widgets
g Advanced Touch Support
pp
• Demo
• The Future
• Q&A

5
 Touchscreens

6
What is a touchscreen?
• A touchscreen is a screen…
…you can touch

Resistive
Surface acoustic
Capacitive
Infrared
Strain gauge
Optical imaging
Dispersive signal
…Mouse wheels
…Pixie dust
…Flux capacitors
p

• Two or more layers including

• A screen
• O or more touch
One t h panels
l

7
Using a touchscreen
• There ordinary
Many are two
alsopopular
items
items can
that
items
be
should
used
usednot
on
onbe
aa touchscreen
touchscreen
used on a touchscreen
– Plastic
Hunting
A stylusspoon
Knife
– Chopsticks
Sharpie
A finger Brand
(multi-touch
(multi
Permanent
touch sushi
Markers
game anyone??)

8
 Architecture

9
The Touchscreen Stack

Application Layer Application receives

Touch Applications events
• And goes wild!

BREW Layer BREW manages events

Touch Event Management • And sends them to application
Touch Event Queue

Kernel (REX, L4, POSIX, …) Kernel processes

Native Application Task signals from hardware
Touchscreen Driver • And forwards events to BREW

Hardware Layer Touchscreen delivers

Touchscreen Signals signals to kernel
• At an average rate of 6ms

10
The Touchscreen Events
• Consistency and Flexibility
– Structures dangerous for expansion
– Arbitrary parameters expensive on performance
• EVT_POINTER_ (UP/DOWN/MOVE)
– “x=0000001F,y=0000002E,time=00001436,clkcnt=2”
– String based, fixed offset standard parameters
• X,
X Y position
iti iin screen coordinates
di t
• Time in milliseconds when event was recorded
• Number of clicks in last .5 seconds
• Pointer identification (multi-touch
(multi touch support)
– Helper functions provided for all standard parameters
– Flexible offset custom parameters
• EVT_PEN
– Deprecated
– Sent if EVT_POINTER is not handled for compatibility

11
 …A day in the life of handling events

BREW: Okay touchscreen, lets see some events!

Driver: First event on it’s
it s way!
Horizontal measurement: 37,546
Vertical measurement: 198,436
BREW: ?!? This is a QVGA device!

12
 Calibration

13
Calibration Necessity
• Calibration is necessary to turn driver coordinates...
…into screen coordinates
11
1,1 x

• This can include

• Offsetting
• Scaling
• Rotation
• Skewing
• …and d worse

4800,320
240, 6400

14
Calibration in BREW
• The software stack can include calibration components

Application Layer
Touch Applications

Calibration Application

BREW Layer
Calibration Settings
Application Kernel uses Apps receive
Touch Event Management
sets calibration calibrated
Touch Event Queue calibration settings events
•Comparison
p of •Shared •Calibration
raw events between app can be
Kernel (REX, L4, POSIX, …) and fixed BREW and run again as
Native Application Task display targets kernel necessary
Touchscreen Driver

Calibration Logic

Hardware Layer
Touchscreen Signals

15
…Another day in the life of handling events

BREW: Okay now we’re really ready for some events!

Driver: First event on it’s
it s way!
Horizontal measurement: 37,546
Vertical measurement: 198,436
Kernel: No problem! I can calibrate the event. Here
you go BREW!

BREW: Thanks kernel! Now I’ll just deliver this event to the
application, get ready to draw, and…

Driver: While you were doing that 342 more events came in.

16
Stale Events and Disposal

17
Stale Events
• There is a need for touch event indications
– Applications need to know if more events are pending
• BREW features an event for this purpose
– Stale move event: EVT_POINTER_STALE_MOVE
– Event has same format as EVT_POINTER_MOVE
– Event indicates that one or more MOVE events remain in the queue
– UP DOWN events
UP, t remain i unchanged
h d
– Allows applications to throttle MOVE event handling as necessary
• Sometimes applications cannot keep up
– There is still a need for BREW event disposal

18
Event Storage and Disposal
EVT_POINTER_DOWN

• BREW limits move EVT_POINTER_STALE_MOVE

event storage
BREW Queue (50 events) EVT_POINTER_STALE_MOVE

Storage EVT_POINTER_STALE_MOVE

EVT_POINTER_STALE_MOVE

• Old MOVE event EVT_POINTER_STALE_MOVE

deleted
EVT_POINTER_STALE_MOVE

BREW • All UP, DOWN events

Overflow remain EVT_POINTER_STALE_MOVE

EVT_POINTER_STALE_MOVE

EVT_POINTER_STALE_MOVE
• Alternate MOVE
events deleted EVT_POINTER_STALE_MOVE

BREW • Maintains continuity

continuity, EVT_POINTER_STALE_MOVE

Maintenance sacrificing granularity

EVT_POINTER_MOVE

EVT_POINTER_UP

19
 BREW Support

20
BREW Support Summary

IControl support
• IControl objects support minimal pointer event handling
• IControl
IC t l objects
bj t iinclude
l d MMenu, Li
List,
t S
Softkey,
ftk T
Text,
t IImage, D
Date,
t and
d St
Static
ti

IDialog support
• IDialog supports pointer event handling
• Events will be forwarded to proper controls as needed

Application support
• Unhandled events sent to application through HandleEvent
• Application
A li ti can override
id Dialog
Di l and d Control
C t l handlers
h dl ffor custom
t h
handling
dli

21
BREW UI Widgets Support

22
Introduction to Widgets: MVC
Widgets are implemented using the Model-View-Controller pattern

The
e Model
ode
• Models hold data
• Example: text strings, selection range, and caret position
The View (AKA The Widget)
• Views show data
• Example: Draw text to the screen
The Controller
• Controllers manipulate data
• Example: Translate ‘2’ key into A, B, or C
Views also store Controller and Model, using them as needed
• Example: Passing events to Controller
• Example: Listening to model for changes requiring drawing

23
Introduction to Widgets: MVC Example
MVC pattern applied to Text Entry

Hello BREW 2008!

• TextWidget
g ((View)) receives events
• Forwards to Controller
• TextController handles events
• Manipulates Model
• TextModel stores new data Widget
• Notifies listeners of changes Primary POC
• TextWidget receives notification View
• Refreshes the View

Model Controller
Data Manager Event Handler

24
 …A day in the life of Widgets

User: <drags across some text> Okay, let me just select

this phone number…
W: What are all these touch events? TextController, can
you help me out here?
C: Sure thing. I just need to know what text was drawn
here so I can select it.
W: I don’t remember, that was a long time ago…

C: It was 20 milliseconds ago!

M: Did someone say selection? I know about selection!

C: Not now TextModel!
User: Hi Tech Support? My phone has gone crazy and is
arguing with itself…

25
Elements and Touch Controllers

26
Touch Support in Widgets: MVC
Touch support in an MVC framework has challenges
• Controllers shouldn’t know about view details
• Views shouldn’t
shouldn t handle events
Some widgets are simple
• Touch a CheckWidget: Check it
Oth widgets
Other id t are nott
• Touch a ScrollWidget: ???

9
27
Touch Support in Widgets: MVC 2
To uphold the MVC pattern, two new concepts were introduced

The Element
• Widgets now publicly expose each individual object (element) they draw
• Example: WETYPE_SCROLL_VHANDLE (Vertical scroll handle)
• Widgets also expose useful information about the element
• Example: Local element rectangle coordinates within widget

The Touch Controller

• Touch controllers are designed specifically for pointer event handling
• Touch controllers co-exist with other attached controllers
• Example: Key controllers
• Touch controllers work off of element information provided by widget

The result
• Public API based touch support which maintains MVC pattern

28
 …Another day in the life of Widgets

App: Touch event from BREW on its way!

Controller: No problem, I can handle it. Widget, what
element is at location 145, 78 ?
Widget: That element is… umm… slight problem…

Controller: *Grumble* What is it?

Widget: I’m only 30 pixels wide!

29
Containers and Localization

30
Introduction to Containers
There are two primary types of Widgets:
• Parents (Containers) and Children (Widgets)
These two types form a hierarchical tree of the widget application
• Containers store relative widget locations
• Containers are widgets to their parent container as are siblings
• Events traverse through the tree until they reach their destination

Container
40,10

20, 60 Check it!

Container Container

Image Image

Text Checkbox
9
31
Containers and Localization
Touch event traversal presents a new challenge

With key events, traversal is simple

• Focus determines traversal route

With touch events, traversal requires two steps

• Containers must determine which child should receive the event
• Containers must localize the event to children coordinates
• This process repeats at each tree level until destination widget is reached

32
Localization Example
User clicks on the screen
• Pointer event generated and sent to application
x=00000032,y=000000AF,time=00001436,clkcnt=1
x=0000000A,y=0000000A,time=00001436,clkcnt=1
x=00000028,y=00000064,time=00001436,clkcnt=1
• Roott container
R t i finds
fi d child,
hild llocalizes
li event,
t andd fforwards
d th
the eventt
• Next container finds child, localizes event, and…
• Checkbox receives event, controller handles it, and the box is checked!

Container

Container

Checkbox
40,100
50,
10,175
10 9
33
 …Another day in the life of Widgets

App: <Creates some widgets…>

Widgets: Okay, lets get ready for touch support!
Container handling: Check.
Touch controllers: Check.
El
Elements:
t Ch k
Check.
Localization logic: Check.
Alright, lets boogie!
App: BTW this device doesn’t have a touchscreen

34
 Initialization

35
Initialization Support
Applications will not always want Widgets to be ready for touch events
• If no touch screen is present, memory footprint and creation cost of touch
controllers and element storage should not be incurred
• L
Legacy applications
li ti may nott wantt undesired,
d i d untested
t t d ttouch
hbbehavior
h i on
devices which also have a keypad

To offer control over touch handling, g, widgets

g must be initialized
• Applications initialize widgets by sending EVT_WDG_TOUCH_ENABLE
• This event triggers touch controller creation, localization calculation, and
element caching
• When
Wh applied li d tto th
the root-most
t t container,
t i this
thi eventt will
ill ttraverse th
the entire
ti
tree, initializing each widget along the way
• As future widgets are inserted into their parent this state is maintained,
allowing a single initialization event to last the lifetime of the application

36
 Basic Widget Support

37
Widget Support Summary
Basic widgets with touchscreen support

• ButtonWidget
g
• Multipurpose pressed state management container
• Containers
• Child determination and localization
• CheckWidget
• Check box and Radio button support
• TabWidget
• Tab support for multiple children
• SoftkeyWidget
• Soft key click management, app notification
• Slid Wid t
SliderWidget
• Slider bar with handle jump/drag
• DateTimeWidget
• Date and time field selection,
selection up/down arrow click support

38
Widget Support Summary
Enhanced widgets with touchscreen support

• TextWidget
g
• Caret positioning
• Character selection (in place and scroll-by selection capture mode)
• Word and paragraph selection (double and triple-click)
• ListWidget (Lists and Grids)
• Pixel-by-pixel and item-by-item scroll support
• Item selection and activation (click and double-click)
• Live scrolling (drag item to scroll list)
• Flick support (flick to scroll up/down with deceleration support)
• Intelligent selection determination (item selection vs. list scrolling)
• ScrollWidget
• 4 way scrolling via scroll handles, arrows, and scroll area
• Jump, page, item, line, and pixel based scrolling with repeat logic

39
Advanced Widget Support

40
Touch Observers
Observations about streams of touch events can be useful
• How fast (pixels / second) is the pointer moving?
• How far has the pointer moved since the DOWN event?
• What angle was the pointer moving when it left the screen?

To support this across many widgets, the Touch Observer concept was created
• Observers can be attached to the root most container
• Observers receive all touch events
• Observers are passive, they do not handle events on behalf of the widget
• Observers can be q queried by
y applications,
pp , widgets,
g , or touch controllers
• Useful information in observer frees up calculations by each controller

Observer Container

Container List

41
Flick Support
Flicks are coordinated through Controllers and Observers
• Observer stores useful information like pointer speed and direction
• Controller handles events and, upon POINTER_UP, queries observer
• Information is used to start a flick transition on list, including deceleration
• Controller maintains ability to stop transitions, as on POINTER_DOWN

Observer Container

Controller List

42
Propagation and Transparency
Touch events are intelligently propagated through the widget tree
• Containers can store overlapping children in predefined Z-order
• Events are sent to the top-most widget under the pointer

Transparency properties can control Z-order propagation

• Allows widgets further down the Z-order to receive events instead
• Offered in 3 modes (always transparent
transparent, content based
based, always opaque)

Container

Widget

Widget

43
Customization Support
Widgets have a number of methods to customize touch handling
• Widgets come with a default set of controllers for primary widget types
• Manufacturers supplied controllers will be used by default if they exist
• Applications can replace observers or controllers at runtime for any widget
• Default controllers can be applied to custom widgets of similar type

OEM
App
Observer Container
Observer

App
OEM App
Controller
Controller List
List

44
 Touch Demonstration
(AKA How to Crash the BREW Simulator)

45
 Future Support

46
 …A future day in the life of BREW

User: I see you have claimed *some* support for

touchscreens
BREW: We like to think we support the basics, yes.
User: So when can I expect multi-touch handwriting
recognition support?
BREW: Beg your pardon?
User: I write with both hands. At the same time.
BREW: We’ll get right on that feature then. Would you also
like a mid-air hover feature or do you have a third
hand to hold the phone for you?
User: Hover would be great, thanks.

47
Future Touchscreen Support
What are some of the things we are looking at next?
• Multi-touch support (sizing, moving, rotating, zooming)
• Hardware acceleration (advanced gesture recognition, panning, zooming)
• Drag and drop, Multi-window management
• Multiple item selection (lists)
• Virtual keypads
• Gesture and handwriting recognition engine support
• Multi-touch handwriting recognition engine support
• Quadruple click support
• Quintuple
p click support
pp
• Mood-screen support
• Multi-touch chopstick based sushi game (2009)
• Plugins for soy sauce and wasabi (2010)

48
Questions and Answers
I will now pretend to know the answers to any questions you have

Frequently
q y Asked Questions:
Q: How did BREW get to be so good at touchscreen support?
A: We employ hundreds of trained ferrets who write solid C code

Q: Why have we seen some of these features on competitive platforms?

A: Several senior ferrets were kidnapped and interrogated by competitors
There was nothing we could do
Luckily they escaped and are now safe

• Open Questions

49
Thank You

And enjoy
j y the rest of the BREW 2008 Conference

50