Mosaics of Scenes with Moving Objects

James Davis Computer Science Department Stanford University

Mosaics of Scenes with Moving Objects - CVPR98

Motivation

Panoramic imagery Large high resolution images

Mosaics of Scenes with Moving Objects - CVPR98

Overview
Registration algorithms do not account for moving objects. Use phase correlation and estimate correct projective geometry.

Mosaics with many images are ruined by accumulated registration errors. Find many local registrations and solve a linear system to obtain global registration.

The final mosaic is blurry in regions of motion. Segment the mosaic into disjoint regions and fill each from a single source image.
Mosaics of Scenes with Moving Objects - CVPR98

 Pairwise Registration

@UV " @ @U S 6TE ( S 3 ' 01

 ! 66VC 2 & @9 Y 6 " X V`! " "  ! D

Moving Objects

W0

Global Registration

766   2 # C 2 Q#P   ) 1 &$ I "  0 B   $ # @A $ )0 69 "    (#( ( ) ' 8 67  & #% & #( 0 )$ G&H #$  $ 2 0  5 & %2 # #   " #(  ) ! 4)  ) F       ( !' )#  0 E 3  0#  $1# # D

 )01

R & #$

) # & 0& )0 R! D  6T6 C) % 1

 0# 2  # 6  6VC  6V 2B $ E! #$ 3 S  "  0 2     D )) 0 ` )  0) ! )   0& # c a  & 'b X # P  $ G # # D &0 B (( '  &

&

 $ U6 )0 6C   & &0 $ ' 1 B  R @d 0 A" & TC  0# A  85 A H &) AC () SDB  # E (  0 S3  0# )   B P  &  0 X  0 P 2  G  ( 2 #  ' 0

Related Work

3  &# D &0 #$ S  #) 0# E

Mosaics of Scenes with Moving Objects - CVPR98

)0

0

 #

Pairwise Registration

Fixed center of projection

Parameter estimation

Robust registration with moving objects

Mosaics of Scenes with Moving Objects - CVPR98

Phase correlation recovers translation

I2(x, y) = I1(x-x0, y-y0) F*[I2]F[I1] |F *[I2]F[I1]|

Mosaics of Scenes with Moving Objects - CVPR98

x0

(x-x0, y-y0) = F-1

Phase correlation is not biased by moving objects

(a)

(b)

(c)

(d)

Mosaics of Scenes with Moving Objects - CVPR98

Mellin transform recovers rotation and translation

Extends phase correlation Polar transform converts rotation into translation Assumes orthogonal projection

Mosaics of Scenes with Moving Objects - CVPR98

Finding the projection matrix

f x0 x0

Possess 2D parameters (x0, y0, 0)

Desire 3D Euler rotation angles (, , )

f
2 2

Assume small angular rotation

f + x0 , ) 0 y0
I2

A = C-1RC

x0 ( , , ) = ( 2arctan , 2arctan f

A : Image plane projection matrix

f2
x

C : Intrinsic camera matrix

f1
f

R : 3D rotation matrix

I1

Mosaics of Scenes with Moving Objects - CVPR98

Without global registration errors accumulate

Mosaics of Scenes with Moving Objects - CVPR98

Relating pairwise and global registration

Reference image
Image 1
Im age 2

P1 A 21 P2

Aij Pj = Pi Given all Aij , find all Pk

Aij : Pairwise projection of image i onto image j Pk : Projection of image k onto global reference plane
Mosaics of Scenes with Moving Objects - CVPR98

Globally registered mosaic

Mosaics of Scenes with Moving Objects - CVPR98

Compositing

Blending produces blurring How do we avoid this?

Mosaics of Scenes with Moving Objects - CVPR98

Segment the mosaic

A single source image per region Avoiding artifacts along boundaries?

Mosaics of Scenes with Moving Objects - CVPR98

Finding boundaries

Avoid contradictory information Relative difference image Minimum difference path

Mosaics of Scenes with Moving Objects - CVPR98

Mosaic without blurring

Mosaics of Scenes with Moving Objects - CVPR98

Global registration comparison

Mosaics of Scenes with Moving Objects - CVPR98

Compositing comparison

Mosaics of Scenes with Moving Objects - CVPR98

Summary

Contributions

Mosaics of Scenes with Moving Objects - CVPR98

Discussion

Quantify small angle approximation Matrix elements do not have uniform scales Avoid segmenting into tiny regions Moving objects must appear in one image

Mosaics of Scenes with Moving Objects - CVPR98