Distributed Multimedia Synchronization

Based on Fuzzy Causal Relations

Luis Alberto Morales Rosales
Congreso Internacional de Informtica Aplicada
Misantla, Veracruz, a 25 de Abril del 2012
2
Overview

Practical Aspect (application area)

Distributed multimedia synchronization in real time
Distributed sources
Heterogeneous data (discreet and continuous)
Loss of data
Transmission delay

Theoretical Aspect
Development of a flexible causal relation for distributed systems
unlike the one proposed by Lamport (1978)
For applications where certain degradation of the system is allowed
is not necessary to assure a strict causal delivery, for example,
multimedia, scheduling, cooperative work and planning
INAOE 3
Outline
1. Introduction
2. Related work
3. Problem description
4. Research proposal
5. Results
6. Conclusions
4
Introduction

Distributed systems

Absence of a global clock

Concurrency

Fail tolerance

Examples: cooperative applications, mobile systems,

multimedia applications, etc.

Distributed multimedia systems

The exchange of big volumes of multimedia data in a
communication network among a group of participants
5
Introduction

Multimedia synchronization

The preservation of temporal dependencies

among the application data from the time of
generation to the time of presentation
Multimedia
Text, slides, images, etc.
Video
Audio
Multimedia data
6
Introduction
A
A
A
P1
P2
P3
V
V
A
V
V
V
V
S S
A- Audio (Voice)
V- Video
S- Slides
P1 Participant 1
P2 - Participant 2
P3 - Participant 3
Example of a multimedia scenario
7
Introduction
Representation of a multimedia scenario
A
V
A
V
V
S
P
1
P
3
P
2
A- Audio (Voice)
V- Video
S- Slides
P
1
Participant 1
P
2
- Participant 2
P
3
- Participant 3
Time Line
Related Work
Synchronization
Synchronous
Asynchronous

Common reference

Physical clock
Disadvantages

Bottlenecks

Introduction of random delays

Not scalable

Logical dependencies
Disadvantages

Introduction of random delays

Discard of messages

Halt of the system

8
9
Related work: Synchronous

Works that use fuzzy logic

Zhou et al. [6]

Temporal model based on fuzzy petri networks to

represent the multimedia synchronization

Video on demand

Ramaprabhu et al. [3]

Broadcast transmission of video on demand

Consider the parameters: available bandwidth, network

delay and buffer space
10
Related work: Synchronous (Cont.)

Works that use fuzzy logic

Coelho et al. [1]

Methodology for the high level specification and

decentralized coordination of temporal interdependences
among objects of multimedia documents previously stored

Membership function to calculate the lifetime of the data

Strict causal relation for event ordering

Global time reference

11
Related work: Asynchronous
Causal and -causal algorithms
The causal and -causal algorithm use the causal relation
proposed by Lamport
Main works: Baldoni [2] et al., Tomoya et al.[5] and Pomares et al. [4]
Introduce delays and/or discard of messages (packets)
bc
Strict delivery order -causal order
bc and
c
<
max
( b, c)
Infinite halt
( b, c )
Deliver c and discard b
a b c
Time Line
12
Context problem

Distributed multimedia synchronization in real

time by considering:

Asynchronous sources

Heterogeneous data

Communication network characteristics

Loss of the data and transmission delay

Without previous knowledge of the system behavior

How can we ensure the temporal dependencies
among events in this kind of enviroments?
13
Research proposal

As hypothesis we claim that for systems where some

degradation of the system is allowed, it is not
necessary ensuring a strict causal order of the data,
which is associated with a binary value

We propose the fuzzy causal relation and the fuzzy causal

consistency to relax the order delivery

We show that relaxing the causal delivery order of the data

can improve the performance of the synchronization
14
Fuzzy causal relation (FCR) denoted by a b
The FCR is based on a notion of distance among the events
The distance can be established considering three main domains: spatial
(R
S
), temporal (R
T
)

and/or logical(R
L
)
Using the notion of distance, the FCR establishes a cause-effect
degree that indicates how long ago an event a happened before
an event b

One assumption considered for the FCR is that closer events

have a stronger cause-effect relation, according to the addressed
problem

Results
Fuzzy causal relation (FCR)

The distance between events is determined by the fuzzy

relation DR: E E [0, 1]
DR(a,b) = R
S
R
T
R
L
Where:
R
S
=(R
1
R
2
... R
s
)
R
T
=(R
1
R
2
... R
t
)
R
L
=(R
1
R
2
... R
l
)
R
i
= membership function

The fuzzy union operator chosen for intra and inter domains
is the max operator
15
Results (Cont.)
Fuzzy causal relation (FCR)

Distance relation
The DR grows monotonically and it is directly proportional to the
spatial, temporal and/or logical distances between a pair of
events
DR(a,b) with a value tending to zero indicates that the events a
and b are closer
The DR cannot determine precedence dependencies among
events, it only indicates certain distance among them
16
Results (Cont.)
Fuzzy causal relation (FCR)
17
Results (Cont.)
Fuzzy causal relation (FCR)

The fuzzy causal relation on a set of events E satisfies

the two following conditions:
1. a b If a b 0 DR(a,b) < 1
FCR between two events
2. a c If b |abc DR(a,b)

DR(a,c) : DR(a,b), DR(a,c) < 1
Transitivity

The fuzzy concurrency (FCNR) is defined as follows:

a b If (ab ba) ( (DR(a, b)= DR(b, a) )< 1)

The value of the fuzzy causal relation

between a pair of events is represented as: FCR(a,b)
18
Results(Cont.)
Example of the FCR and the FCNR
a e If b |abe DR(a,b)

DR(a,e) : DR(a,b), DR(a,e) ) < 1

FCR(a,e)
a
b
c
d
e
f
1
2
Example of fuzzy precedence among causal messages FCR(a,e)
19
Results(Cont.)
Fuzzy Causal Consistency

The Fuzzy Causal Consistency (FCC) is based on the

FCR

The goal of the FCC is to indicate how good the

performance of the system is in a certain time

The meaning of the performance can be indicated

according to the problem to solve
20
Results(Cont.)
Fuzzy Causal Consistency
c
d
w
FCR
p
(b,a)
b
H(a)
Fuzzy causal
consistency
FCC
p
(a)
FCR
p
(c,a)
FCR
p
(d,a)
FCR
p
( w,a)

) (
) (
) (
) , ( ) (
) (
a H b
a H b
p
p
b GP
a b FCR b GP
a FCC
Calculation of the Fuzzy causal consistency
a
b
c
d Average
Weight
FCR
p
( b,d)
FCR
p
( c,d)
FCCp(a)
Example of calculation of the FCC
GP(b) is a weighting degree used to determine priorities or weight
for every fuzzy causal relation when it is needed
Results (Cont.)
Fuzzy causal delivery order

The fuzzy causal delivery order must satisfy the following

condition:
If send(m) send(m) then
p dests(m)dests(m), FCC
p
(m) FCC
max
delivery
p
(m) delivery
p
(m) or
delivery
p
(m) delivery
p
(m)
21

22
Results
Causal order vs fuzzy causal order
a
+
d
-
Strict order delivery -causal order delivery
Fuzzy causal order
delivery
a
+
d
-
y
d
<
max
a
+
d
-

( a
+
, d
-
)
Infinite halt
( a
+
, d
-
)
Deliver d and discard a
Deliver: (a
+
, d
-
) or
(d
-
, a
+
)
Synchronization Period
k =Part(Z)
Example of a fuzzy causal delivery
a-

a
+
a
+
d
-

c
-

a
+
a
-
=
x
1
d
-
=
y
1
D
c
+
=
x
n
b
+
=y
p
A
C
B


Causal delivery
order of events
i
=Part(X)
d
-
Fuzzy causal delivery
a
+
j =Part(Y)
23
Temporal
Distance
R
N
Causal
Distance
R
D
FCC
p
(a)
System
Component
of input
variables
Current State
of
the System
Adjustment
and
Selective
Discard
Weighting
Degree
GP
Fuzzy Causal
Relations
Fuzzy
Causal
Consistenc
y
Fuzzy Causal Component
Fuzzy Control
System
Performanc
e of
the System
Scheme of the Distributed Multimedia Mechanism
FCR
p
(m) m H(a)
Fuzzy Control
Synchronizati
on
Model
Network
Conditions
NC
24
Synchronization model
i j
i =Part(X)
j =Part(Y)
Synchronization periods
F
A
B F
E
E C
D
Fuzzy causal delivery
Formally a period is defined as: (e, f) (E x F), e f e f
B F E D C E F A
I I I I
' ' ) | | | (

EF
25
Synchronization model
26
Results (Cont.)
The FCR applied to the intermedia synchronization

The FCR for the intermedia synchronization problem gives a

qualitative measure of the temporal and logical dependencies
between two events with regard to a partial view of a
participant

The value of the FCR is calculated by:

) , max( ) , (
if ) , ( ) , (
N D
R R b a DR
b a b a DR b a FCR

Results (Cont.)
The FCR applied to the intermedia synchronization

The meaning of the values obtained by the FCR(a,

b) is as follows:

When FCR(a, b) tends to zero indicates that the events a

and b are closer, which means that the network present
low transmission delay and low loss of messages

When FCR(a, b) tends to the unit indicates the opposite

27
28
Results (Cont.)
FCC applied to the intermedia synchronization

FCC gives a qualitative measure of the synchronization

error, according to the temporal and logical dependencies
in the whole system in a certain time, with regard to the
partial view that every participant has
When FCC
p
(a) tends to zero, this indicates that the
performance of the system is good
When FCC
p
(a) tends to the unit, this indicates that the
system performance is regular or bad
The actions that the control will be able to execute are:
the immediate deliver of the received message a or its delay, and
the determination whether a selective discard of the messages contained
in the causal history of the message a is carried out
The Mandamis model is used as inference mechanism
Triangular membership functions are included as part of the fuzzy
control
Fuzzy
Control
System
= delivery time
Current State
of the System
(CS)
Fuzzy Causal
Consistency, FCC
p
(a)
Network Conditions
NC(a)
Diagram of the fuzzy control system
Results (Cont.)
Fuzzy control system
29

The NC determines the network condition

The NC consider: transsmision delays, jitter among messages, loss of
messages, network congestion and bandwith available
When the value of NC(a) is near to zero, this means that the network
conditions are good
When NC(a) tends to one, this represents that the network conditions
are bad
Results (Cont.)
Test of a multimedia scenario

The Host from W to Y

send:
Audio
Video, and
Animation

Host Z only listens

We simulated three
scenarios:
Soft, Medium and Hard

100,000 simulations

Fuzzy causal order vs.

-causal order
Host
W
Host
X
Host
Z
Host
Y
30
When Lamports relation is used the system halts if a message is lost
Results (Cont.)
Test: Fuzzy causal order vs. -causal
31
Soft case
Medium without discard
Big without discard
Classification of delivered messages
Results (Cont.)
Test: Fuzzy causal order vs. -causal
INAOE 32
Soft case
-Causal Order
Fuzzy causal
order
Classification of messages loss
33
Results (Cont.)
Test: Fuzzy causal order vs. -causal
Medium case
Classification of delivered messages
Small with
discard
Medium without
discard
34
Results (Cont.)
Test: Fuzzy causal order vs. -causal
Medium case
-Causal Order
Fuzzy causal
order
Classification of messages loss
35
Results (Cont.)
Test: Fuzzy causal order vs. -causal
Hard case
-Causal Order
Fuzzy causal
order
Classification of messages loss
Conclusions

The definitions of FCR and FCC permits to establish a more

asynchronous ordering for application where certain
degradation of the system is allowed

The FCO allows a more asynchronous delivery of events

compared with the causal delivery order based on the
happened-before relation introduced by Lamport

The FCR and FCC allow a measure of the performance of

the application by a participant at runtime without halt the
system
36
Conclusions

A novel fuzzy control for intermedia

synchronization that works in a distributed
manner was presented

By using the fuzzy control and FCC the

mechanism discard fewer messages than the -
causal mechanism
38
References
1. Andr L. V. Coelho, Alberto B. Raposo, Ivan L. M. Ricarte. Bringing Flexibility to
the Specification and Coordination of Temporal Dependencias among Multimedia
Components. VII Simposio Brasileiro de Sistemas Multimdia e Hipermdia,
Florianpolis, Brazil, SBC. 2001, pp. 37-52.
2. Roberto Baldoni, Achour Mostfaoui, Michel Raynal. Efficient Causally Ordered
Communications for Multimedia Real-Time Applications. The 4th International
Symposium on High Performance Distributed Computing (HPDC '95),
Washington, DC, USA, August 2-4, 1995, pp. 140-147
3. Ramaprabhu Janakiraman, Marcel Waldvogel and Lihao Xu. Fuzzycast: Efficient
Video-on-demand over Multicast. Proceedings INFOCOM 2002, New York, NY,
USA, June 2002.
4. Saul E. Pomares Hernandez, Luis A. Morales Rosales, Jorge Estudillo Ramirez,
and Gustavo Rodriguez Gomez, Logical Mapping: An Intermedia Synchronization
Model for Multimedia Distributed Systems, Journal of Multimedia, Eds. Academy
Publisher, Vol. 3 No.5, 2008, ISSN: 1796-2048, pp. 33-41.
5. Tomoya Enokido, Sei-ichi Hatori, Takuya Tojo Makoto Takizawa. Group
Communication in Distributed Multimedia Objects. Proceeding of The Eighth
IEEE International Workshop on Object-Oriented Real-Time Dependable Systems
(WORDS 2003), 2003, pp. 258.
6. Yi Zhou, Tadeo Murata. Modeling and Analysis of Distributed Multimedia
Synchronization by Extended Fuzzy-Timing Petri Nets. Transactions of the
Society for Desing and Process Science, Volume 5, Number 4, December 2001,
pp. 23-37.
39
Thanks !
Questions?
INAOE 40
Resultados
Causalidad estricta
Relacin causal difusa y
consistencia causal difusa
Lnea del tiempo
Restricciones de una causalidad estricta vs.
relacin causal difusa y consistencia causal difusa
0
1
INAOE 41
Related work
Characteristics Coelho et. al Proposal
Predefined synchronization points Yes No
Decisions Centralized Distributed
Real time No Yes
Communications Producer-Consumer Group
Time Constraints No Yes
Quality of services No Yes
Clocks to order the events Physics Logics
Comparison of characteristics between the work done by
Coelho et al. in [3] and our proposed research
INAOE 42
Algoritmo
Historia Causal H(m) de a:
{d,f,z}
Part(w),
Part(x)
Part(y)
a
a
C
A
B
E
3.- Calculo de la Distancia Temporal: t(a)-t(a)
4.- Calculo de la Distancia Causal (a): {d,f,z}
F H
G
D
Part(y) orden de entrega causal:
Periodo 1
Periodo 2
Part(z)
f
z
d
g
2.- Obtencin de los parmetros RTT, Bw, jitter p/c evento de H(m)
1.- Calculo de los Periodos de cada elemento contenido en H(m)
5.- Calculo de RCD(a) y CCD a partir de H(a)={d,f,z}
6.- Calculo del estado actual
7.- Calculo de CR
8.- Paso de parmetros
al control difuso (CR y CCD)
9.-Salida del control difuso
Retrasar
Entregar
Descarte Selectivo
The GP variable is the weighting grade that determines the
degradation of a channel (participant) according to the best
channel of the system
( ) [ ]
i i i
n
i
PP jitter RTT BChannel + +
1
min
( )

,
_

+ +

BChannel
PP jitter RTT
GP
i i i
i

C
i =Part(A) audio
k =Part(C) audio
A
B
(m
k-2
)
Global Time 95
0
190 250
m
i
H(m
i
)={m
j
, m
k
}
l
m
1
m
k-2
m
k-1
m
k
m
j-1
m
j
(m
j-1
)
j =Part(B) video
INAOE 45
Tabla 2. Maximum error tolerable for multimedia synchronization
Media Mode, application Maximum Error
synchronization
(QoS, d
max
)
Video Animation Correlated t 120ms
Audio Lip synchronization t 80ms
Image Overlay t 240ms
No-overlay t 500ms
Text Overlay t 240ms
No-overlay t 500ms
Audio Animation Event correlation (e.g. dancing) t 80ms
Audio Tightly coupled(stereo) t 11ms
Loosely coupled (dialogue mode with
various participants)
t 120ms
Loosely coupled (e.g. background music) t 500ms
Image Tightly coupled (e.g. music with notes) t 5ms
Loosely coupled (e.g. slide show) t 500ms
Text Notes of text t 240ms
Pointer Audio related to the item t 700ms
INAOE 46
Algorithm
Historia Causal H(m) de a:
{d,f,z}
Part(w),
Part(x)
Part(y)
a
a
C
A
B
E
3.- Calculo de la Distancia Temporal: t(a)-
t(a)
4.- Calculo de la Distancia Causal (a):
{d,f,z}
F H
G
D
Part(y) orden de entrega causal:
Part(z)
f
z
d
g
2.- Obtencin de los parmetros RTT, Bw, jitter p/c evento de H(m)
1.- Calculo de los Periodos de cada elemento contenido
en H(m)
5.- Calculo de RCD(a) y CCD a partir de
H(a)={d,f,z}
6.- Calculo del estado actual
7.- Calculo de CR
8.- Paso de parmetros
al control difuso (CR y CCD)
9.-Salida del control difuso
Retrasar
Entregar
Descarte Selectivo

C
i =Part(A) audio
k =Part(C) audio
A
B
(m
k-2
)
Global Time 95
0
190 250
m
i
H(m
i
)={m
j
, m
k
}
l
m
1
m
k-2
m
k-1
m
k
m
j-1
m
j
(m
j-1
)
j =Part(B) video
Example of a synchronization
multimedia scenario

C
i =Part(A) audio
k =Part(C) animated
images
A
B
(m
k-2
)
Global Time 95
0
165 253
m
i
H(m
i
)={m
j
, m
k
}
l
m
1
m
k-2
m
k-1
m
k
m
j-1
m
j
(m
j-1
)
j =Part(B) video
INAOE 49
Results(Cont.)
Fuzzy causal relation
Temporal cause-effect
degree
Cause-effect
Logical cause-effect degree
Cause-effect
1
1
INAOE 50
Related work
Characteristics Coelho et. al Proposal
Predefined synchronization points Yes No
Decisions Centralized Distributed
Real time No Yes
Communications Producer-
Consumer
Group
Time Constraints No Yes
Quality of services No Yes
Clocks to order the events Physic Logic
Comparison of characteristics between the work done by
Coelho et al. in [3] and our proposed research
INAOE 51
Results(Cont.)
The FCR applied to the intermedia synchronization
The membership functions, R
N
and R
D

can be calculated by a triangular function
defined as:
0 If x < f
If f x < h

1 If x h
f h
f x

A(x)
f h
1
O
INAOE 52
Results(Cont.)
The FCR applied to the intermedia synchronization
Temporal cause-effect degree
Cause-effect
d
max
Logical cause-effect degree
Cause-effect
1
1
Causal distance
d
max
is assigned according to the maximum error allowed
Causal distance is established to four events
INAOE 53
Results (Cont.)
Input value for the fuzzy causal relation

Temporal distance
(m)= time_arrival(m) time_arrival(m) p H(m) m = last_message
p
Part(X)
Part(Y)
m m
Local physical clock of Part(Y)
(m)
INAOE 54
Results(Cont.)
FCC applied to the intermedia synchronization
RTT is the Round trip time of a
message
Jitter is the fluctuation of end to
end of a message with the next
message inside the same stream
PP is the number of lost
messages in a synchronization
period
n is the number of channels
(participants) which are causally
related to the event a
( ) [ ]
i i i
n
i
PP jitter RTT BChannel + +
1
min
( )

,
_

+ +

BChannel
PP jitter RTT
GP
i i i
i

GP is the weighting grade that determines the degradation

of a channel (participant) according to the best channel of
the system
INAOE 55
Results (Cont.)
Input value for the fuzzy control system
NC is calculated by the following formula:
Where
B is the bandwidth available in the network.
RTT is the round trip time of a message a across the network.
PE is the number of expected messages inside the synchronization period.
PR is the number of received messages inside the synchronization period.
S
GP
is the sum of the weighted grades of the channels.
n is number of channels(participants) which are causally related with the message
a, known as the causal history of a.
( )
GP
n
i
i
i
i i
i
i
S
GP
PE
PR PE
RTT
B
a NC

1
]
1

,
_

1
*
1
) (
INAOE 56
Results (Cont.)
Fuzzy control system

The set of rules used by the fuzzy control to adjust

the delivery time of the message a has the form:
If NC(a) A
i
y FCC
p
(a) B
i
then C
i

Where:
NC(a), FCC
p
(a) and are the enter parameters of the fuzzy
control system
A
i
= {good, regular, bad} are the linguistic labels for NC(a)
B
i
= {good, regular, bad} are the linguistic labels for the FCC
p
(a)
C
i
= {big, medium, short} are the linguistic labels for the output
INAOE 57
Results (Cont.)
Fuzzy control system
The set of rules that the fuzzy control system considers is the
following
1. If NC(a)= bad and FCC
p
(a)= good, then = medium (delivery without discard)
2. If NC(a)= bad and FCC
p
(a) = regular, then = short (delivery with discard)
3. If NC(a)= bad and FCC
p
(a) = bad, then = short (delivery with discard)
4. If NC(a)= regular and FCC
p
(a) = good, then = medium (delivery without discard)
5. If NC(a)= regular and FCC
p
(a) = regular, then = medium (delivery without discard)
6. If NC(a)= regular and FCC
p
(a) = bad, then = short (delivery with discard)
7. If NC(a)= good and FCC
p
(a) = good, then = big (delivery without discard)
8. If NC(a)= good and FCC
p
(a) = regular, then = big (delivery without discard)
9. If NC(a)= good and FCC
p
(a) = bad, then = medium (delivery without discard)
The Mamdani model has been selected as the inference
mechanism

The centroide function has been selected for the defuzzyfication of

the output of the fuzzy control system