IJRET: International Journal of Research in Engineering and Technology

eISSN: 2319-1163 | pISSN: 2321-7308

A NEW NON-SYMMETRIC INFORMATION DIVERGENCE OF

CSISZAR'S CLASS, PROPERTIES AND ITS BOUNDS
K.C.Jain1, Praphull Chhabra2
1

Professor, Department of Mathematics, Malaviya National Institute of Technology, Jaipur (Rajasthan), India
Ph.d Scholar, Department of Mathematics, Malaviya National Institute of Technology, Jaipur (Rajasthan), India

Abstract
Non-parametric measures give the amount of information supplied by the data for discriminating in favor of a probability
distribution P against another Q , or for measuring the distance or affinity between P and Q .
There are several generalized functional divergences, such as: Csiszar divergence, Renyi- like divergence, Bregman divergence,
Burbea- Rao divergence etc. all. In this paper, a non-parametric non symmetric measure of divergence which belongs to the
family of Csiszrs f-divergence is proposed. Its properties are studied and get the bounds in terms of some well known divergence
measures.

Mathematics Subject Classification: 94A17, 26D15

Keywords: Csiszar's f divergence, convex and normalized function, non-symmetric divergence measure, information
inequalities, bounds of divergence measure.
----------------------------------------------------------------------***------------------------------------------------------------------1. INTRODUCTION
n

Let n P p1 , p2 , p3 ..., pn : pi 0, pi 1 , n 2
i 1

be the set of all complete finite discrete probability

distributions. If we take pi 0 for some i 1, 2, 3,..., n ,

0
then we have to suppose that 0 f 0 0 f 0 .
0

pi qi

i 1

qi

P, Q

h P, Q

Ra P, Q

known divergences can be obtained from this generalized

measure by suitably defining the convex function f. Some of
those are as follows:

(1.5)
n

B P, Q pi qi

Bhattacharya

i 1

and

pi and qi are probability mass functions. Many

Hellinger

pi a
, a 1 = Renyis a
qi a 1

order entropy [6]

function

(1.4)

i 1

(1.1)

P p1 , p2 , p3 ..., pn , Q q1 , q2 , q3 ..., qn n,

where

pi qi

Where f : 0, R (set of real no.) is real, continuous

Square

(1.3)

convex

Chi-

discrimination [3]

and

divergence measure [5]

i 1

Csiszar [2], given the generalized f- divergence measure,

which is given by:

p
C f P, Q qi f i
i 1
qi

divergence measure [1]

Relative information of type s [9]

(1.6)

n
1

s P, Q s s 1 pi s qi1s 1 , s 0,1 and s R

i 1

(1.7)

n
p
K P, Q pi log i = Kullback- Leibler
i 1
qi

divergence measure [4]

(1.2)

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

665

IJRET: International Journal of Research in Engineering and Technology

Particularly

eISSN: 2319-1163 | pISSN: 2321-7308

700

lim s P, Q K P, Q , lim s P, Q K Q, P
s 1

600

s 0

500

(1.8)

400

Where K P, Q is given by (1.2).

300

p qi
G P, Q i
2
i 1
n

pi qi

log 2 p

Relative AG Divergence [7]

200

100

(1.9)
2

Similarly, we get many others divergences as well by

defining suitable convex function

2.

NEW

INFORMATION

DIVERGENCE

MEASURE

10

Fig1: Graph of convex function f t

Now, put (2.1) in (1.1), we get the following new
divergence:

In this section, we shall obtain a new divergence measure

corresponding to new convex function, and will study the
properties.

C f P, Q V

pi qi

i 1

pi qi

P, Q

(2.3)

The following theorem is well known in literature [2].

Theorem 1: If the function f is convex and normalized, i.e.,

Properties of divergence defined by (2.3), are as follows:

In view of theorem 1, we can say that

V * P, Q 0 and convex in the pair of probability

f 1 0 , then C f P, Q and its ad joint C f Q, P are

both non-negative and convex in the pair of probability

distribution

P, Q n n .

t 1
t

minimum value).

, t 0,

Since

Figure 2: Comparison of divergence

measures

f t

t 1
t

3t 1 ,

f t

2 t 1
t

6
2

3t

2t 1

(2.2)

Since

f t 0 t 0, f t is a convex

function.

Since

Since

f 1 0 f t is a normalized function.
f t 0 at

0,1 and

5
4

Properties of function defined by (2.1), are as follows:

V * P, Q V * Q, P V * P, Q is non-

symmetric divergence measure w.r.t. P & Q .

(2.1)

And
3

P, Q n n .

V * P, Q 0 if P Q or pi qi (Attains its

Let f: (0, ) R, be a mapping, defined as:

f t

distribution

f t 0 at

1, f t is monotonically decreasing in
0,1 and monotonically increasing in 1, , and
f 1 0 .

V*(P,Q)

3
2
1
0
0

0.5

Relative
Arithmetic
Geometric div.
KullbackLeibler div.

a
Figure 2 shows the behavior of V * P, Q , Relative
Arithmetic-Geometric divergence G P, Q and KullbackLeibler divergence K P, Q . We have considered

pi (a,1 a) and qi (1 a, a) where a (0,1) . It

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

666

IJRET: International Journal of Research in Engineering and Technology

is clear from figure 2 that the new measure V

P, Q has a

eISSN: 2319-1163 | pISSN: 2321-7308

4.1 Proposition 4.1(at s=2)

P, Q and V * P, Q be defined as in (1.3) and

steeper slope then G P, Q and K P, Q .

Let

3. CSISZARS FUNCTIONAL DIVERGENCE

(2.3) respectively. Then, we have

i.
If 0 1 , then

AND INEQUALITIES

2
12 2

1 2

0 V P, Q max. 3 3 2 1 , 3 3 2 1 2 P, Q

*
*
0 V P, Q V P, Q
(4.1)
*

The following theorem is well known in literature [8].

Theorem 2: Let

f : I R R (I is an open interval) be

a mapping which is normalized, i.e., f 1 0 and suppose

that
I.

2
12

2
max. 3 3 2 1 , 3 3 2 2 1 2 P, Q

f
is
twice
differentiable
on
(4.2)
, , 0 1 with .
II.
There exist real constants m, M such that
ii.
If 1 , then
m M and m t 2s f " t M t , and s R
2
1

*
and
0 V P, Q
3 2 2 1 2 P, Q (4.3)
3
p
If P, Q n with 0 i i 1, 2,3..., n
2
1 3 2 2 1 2 P, Q
qi
0 V* P, Q V * P, Q

, then
3
m s P, Q C f P, Q M s P, Q

(4.4)
(3.1)
Proof:
Firstly, put s=2 in (1.7) and (3.4) respectively, we get

And

m s P, Q s P, Q C P, Q C f P, Q M s P, Q s P, Q
(3.2)

1 n p2
1 n p2
1 n p q 1
s P, Q i 1 i 2 pi qi i i 2 P, Q
2 i1 qi
2 i1 qi
2 i1 qi
2
2

(4.5)
Where
n
p
P2
C P, Q C f , P C f P, Q pi qi f i
i 1
Q
qi

s P, Q pi qi
i 1

n
n
pi
p2
p2
i pi i 1
qi i 1 qi
i 1 qi

(3.3)
s 1

n
p
P
1
s P, Q Cs , P Cs P, Q s 1 pi qi i , s 1
i 1
Q
qi
2

(3.4)
And C f P, Q , s P, Q are given by (1.1) and (1.7)
respectively

4.

BOUNDS

i 1

n
p q
pi 2
2 pi qi i i 2 P, Q
qi
qi
i 1
2

(4.6)
And by putting f t in (3.3), we get
n
P2
p q
V P, Q V , P V f* P, Q i i 2 3 pi qi
i 1 pi qi
Q
4

OF

NEW

INFORMATION

*
f

(4.7)

DIVERGENCE MEASURE
In this section, we derive bounds for V

P, Q in terms of

the well known divergences in the following propositions at

s 2,1,1/ 2, 0 and 1, by using the theorem 2.

Let

g t f t

putting s=2 in t

2 s

2 t 1

f " t )

t3

3t

2t 1

(After

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

667

IJRET: International Journal of Research in Engineering and Technology

6 t 4 1

g t

Then

t4

eISSN: 2319-1163 | pISSN: 2321-7308

2 1
0 V P, Q V P, Q
3 2 2 1 K Q, P

24
, g t 5
t

(4.14)

If g t 0 t 1 0 t 1, 1
4

It is clear that g (t) is monotonic decreasing on (0, 1) and

monotonic increasing on [1, ).
Also

(t)

has

minimum

g 1 24 0 so

value

at

t=1,

m inf g t g 1 0

since

(4.8)

t 0,

Proof:
Firstly, put s=1 in (1.7) and (3.4) respectively, we get
n
p
lim s P, Q pi log i K P, Q
s 1
i 1
qi

(4.15)

n
p
q
lims P, Q pi log i qi log i K P, Q K Q, P
s 1
i 1
qi
pi

(4.16)
Now, we have two cases:
i.
If 0 1 , then

M sup g t max g , g

putting s=1 in t

t ,

2
2 12

2 1
2
2
max
3

1
,
3

If

1 , then

M sup g t

2 1

f " t )

t2

3t

2t 1

(After

12 t 1
4 t 1 3 2

3
t

1
,
g
t

t3
t4
4

If
2

t[1, )

2 s

2 t 1

Then

g t

(4.9)
ii.

g t t f t

Let

2 1

(4.10)

g t 0 t 1 3t 3 t 2 t 1 0 t 1, 0.63

The results (4.1), (4.2), (4.3) and (4.4) are obtained by using
(2.3), (4.5), (4.6), (4.7), (4.8), (4.9) and (4.10) in 3.1 and 3.2.

It is clear that g (t) is monotonic decreasing on (0, 1) and

monotonic increasing on [1, ).

4.2 Proposition 4.2(at s=1)

Also

Let K P, Q and V * P, Q be defined as in (1.2) and

(t)

has

minimum

g 1 24 0 so

value

m inf g t g 1 0

(2.3) respectively. Then, we have

at

t=1,

since

(4.17)

t 0,

0 1 , then
2
2
we have two
cases:

2 1 Now,
2 1

*
2
2
0 V P, Q max.
3

1
,
3

1
K
,Q
0

i.
If

1 ,Pthen
2
2

M sup g t max g , g

i.

If

(4.11)

0 V P, Q V
*

t ,

P, Q

2
2 12 2

2 1
2
max.
3

1
,
3

K Q, P
2
2

(4.12)

1 , then
2
2 1
*
0 V P, Q
3 2 2 1 K P, Q

ii.

2
2 12 2

2 1
max
3 2 1 ,
3 2 2 1

2
2

(4.18)
ii.

If

1 , then

If

(4.13)

M sup g t
t [1, )

2 1

2 1

(4.19)

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

668

IJRET: International Journal of Research in Engineering and Technology

The results (4.11), (4.12), (4.13) and (4.14) are obtained by

using (2.3), (4.7), (4.15), (4.16), (4.17), (4.18), and (4.19) in
3.1 and 3.2.

Let h P, Q ,R a P, Q ,B P, Q and V * P, Q be
defined as in (1.4), (1.5), (1.6) and (2.3) respectively. Then,
we have
i.
If 0 1 , then

8 1

8 1
0 V * P, Q max 3/2 3 2 2 1 , 3/2 3 2 2 1 h P, Q

(4.20)

g t t f t

Let

Then g t

(4.21)

3/2

2 1 h P, Q

2t 1

t 1 , g t

3
2t

7
2

15t

(After

4t 3 5

It is clear that g (t) is monotonic decreasing on (0, 1) and

monotonic increasing on [1, ).
g

(t)

has

minimum

value

at

m inf g t g 1 0

t=1,

since

(4.26)

t 0,

Now, we have two cases:

i.
If 0 1 , then

2
2

2 1
2 1
2
2
max
3 2 1 , 3 3 2 1
3
2

(4.27)

8 1
1

3 2 2 1 R3/2 Q, P B P, Q h P, Q

3/2

If

ii.

Proof:
Firstly, put s=1/2 in (1.7) and (3.4) respectively, we get
n

i 1

i 1

s P, Q 41 pi qi 2 2 2 pi qi
n

i 1

i 1

2 pi qi 2 pi qi 4

pi qi
2

1 , then

M sup g t
t [1, )

(4.23)

2 1

3
2

2 1

(4.28)

The results (4.20), (4.21), (4.22) and (4.23) are obtained by

using (2.3), (4.7), (4.24), (4.25), (4.26), (4.27), and (4.28) in
3.1 and 3.2.

4.4 Proposition 4.4(at s=0)

4h P , Q

(4.24)

i 1

If g t 0 t 1 5t t t 1 0 t 1, 0.53

0 V* P, Q V * P, Q

s P, Q 2 qi pi

3t

t ,

(4.22)

5t t

3
2

M sup g t max g , g

1 , then
8 1

5
2

2 t 1

f " t )

g 1 24 0 so

2
8 12 2
1
8 1 2

max 3/2 3 2 1 , 3/2 3 2 1 R3/2 Q, P B P, Q h P, Q

0 V * P, Q

2 s

3 t 1
t

Also

0 V* P, Q V * P, Q

If

3
2

putting s=1/2 in t

4.3 Proposition 4.3(at s=1/2)

ii.

eISSN: 2319-1163 | pISSN: 2321-7308

n
q 3/2

qi
2 i 1/2 pi qi 2 R3/2 Q, P B P, Q
pi i1 pi

Let K P, Q ,

P, Q and V * P, Q be

defined as

in (1.2), (1.3) and (2.3) respectively. Then, we have

i.

If

0 1 , then

2
2 12 2

2 1 2
0 V P, Q max
3 2 1 ,
3 2 1 K Q, P

(4.25)

(4.29)

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

669

IJRET: International Journal of Research in Engineering and Technology

0 V* P, Q V * P, Q
2
2 12 2

2 1 2
max
3 2 1 ,
3 2 1 2 Q, P K Q, P

(4.30)

1 , then
2
2 1
*
0 V P, Q
3 2 2 1 K Q, P

ii.

If

(4.31)

0 V* P, Q V * P, Q

2 1
3 2 2 1 2 Q, P K Q, P

eISSN: 2319-1163 | pISSN: 2321-7308

Now, we have two cases:

i.
If 0 1 , then

M sup g t max g , g
t ,

2
2 12

2 1
2
max
3 2 1 ,
3 2 2 1

(4.36)
ii.

If

1 , then

M sup g t

2 1

t [1, )

2 1

(4.37)
The results (4.29), (4.30), (4.31) and (4.32) are obtained by
using (2.3), (4.7), (4.33), (4.34), (4.35), (4.36), and (4.37) in
3.1 and 3.2.

(4.32)
Proof:
Firstly, put s=0 in (1.7) and (3.4) respectively, we get

4.5 Proposition 4.5(at s =-1)

q
lim s P, Q qi log i
s 0
i 1
pi
n

K Q, P

Let

2 P, Q ,R a P, Q and V * P, Q be

defined as

in (1.3), (1.5) and (2.3) respectively. Then, we have

(4.33)
i.
n
n
n
n
p q
q2
q2
q2
s P, Q i qi i 1 i 2qi pi i i 2 Q, P
pi
i 1 pi
i 1 pi
i 1 pi
i 1

If

0 1 , then

0 V * P, Q max 1 3 2 2 1 , 1 3 2 2 1 2 Q, P
2

(4.34)
Let g t t f t
2

s=0 in t

2 s

Then

g t

f " t )

2 t 1

3t

2t 1 (After putting

0 V* P, Q V * P, Q

(4.38)

max 1 3 2 2 1 , 1 3 2 2 1 R2 Q, P R3 Q, P 2 Q, P

2 t 1 3 2
1
9t t t 1 , g t 3 36t 4 16t 3 4 ii.

2
t
t

(4.39)
If

1 , then

0 V * P, Q 1 3 2 2 1 2 Q, P
2

If

g t 0 t 1 9t 3 t 2 t 1 0 t 1, 0.43

It is clear that g (t) is monotonic decreasing on (0, 1) and

monotonic increasing on [1, ).

(4.40)

0 V* P, Q V * P, Q 1 3 2 2 1R3 Q, P R2 Q, P 2 Q, P
2

(4.41)
Also

(t)

has

g 1 24 0 so

minimum

m inf g t g 1 0
t 0,

value

at

t=1,

since
Proof:
Firstly, put s=-1 in (1.7) and (3.4) respectively, we get

(4.35)

1 n q2
1 n q2
1 n p q 1
s P, Q i 1 i 2qi pi i i 2 Q, P
2 i 1 pi
2 i 1 pi
2 i 1 pi
2
2

(4.42)

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

670

IJRET: International Journal of Research in Engineering and Technology

s P, Q

qi 2 1 n qi3 qi 2 1
1 n
q

i i p 2 2 i1 p 2 p 2 R3 Q, P R2 Q, P
2 i1
i
i
i
(4.43)

Let g t t 3 f t 2 t 1
s=-1 in t

3t

2t 1 (After putting

f " t )

2 s

Then g t 24 t

t 1 , g t 72 t 2 48t

If g t 0 t 0, 1
It is clear that g (t) is monotonic decreasing on (0, 1) and
monotonic increasing on [1, ).
Also

(t)

has

g 1 24 0 so

minimum

value

at

m inf g t g 1 0

t=1,

eISSN: 2319-1163 | pISSN: 2321-7308

[5]. Pearson K., On the Criterion that a given system of

deviations from the probable in the case of correlated system
of variables is such that it can be reasonable supposed to
have arisen from random sampling, Phil. Mag., 50(1900),
157-172.
[6]. Renyi A., On measures of entropy and information,
Proc. 4th Berkeley Symposium on Math. Statist. and Prob.,
1(1961), 547-561.
[7]. Taneja I.J., New developments in generalized
information measures, Chapter in: Advances in Imaging and
Electron Physics, Ed. P.W. Hawkes, 91(1995), 37-135.
[8]. Taneja I. J. and Kumar P., Generalized non-symmetric
divergence measures and inequalities (2000), The Natural
Science and Engineering Research Councils Discovery
grant to Pranesh Kumar.
[9]. Taneja I.J. and Kumar P., "Relative Information of types, Csiszar's f-divergence and information inequalities",
Information Sciences, (2003).

since

(4.44)

t 0,

Now, we have two cases:

i.

If

0 1 , then

M sup g t max g , g
t ,

max 2 1 3 2 2 1 , 2 1 3 2 2 1
2

(4.45)
ii.

If

1 , then

M sup g t 2 1 3 2 2 1
2

t[1, )

(4.46)
The results (4.38), (4.39), (4.40) and (4.41) are obtained by
using (2.3), (4.7), (4.42), (4.43), (4.44), (4.45), and (4.46) in
3.1 and 3.2.

REFERENCES
[1]. Bhattacharyya A., On some analogues to the amount of
information and their uses in statistical estimation,
Sankhya, 8, 1-14.
[2]. Csiszar I., Information type measures of differences of
probability distribution and indirect observations, Studia
Math. Hungarica, 2(1967), 299-318.
[3]. Hellinger E., Neue begrundung der theorie der
quadratischen
formen
von
unendlichen
vielen
veranderlichen, J. Rein.Aug. Math., 136(1909), 210-271.
[4]. Kullback S. and Leibler R.A., On Information and
Sufficiency, Ann. Math. Statist., 22(1951), 79-86.

_______________________________________________________________________________________
Volume: 03 Issue: 05 | May-2014, Available @ http://www.ijret.org

671