VIDEO RETRIEVAL PROJECT REPORT
Sohaib Abdul Rehman, Usama Mehmood and Salik Mahmood
1. INTRODUCTION
The efficient and accurate retrieval of videos
similar to the query made by the user is much difficult than
the image retrieval. On YouTube alone more than 48 hours
of videos are posted every minute [1]. This fast expanding
of the video databases requires fast and efficient methods
for the retrieval of videos. The aim of this project is to
retrieve the videos which match the best to the query made
by the user. The query can be in the form of a video or a
text.
An important point to note about the data set is
that, the data set provided on LMS for Merlin and Sons of
Anarchy seasons was not good to work for semantic
concepts. It firstly consisted of very long videos and
secondly not diverse concepts were present in the video, so
very vague results were obtained.
To overcome this problem, we took a set of videos
which consisted of diverse concepts. We ran the semantic
concept on this data base. So we could not compare it with
part 1&2 of the project, as it consisted of videos from
Merlin and Sons of Anarchy.
In the first part of the project, we looked at the
content based video retrieval (CBVR). This concept is an
extension to the content based image retrieval and is
discussed in section 2.A, of the report. The second part of
the project was to retrieve the videos from a data set using
the method of tiny videos [2]. This method involves the
conversion of frames of a video to 32×32 feature vectors
and also incorporates the temporal dependence of the video
(which CBVR does not). The method is discussed in detail
in section 2.B of the report. Finally we looked at the video
retrieval using semantic concepts present in the video. We
also formulated a method for automatic annotation of the
videos. This part of the project is described in section 2C of
the report.
We have also discussed merits and demerits of all
the methods in section 2. The results from three parts of the
project are discussed in section 3 of the report.
2. METHADOLOGY
In this part of the project we explain our approach for
CBVR, tiny videos and semantic concepts based video
retrieval.
A. Content Based Video Retrieval
As discussed earlier CBVR is an extension to CBIR. A
video can be sampled to obtain images (frames), so it makes
sense to make use of image retrieval methods for video
retrieval.
For CBVR we extracted the key frames from the video
by uniform sampling of the video (we sampled at 1/15
frames per second). We then divided each frame into 9
blocks. Three histograms (for red green and blue colors)
were computed for each block and were concatenated to get
a feature vector of a block of frame. These feature vectors
were concatenated to get a feature vector for the entire
frame. The feature vectors of frames were then concatenated
to get feature vector of a video.
To save space and to make the method efficient, we
quantized the histograms. In section 3.A of the report we
look at the effect of quantization on video retrieval. Another
technique to make the process more efficient could be to use
global histogram for the features (instead of dividing it into
blocks). Although it does not include the spatial dependence
of the frames, but is time efficient. Results for these global
histograms are also discussed in section 3A.
There are certain limitations of this technique which is
due to the fact that CBVR does not include the temporal
dependency of the frames. As we were using uniform
sampling so the videos which differ by a delay of 1 second
are not rendered similar. Moreover, as there is no
dependency on the concept present in the video, so this
method can declare two totally different (on the basis of
concept) videos to be similar.
B. Tiny Videos based Video Retrieval
We used uniform sampling in this method for video
retrieval. In the case of uniform sampling we took frames
after every 15 second interval from the video.
B.1 Data Base:
After extracting the key frames we made the data
base of the feature matrix of a video. The feature matrix of a
video was computed by computing the feature vector for all
the key frames and stacking them into rows.
The feature vector was computed by down sampling the key
frame to 32×32 size. Then all the three channels were
concatenated to obtain a vector. The vector was normalized
and made zero mean to obtain the feature vector. Making
the video zero mean was necessary to cancel the effect of
brightness difference between two otherwise similar images.

B.2 Similar Video Finding:
For this part of the project we assume two videos to
be similar if they have at least one similar frame. The
distance between videos ‘a’ and ‘b’ were computed as
follows:
Where D2ssd(Ia,Ib) was defined as:
Where Ia ans Ib are the feature vectors of key frame ‘a’
and key frame ‘b’ respectively. Here Ia belongs to video ‘a’
and Ib belongs to video ‘b’.
We define correlation between two videos as:
The next step was to select frames from all shots of a
video and divide them into groups of 16×16 pixels, and run
the DCT on the group. This gave us 256 coefficients for
each group; we selected the first 100 (40% coefficients
used) coefficients for each group. We selected 20 frames
from each shot.
We do not lose a lot of information by discarding the
DCT coefficients as more energy is present in the low DCT
coefficients. Figure 3 show the result of image
reconstruction from 67% coefficients, 50% coefficients and
25% coefficients. The image is seen to be reconstructed
without much loss of information (although some blurring in
25% case because only low very low DCT coefficients are
used).

We consider two videos to be similar if their

correlation is above a certain threshold. The effect of
thresholds is discussed in section 3.C.
Contrary to CBVR, this method takes into account the
temporal nature of the video. The limitations of this method
are due to the fact that there is no dependency on the
concepts present in the video, two videos with totally
different concepts can be considered to be similar by this
method. Moreover, as uniform sampling was done, so Figure 1: A Shot
videos with a delayed frames will not be rendered similar.
Also the method is dependent on the length of the video.

C. Semantic Concepts
This method was based on video retrieval on the basis
of semantic concepts present in the video. This method of
video retrieval consists of the following steps.

C.1 Manual Annotation

As the dataset provided on LMS was not annotated, so
the first task was to manually annotate the videos present in
the data set. The annotation was done by first detecting the
shots in the video and then by labeling the shots with the
concept present in the videos. This was done for all the shots
in a video. Figure 2: A Shot

C.2 Shot Detection

The shots in the video were detected on the basis of
intensity of motion. In this method the difference between
two consecutive frames was computed and the first 25
frames for which this difference was maximum were
selected. This gave us the boundries for 25 shots in the
video. Figure 1&2 shows uniformly sampled 9 frames. As
seen from the figure, the frames of a shot belong to a similar
concept, so it makes sense to give all the frames of the shots
the same label.

C.3 DCT Co-efficient:


Figure 3: DCT Results
C.4 Clustering:
After obtaining the DCT coefficients (feature vectors)
for all the selected frames of the shot, we separated the
feature vectors belonging to particular concepts. For
example, for the concept ‘mountain’, we searched all the
videos for the shots labeled ‘mountain’. We obtain the
feature vectors from all the shots having the concept and ran
the ‘kmeans clustering’, algorithm on the feaure vectors.
This clustering was done in 100 dimensional space in which
all feature vectors were points. We divided each concept
into three clusters. The three centroids were saved for each
concept.
C.5 Automatic Annotation
Next task was to automatically annotate the videos, for
this a shot to be annotated was first converted to feature
vectors (DCT coefficients for all the frames (or selected
frames)). Then the clustering was done on the resultant
feature vectors, as described in the previous section. The
centroids obtained were matched to the centroids of all the
concepts (previous section), and the shot was labeled with
the concept for which the score was minimum. We used the
sum of absolute difference as the distance measure. Below
are the results for automatic annotation:
Video Name: Formula 1 2014 Australian Grand Prix Official Race
Edit [1080p] - Video Dailymotion.avi
Given Labels for shot 2: people
Given Labels for shot 3: race, road
Given Labels for shot 4: race, people
Computed Labels for shot 2: road, people, building, race, person
Computed Labels for shot 3:road, race, people, building, person
Computed Labels for shot 4:building, person, people, race, road
The 9 frames from the shot 4 of the video are shown in
figure 4. The frames from the shot show that the annotation
is not bad, as the concept of building was missed in labeling,
but automatic annotation did not miss it.
The advantage of this method is that, in it we used
intensity of motion to evaluate the boundaries, so videos
with delayed frames will be considered similar, which was
not the case in the first 2 parts of the project. Moreover, as
we have included the concepts present in the video, so only
those videos will be declared similar which have similar
concept.
Although this method has advantages, but we need a lot
of data for the training of the algorithm, but at the same time
this large data can cause memory problems when fitting a
distribution (a reason why we used clustering instead of
GMM).
Figure 4
3. RESULTS
In this part of the report we discuss results from the three
approaches of video retrieval.
A. Content Based Video Retrieval
The results are based on first seasons of TV shows
Merlin and Sons of Anarchy (which were used to make the
data set).
A.1 Quantization Levels
In table 1, results are shown for different quantization
levels. As discussed earlier, we were only using the
histograms to compute the feature vectors, hence these
vectors do not have any information regarding the concept
of video, objects in video, etc. Hence, feature vectors of two
entirely different TV shows (on the basis of concepts and
objects in videos) can give high similarity as shown in table.
The table was computed by giving ‘Merlin S01E11’ as a
query. The similarity measure used was absolute difference.
If we assume 256 quantization level retrieval to be the
perfect case, we can see that as the quantization levels
increase the error (w.r.t. to 256 quantization levels) in the
retrieval of videos also increase. For example we can see
from the table that for 128 quantization levels although the
top 10 retrieved videos remain the same but there ranks
change i.e. positions 4&5 are switched, positions 6 & 7 are
also no longer the same and so on. But if we are only
concerned with top 10 matches (irrespective of rank), this
can work.
 For quantization levels of 64 and 8, new videos are
introduced in the top 10, which shows more error. Here
again the top 3 videos are the same (not considering rank).
So depending on the application we can use any of the
quantization levels.
A.2 Global Features
Table 2 gives a comparison for global and local feature
vectors. The table was computed for 256 quantization levels
and using absolute difference as similarity measure.
As can be seen from table 2, in the top 10 retrieved
videos, only 2 are different, shown with red color, although
rank can differ. But global features with same quantization
levels require much less space and are time efficient. Hence
depending upon the application we can used global or local
features.
matches with certain correlation, it is considered to be a
similar video, but for CBVR every sampled frame should
match considerably with every frame of the reference video
for a considerable match.
References
[1] Yang Cai, Linjun Yang, ‘Large Scale Near Duplicate
web Video Retrieval: Challenges and Approaches’
[2] Alexandre Karpenko, Parham Aarabi, ‘Tiny Videos: A
Large Data Set for Nonparametric Video Retrieval and
Frame Classification’

B. Tiny Videos

The results are based on first seasons of TV shows

Merlin and Sons of Anarchy (which were used to make the
data set).
We first look at the effect of threshold of correlation on
the results. The input video was a small segment of the
‘MerlinS01E01’. Table 3 has 10 top similar videos for
different correlation values. We conclude that the results
improve as the correlation value increase.

B.1 Effect of Size of Video

As discussed earlier, this method is dependent on the size
of the video, so if we give the entire ‘MerlinS01E01’ we get
different results, shown in table 4. Here again the results
improve as the value of correlation increase.

C. Semantic Concepts
To retrieve the video, user searches for a word and the
videos in which the shot labeled with the entered word are
shown on screen. The videos are shown in the order such
that the video with maximum number of shots labeled with
the concepts is shown in the top and so on.
The results for searching different key words are shown
in table 5.

D. Comparison Between CBVR and Tiny Videos

Here we give a comparison of the results from CBVR
and tiny videos methods. We give MerlinS01E01 as input
query. The results from tiny videos based method is shown
in table 4 for different correlation values. For CBVR, we
used 256 quantization levels and local histograms, results
are in table 6.
As seen from table there is a difference in the results of 2
methods. Tiny videos method give MerlinS01E01 and
MerlinS01E02, for correlation of 0.9, but MerlinS01E02 is
not among the top 5 similar matches of CBVR. This can
happen because both techniques are based on totally
different concepts, in tiny videos even if a single shot
Sr. No 256 Quant Levels 128 Quant Levels 64 Quant Levels 8 Quant Levels
1 'Merlin S01E11’ ‘Merlin S01E11’ ‘Merlin S01E11’ ‘Merlin S01E11’
2 'Merlin S01E05’ 'Merlin S01E05’ 'Merlin S01E05’ 'Sons.of.Anarchy.S01E10’
3 'Merlin S01E01’ 'Merlin S01E01’ 'Sons.of.Anarchy.S01E10’ 'Merlin S01E05’
4 ‘Sons.of.Anarchy.S01E03’ ‘Sons.of.Anarchy.S01E10’ 'Merlin S01E01’ 'Sons.of.Anarchy.S01E06’
5 ‘Sons.of.Anarchy.S01E10’ ‘Sons.of.Anarchy.S01E03’ 'Merlin S01E13’ ‘Sons.of.Anarchy.S01E12’
6 'Sons.of.Anarchy.S01E07’ 'Merlin S01E13’ ‘Sons.of.Anarchy.S01E03’ 'Merlin S01E13’
7 'Merlin S01E10’ 'Sons.of.Anarchy.S01E07’ 'Merlin S01E04’ 'Sons.of.Anarchy.S01E02’
8 'Sons.of.Anarchy.S01E08’ 'Sons.of.Anarchy.S01E05’ ‘Sons.of.Anarchy.S01E05’ 'Sons.of.Anarchy.S01E01’
9 'Sons.of.Anarchy.S01E05’ 'Merlin S01E10’ 'Sons.of.Anarchy.S01E12’ 'Merlin S01E04’
10 ‘Merlin S01E13’ 'Sons.of.Anarchy.S01E08’ 'Merlin S01E10’ 'Sons.of.Anarchy.S01E11’
Table 1: CBVR result for MerlinS01E11 as Query, Local Histogram

Sr. Local Feature Vector
No
1 'Merlin S01E11’
2 'Merlin S01E05’
3 'Merlin S01E01’
4 ‘Sons.of.Anarchy.S01E03’
5 ‘Sons.of.Anarchy.S01E10’
6 'Sons.of.Anarchy.S01E07’
7 'Merlin S01E10’
8 'Sons.of.Anarchy.S01E08’
9 'Sons.of.Anarchy.S01E05’
Global Feature Vectors No Corr = 0.5 Corr = 0.6 Corr = 0.7 –
0.999
1 'Merlin S01E01’ ‘Merlin S01E01’ ‘Merlin S01E01’
‘Merlin S01E11’ 2 'Merlin S01E02’ 'Merlin S01E03’
'Sons.of.Anarchy.S01E10’
3 'Merlin S01E03’ 'Merlin S01E06’
'Sons.of.Anarchy.S01E09’
'Sons.of.Anarchy.S01E05’ 4 'Merlin S01E04’ 'Merlin S01E07’
'Merlin S01E05’ 5 'Merlin S01E05’ 'Merlin S01E11’
‘Sons.of.Anarchy.S01E07’ 'Merlin S01E06’ 'Sons.of.Anarchy.S01E01’
6
'Sons.of.Anarchy.S01E08’
7 'Merlin S01E07’ 'Sons.of.Anarchy.S01E05’
‘Sons.of.Anarchy.S01E03’ 8 'Merlin S01E08’ 'Sons.of.Anarchy.S01E09’
'Sons.of.Anarchy.S01E06’ 'Merlin S01E09’ 'Sons.of.Anarchy.S01E11’
9

10 ‘Merlin S01E13’ ‘Merlin S01E01’ 10 ‘Merlin S01E10’ 'Sons.of.Anarchy.S01E12’

Table 2: CBVR result for MerlinS01E11 as Query, Global Histogram Table 3: Tiny Videos results, short segment ofMerlinS01E01 as query

No Corr = 0.5 Corr = 0.9 No. KeyWord: Key Word: Vehicle Key Word: arrow
1 Merlin S01E01 ‘Merlin S01E01’ Mountain
2 Merlin S01E02 ‘Merlin S01E02’
1 Sons of Anarchy Sons of Anarchy S01E02 MerlinS01E01
3 Merlin S01E03 S01E01
4 Merlin S01E04 2 MerlinS01E01 Sons of Anarchy S01E01 MerlinS01E02
5 Merlin S01E05
3 MerlinS01E02 MerlinS01E01 MerlinS01E03
6 Merlin S01E01
7 Merlin S01E02 4 MerlinS01E03 MerlinS01E02 Sons of Anarchy
8 Merlin S01E03 S01E01

9 Merlin S01E04 5 Sons of Anarchy MerlinS01E03 Sons of Anarchy

S01E02 S01E02
10 Merlin S01E05
Table 4: Tiny Videos results, complete MerlinS01E01 as query
Table 5: Semantic Concept Result
Result
No
1 Merlin S01E01
2 Merlin S01E05
3 Sons of Anarchy S01E03
4 Merlin S01E10
5 Sons of Anarchy S01E07
Table 6: CBVR result for MerlinS01E01 as Query, Local Histogram