H.
264 Decoder Implementation on a Dynamically Reconfigurable
Instruction Cell Based Architecture
Adam Major, Ying Yi, Ioannis Nousias, Mark Milward, Sami Khawam and Tughrul Arslan
School of Engineering and Electronics
University of Edinburgh, Edinburgh, EH9 3JL
Tel: (+44)131 650 5619 Email: adam.major@ed.ac.uk
ABSTRACT
This paper presents a new Baseline Profile
compliant h.264 decoder implementation specifically
tailored for an ANSI-C programmable, dynamically
reconfigurable, instruction cell based architecture
which has recently been developed [10]. We use the
ffmpeg libavcodec library as the basis for our
decoder and identify the most processor intensive
functions. These functions are tailored in a novel
framework incorporating established software
techniques alongside several architecture specific
transforms. Initial results demonstrate that our
reconfigurable architecture based decoder provides
a significant performance boost with power figures
below that of a microcontroller such as ARM.
I. INTRODUCTION
Future mobile imaging and video applications,
from personal entertainment systems to aerospace
reconnaissance platforms will demand high image
quality at extreme resolutions whilst remaining
constrained to relatively low transmission
bandwidths, storage capacity and energy resources.
H.264/AVC is the latest video compression standard
from the Joint Video Team (JVT) of the ISO MPEG
and ITU-T VCEG which aims to address the needs
of these next generation video systems [4]. The
standard builds on and extends the feature sets of
previous standards such as MPEG-2, H.263+, and
MPEG-4, and adds several key components such
as an in-loop de-blocking filter; variable block size
quarter pixel precision motion prediction; and
context adaptive binary arithmetic coding (CABAC).
Studies show that this enables it to achieve bit rate
reductions of around 50% compared with MPEG-2
streams of comparable subjective image quality
[9][13]. However, the computational cost associated
with the encoding and decoding algorithms is also
significantly elevated [3][7][16].
At present there are three established
technologies for use in mobile applications: General
purpose Digital Signal Processing cores (DSP),
Application Specific Integrated Circuits (ASIC) and
Field Programmable Gate Arrays (FPGA). Each of
these has advantages but, importantly, none of
0-7803-9782-7/06/$20.00 ©2006 IEEE
Authorized licensed use limited to: The Ohio State University. Downloaded on January 8, 2009 at 08:58 from IEEE Xplore. Restrictions apply.
them can provide a comprehensive solution to all
future mobile requirements. DSPs are highly
flexible, allowing full general purpose
programmability which can be updated at a low cost
after release to respond to changing standards.
They are however significantly slower than
alternative platforms. At the other end of the scale,
ASIC design offers the best possible combination of
high throughput and low power operation but
development is costly and it lacks post-fabrication
flexibility. FPGA solutions are able provide
performance close to that of ASIC with the added
benefit that post-fabrication modifications are
possible. They are less attractive in terms of energy
and area requirements however, due to the large
overhead imposed by complex configuration and
interconnect logic [8]. Therefore, in order to address
all the future requirements of mobile systems we
require new low power, high performance
technologies which are highly flexible and
inexpensive in terms of development and
maintenance costs.
One such technology, which combines an
energy efficient, instruction cell based, dynamically
reconfigurable fabric with ANSI-C programmability,
has recently been developed in industry [10]. This
architecture belongs to the emerging field of
Reconfigurable Computing (RC) and aims to
combine the flexibility and programmability of DSP
with the performance of FPGA and the energy
requirements of ASIC in order to meet the demands
of next generation mobile systems. This paper
provides the initial results of the first h.264 decoder
implemented on a programmable instruction cell
based reconfigurable architecture. We present
timing and performance analyses of the open
source ffmpeg libavcodec library compiled for this
system, and then apply the first stage in a number of
planned optimisations to computationally intensive
code sections in order to improve the throughput.
This paper is organised as follows. Section II
contains a brief overview of previous work. Section
III describes the implementation of code tailoring.
Testing and results are provided in sections IV and
V. Finally, conclusions are drawn in section VI.
49
II. BACKGROUND AND PREVIOUS WORK
A. Target reconfigurable architecture overview
The target core consists of a highly
reconfigurable fabric of interconnected instruction
cells (ICs) with functionality derived from common
machine code instructions which can be dynamically
reconfigured to provide highly parallel FPGA-like
representations of typical software operations. The
fabric is controlled through a series of VLIW
instructions which are extracted from a C based
high level algorithm description by a sophisticated
scheduling algorithm [15]. This enables the
architecture to take advantage of vastly increased
levels of parallelism at the instruction level than
existing processors executing the same code since
it is not constrained by dependencies between
operations or by small branches. As a result it
exhibits a significant performance and flexibility
advantage over traditional solutions. [10]
The resources available to the system are fully
parameterisable with a standard allocation defined
for general purpose applications. At present the
standard core is only around 4 times the area of a
small DSP core such as OpenRISC. When
compared to FPGA, this area equates to between
twice as much for small designs and around 10
times less for large designs. Furthermore, tests
across a range of algorithms indicate energy
requirements of up to 7 times less than DSP and 16
times less than FPGA. This combination of low
power, performance and flexibility makes the
instruction cell based architecture a prime candidate
for the execution of processor intensive image
processing algorithms such as h.264 in a mobile
environment.
B. Code optimisation
The instruction cell based architecture affords
significant scope for code tailoring to achieve even
greater performance. The core has been found to
respond most favourably to techniques which
enable it to take maximum advantage of instruction
level parallelism. In general these techniques
attempt to re-arrange the target algorithm to reduce
the number and density of conditional branches.
Many methods employed to increase performance
on traditional processors, such as loop unrolling,
loop unfolding and loop re-distribution, can be very
effective. Additionally, it is possible to realise small
branches as hardware multiplexers, reducing jumps
and allowing groups of conditional statements to be
evaluated simultaneously. Consequently, throughput
can be increased significantly at the expense of a
few wasted computations.
50
Authorized licensed use limited to: The Ohio State University. Downloaded on January 8, 2009 at 08:58 from IEEE Xplore. Restrictions apply.
Furthermore, the nature of the reconfigurable
fabric is such that a number of conventionally
hardware specific optimisations such as shift
registers and pipelining can be implemented for
appropriate sections of code. This can provide
orders of magnitude greater performance for heavily
iterated loop sections. Since such sections often
account for well over three quarters of the execution
time of an algorithm, these are extremely powerful
techniques.
III. IMPLEMENTATION
For this initial implementation, the standard
libavcodec algorithms were modified based on
findings from several timing profiles of simulated
decode tests which highlighted the most appropriate
target functions for optimisation. In order to assess
the ease of porting existing software to the new core
only minimal and obvious modifications were made
and no advanced methods such as pipelining were
employed.
Loop Transforms: Several standard transforms
were applied selectively to suitable looping sections
of code. Loops with a small number of iterations or a
short string of internal commands were recoded into
a single iteration to reduce the loop overheads and
increase the opportunity to perform commands in
parallel. Longer loops were unrolled by a number of
iterations or re-distributed entirely; for example to
remove a conditional jump from the body of the
loop.
Multiplexer Enforcement: This optimisation relies
on a specialised feature of the reconfigurable core
which is the inclusion of a hardware multiplexer.
Ordinarily branches reduce the number of
computations that must be performed by skipping
sections of code that are not applicable when
certain conditions exist. In the reconfigurable core it
is more efficient, in terms of speed and energy, to
evaluate both branches and then select the
appropriate result using the hardware multiplexer. In
using this technique attention is paid to the relative
costs of evaluating one branch over the other. If one
side requires substantially more computation then it
is often preferable to leave the jump in place and
focus on the body of each branch. However, in
simple situations, where an “if” statement is simply
assigning a value for example, it remains more
efficient to map unbalanced branches as
multiplexers since the benefits of increased
parallelism outweigh the cost of always evaluating
both paths.
Computational Simplifications: A number of
computational simplifications were made to the
code. A scheme of pre-calculation was applied for
common expressions in arithmetic heavy sections to V. RESULTS
reduce duplication of operations.
Memory Access Reduction: Many operations in A. Optimisation
the h.264 algorithm repeatedly require the same Table 1 presents the performance achieved by
piece of input data for a number of calculations. FIR our decoder for each test sequence using both
filter like sections for example will access the same modified and original decoding algorithms.
data location for n results; where n is the number of Executing the standard code, performance ranges
taps. In the libavcodec implementation of the from 12,407 Macro-Blocks per second (MB/s) for
decoder, sets of input data are often stored in an complex scenes to 22,424MB/s for simple
array, which not only incurs a high indexing sequences. The average output bandwidth of
overhead but also translates into one or several 16,848MB/s exceeds the requirements of Baseline
calls to main memory every time the data is Profile Level 2 [4].
required. After minor modification the performance of the
In hardware representations this data would be algorithm on the reconfigurable core is improved by
stored in fast local registers to improve access over 28%. The optimised decoder speed ranges
times. We are able to enforce a similar scheme for from 15,944MB/s to 28,877MB/s. The average
the reconfigurable core by assigning repeatedly output bandwidth is 21,654MB/s and hence this
accessed locations in the input array to non array initial implementation realises just over half of the
variables and thereafter referencing the data speed required for Level 3 compliance [4].
through them. In this way the data is stored in the
core’s internal registers and the memory access Table 1: Decoder Performance in frames per second
overhead is significantly reduced. Sequence Standard Optimised Improvement
Driving 11.64 14.93 28.25%
IV. TEST SETUP Opening 16.61 21.39 28.78%
Whale 9.19 11.81 28.56%
A. Test Sequences Average 12.48 16.04 28.53%
The test sequences “Driving”, “Opening
Ceremony” and “Whale Show” were used to perform
both the initial complexity analysis and subsequent B. Comparison with Existing Platforms
performance assessment. Each sequence uses Table 2 compares our decoder with a selection
YUV 4:2:0 format and consists of 200 frames at a of existing implementations. TIVR Communications
resolution of 720x480. The characteristics and h.264 Baseline Profile Software is aimed at mobile
complexity of each sequence varies which allows platforms powered by ARM and Xscale processors
the performance of the decoder over a range of [12]. The software enables a 220MHz ARM9 based
scene types to be roughly assessed. device to achieve real time decoding of QVGA
The “Opening Ceremony” sequence is relatively (320x240) video streams at 25fps. This equates to a
low in complexity comprising a static backdrop throughput of 7500 MB/s with energy requirements
overlaid with small areas of simple motion as teams of 7.33µJ per decoded Macro Block (µJ/MB). Using
march into an Olympic stadium. Comparatively, tailored code our decoder provides three times this
“Whale Show” is highly complex combining a high performance whilst consuming six times less
level of object motion with fast and irregular camera energy.
panning as the scene follows a whale performing at TIVR’s solution has been highly optimised and
a wildlife park. The complexity of the “Driving” tailored for ARM based systems. In order to gauge
sequence falls between the other two scenes and the true performance differential between ARM and
consists of a regular slow moving background with the reconfigurable architecture the source code was
an essentially stationary object at the image focus. also executed on a simulated ARM9TDMI core
using the ADS 1.0.1 ARMulator [2]. The simulated
B. Encoder Settings core uses an idealised memory bus and default flat
Test sequences were encoded using the JVT memory map which assesses its maximum
Joint Model 10.2 reference software (JM 10.2) theoretical speed. Compared with a similarly
configured using the default baseline profile settings idealised reconfigurable core the ARM performance
[5]. A model with constant QP was used since the is eight times slower. If a similar differential existed
rate control supported by the software is not fully between our decoder and TIVR ARM, the
mature. Hence complex scenes are allocated higher throughput achieved would be close to 60,000MB/s.
bit rates and present a more complex task to the The second device in Table 2 is an AHB AMBA
decode algorithm. based SoC ASIC decoder design proposed in [6].

51

Authorized licensed use limited to: The Ohio State University. Downloaded on January 8, 2009 at 08:58 from IEEE Xplore. Restrictions apply.
The device combines a high performance ARM
microcontroller with a pipelined set of application
specific hardware blocks which enable it to realise a
real time 1080HD decoder at 30fps. To achieve this,
the core must run at 170MHz at which it consumes
554mW which equates to 2.28µJ/MB. Compared
with our decoder this design requires roughly twice
the energy to decode a single MB.
The last device in table 2 is a Baseline decoder
core for Xilinx Virtex and Spartan FPGAs developed
by 4i2i [1]. Documentation for the core states clock
speeds required for a number of standard video
resolutions alongside the maximum frequency
attainable when implemented on a Xilinx Virtex-II
array. Extrapolating from this data yields a
theoretical 79,200MPB/s upper throughput limit
which equates to roughly 4 times the speed of our
current decoder implementation.
To approximate the energy required by this
design to decode a single Macro-Block we refer to
the findings of [11] which estimates the power
consumption of a Virtex-II CLB in a typical design to
be 5.9µW per MHz. The decoder has a stated area
of 15000 slices which equates to 3750 CLBs in a
Virtex-II array [14].
Table 2: Comparison of h.264 decoder implementations
Throughput
Platform
Macro-Blocks / s
ARM9 [12] 7,500
Xilinx Virtex II [1] 79,200
ASIC [6] 243,000
our decoder 16,848
our decoder (tailored) 21,654
VI. CONCLUSIONS
This paper has presented the first h.264 decoder
implementation on a reconfigurable, instruction cell
based architecture. Currently the decoder achieves
between 12fps and 21fps at NTSC D1 after minor
modification to the standard libavcodec based code.
Compared to existing RISC based solutions our
decoder exhibits significant performance
advantages, attaining three times the speed of a
highly optimised ARM9 solution and eight times the
speed of an ARM9 executing the same code.
Furthermore, it vastly outperforms DSP, FPGA and
ASIC accelerated microcontroller designs in terms
of energy requirements.
The aim of future implementations will be to
match and exceed the throughput of FPGA and
ASIC based devices by applying a deeper level of
modifications to the critical modules of the decoder.
Amongst the candidate functions will be the in loop
de-blocking filter, sub-pixel motion compensation
and Inverse DCT. The tailoring of each module will
Authorized licensed use limited to: The Ohio State University. Downloaded on January 8, 2009 at 08:58 from IEEE Xplore. Restrictions apply.
focus on a combination of established h.264
algorithmic simplifications, for both hardware and
software, with the advanced methods available to
the reconfigurable fabric, such as pipelined loops
and shift registers.
REFERENCES
1. 4i2i Communications technical staff, H.264/MPEG-4 Part-10
Baseline Video Decoder IP Core, 4i2i Communications Ltd.,
revision 1.0, 2005
2. Advanced RISC Machines, AXD Debugger for ADS 1.0.1,
ARM Ltd., 1999-2000
3. A. Chang, O.C. Au, Y.M. Yeung: “A novel approach to fast
multi-block motion estimation for H.264 video coding”
Multimedia and Expo, 2003. ICME '03. Proceedings. 2003
International Conference on, Volume 1, 6-9 July 2003
Page(s):I - 105-8 vol.1
4. Joint Video Team of ITU-T and ISO/IEC JTC 1, “Draft ITU-T
Recommendation and Final Draft International Standard of
Joint Video Specification (ITU-T Rec. h.264 | ISO/IEC 14496-
10 AVC),” Joint Video Team (JVT) of ISO/IEC MPEG and
ITU-T VCEG, JVT G050, March 2003.
5. Joint Video Team (JVT), Reference Software JM 10.2,
October, 2005
6. H. Kang, K. Jeong, J. Bae, Y. Lee, S. Lee: “MPEG4
AVC/H.264 Decoder with Scalable Bus Architecture and Dual
Memory Controller”, Circuits and Systems, 2004. ISCAS '04.
Proceedings of the 2004 International Symposium on,
Volume 2, 23-26 May 2004 Page(s):II - 145-8 Vol.2
7. J. Ostermann, J. Bormans, P. List, D. Marpe, M. Narroschke,
F. Pereira, T. Stockhammer, T. Wedi, “Viedo coding with
Energy
H.264/AVC: Tools, Performance, and Complexity”, IEEE
µJ/Macro-Block
Circuits and Systems Magazine, First Quarter 2004, pp. 7-
7.33 28.
15.64 8. Power Comparison, RapidChip© Platform ASICs vs. FPGA
2.28 9. Raja, G.; Mirza, M.J.; “Performance comparison of advanced
1.49 video coding H.264 standard with baseline H.263 and H.263+
standards” Communications and Information Technology,
1.15 2004. ISCIT 2004. IEEE International Symposium on,
Volume 2, 26-29 Oct. 2004 Page(s):743 - 746 vol.2
10. Reconfigurable Instruction Cell Array, U.K. Patent
Application Number 0508589.9.
11. L. Shang, A. S. Kaviani, and K. Bathala. “Dynamic power
consumption in Virtex-II FPGA family”, Proceedings of the
2002 ACM/SIGDA tenth International Symposium on Field
Programmable Gate Arrays, pages 157–164. ACM Press,
2002.
12. TIVR technical staff, H.264 Baseline Profile (BP) Video
Decoder, TIVR Communications Private Ltd.
13. Wiegand, T.; Sullivan, G.J.; Bjntegaard, G.; Luthra, A.;
“Overview of the H.264/AVC video coding standard”
Circuits and Systems for Video Technology, IEEE
Transactions on, Volume 13, Issue 7, July 2003
Page(s):560 – 576
14. Xilinx, Virtex-II Platform FPGAs: Complete Data Sheet, Xilinx
Inc, DS031, version 3.4, 2005
15. Y. Yi, I. Nousias, M. Milward, S. Kawham, T. Arslan and I.
Lindsay: “System-level Scheduling on Instruction Cell Based
Reconfigurable Systems”, Presented at Design, Automation
and Test in Europe Conference 2006 (DATE 2006), 6-10
March, ICM, MESSE Munich, Germany
16. J. Zhang, Y. He, S. Yang, Y. Zhong: “Performance and
complexity joint optimization for H.264 video coding”, Circuits
and Systems, 2003. ISCAS '03. Proceedings of the 2003
International Symposium on, Volume 2, 25-28 May 2003
Page(s):II-888 - II-891 vol.2
52