INTRODUCTION TO THE OPENCORE VIDEO COMPONENTS USED IN THE ANDROID PLATFORM JIM KOSMACH KRISDA LENGWEHASATIT DUSAN VESELINOVIC

Document Sample
INTRODUCTION TO THE OPENCORE VIDEO COMPONENTS USED IN THE ANDROID PLATFORM JIM KOSMACH KRISDA LENGWEHASATIT DUSAN VESELINOVIC Powered By Docstoc
					         INTRODUCTION TO THE OPENCORE VIDEO COMPONENTS
                  USED IN THE ANDROID PLATFORM

                    JIM KOSMACH, KRISDA LENGWEHASATIT, DUSAN VESELINOVIC,
                                 GREG SHERWOOD, RALPH NEFF

                             PacketVideo Corporation, San Diego CA, USA
                                           kosmach@pv.com
                                            krisda@pv.com
                                         veselinovic@pv.com
                                           sherwood@pv.com
                                             neff@pv.com



Video codecs such as H.263, MPEG4, and AVC are used extensively on mobile devices throughout the world. In the
ideal case, such codecs rely on hardware acceleration. However, it is also very common to see software video codecs
running on the main application processor, which is often an ARM core processor. Such codecs must be memory
efficient, processing cycle efficient, portable to multiple operating systems, robust to data loss, and must also have a
modular interface. In this paper, we introduce the OpenCORE multimedia framework and associated optimized
software video codecs that are a part of the Android platform. We show how these components meet the challenging
requirements for use in mobile devices. The OpenCORE video components are currently available from the Open
Handset Alliance as part of the Android SDK, and the source code for these components is scheduled for release in late
2008. The components are thus freely available for use in mobile device projects, and for non-mobile projects as well.


INTRODUCTION                                                   awaited “G-Phone.” Nobody was quite sure what this
The current paper provides an introduction to the              phone was, but many believed or hoped that it would be
OpenCORE multimedia framework and associated                   Google’s answer to the iPhone. Various web articles
video codecs that are a part of the Android platform           were guessing at what revolutionary features the phone
from the Open Handset Alliance. The OpenCORE                   might contain, and some suggested that the phone
multimedia framework provides a modular interface that         would be commercially available in early 2008.
allows the associated video codecs to be easily used and       However, when the actual announcement came in
configured. The framework may be used purely for               November 2007, it was clear that there would be no
video, or in conjunction with companion components             single “G-Phone” coming from Google. Instead, there
such as audio codecs or file format elements, to support       would be a revolutionary new mobile phone platform.
many typical multimedia scenarios. OpenCORE video              The platform’s name was Android, and it would come
codecs are highly optimized for speed, efficient use of        not only from Google but from an alliance of companies
memory, robustness and portability. The codecs are             from around the mobile communications industry [1].
based on PacketVideo’s commercial video codecs,
which have been deployed in over 260 million handsets          Android is the first complete platform for mobile
to date.                                                       devices that is truly free and open. By “open,” we mean
                                                               the operating system, middleware, UI, and an
Section 1 introduces the Android Platform and provides         assortment of typical mobile applications will all be
some relevant history.         Section 2 describes the         available as open-source which may be seen and
OpenCORE multimedia framework on which the                     modified, thus removing many of the development
Android multimedia subsystem is built. Section 3               limitations present on traditional mobile platforms.
describes the OpenCORE video codecs, and provides              Thus the development of mobile applications and
details about the availability, performance, features, and     services for Android will be open in the same way that
usability of the codecs.                                       the development of general internet applications and
                                                               services has been for years. By “free,” we mean the
                                                               complete Android source code will be made available
1 THE ANDROID PLATFORM                                         for use in commercial products under a free software
                                                               license. This should enable manufacturers to produce
In fall of 2007, the mobile phone industry was abuzz
                                                               inexpensive Android-based handsets, since software
with rumors that Google would soon release its long


                         1st International Workshop on Video Coding and Video Processing                              1
                                      Shenzhen, China -- November 26-28, 2008
license fees are typically a significant part of a handset’s   the scope of the current paper, but this section will point
Bill of Materials (BOM).                                       out some of the highlights.

Although Android was initiated by Google, it has               The lowest layer provides the interface to operating
evolved into a collaborative project with membership           system APIs and platform services needed by the
from 34 companies, many of which are key players in            framework including memory management, DNS
the mobile phone industry. The resulting organization          lookup, network and file I/O, process and thread
is called the Open Handset Alliance (OHA) [2].                 control, etc.     Although not shown as a separate
                                                               functional block, the multimedia framework defines
Within a week of the initial November 2007 Android             methods and data structures for negotiating formats and
announcement, the OHA released an “early-look”                 parameters between components, and for passing media
version of the Android SDK. This SDK included                  data between components. The design has provisions
development tools, a device emulator, documentation            that minimize the copying of media data as it flows
and sample projects, as well as the rich set of libraries      through the resulting graph.
that implement the Android system functionality. Many
developers downloaded the SDK and began writing
applications based on it. As a result, Google announced
in April 2008 that nearly 1800 Android applications                          Android Application Layer
were submitted in response to the first phase of its
Android Developer Challenge contest [3]. The winning
applications were announced in August 2008 [4].
                                                                                    OpenCore
The first commercial mobile phone based on Android is                         Content Policy Manager
the T-Mobile G1. Manufactured by HTC, the phone is
expected to be available to consumers in October 2008                           Multimedia Eng ines
[5].
                                                                                    Data Formats
The complete Android source code is scheduled for
release by the end of 2008. The Linux Kernel which
powers Android will be available under the free GNU
                                                                         Video Codecs         Audio Codecs
Public License v2.0 (GPL2.0) [6]. The rest of the
                                                                                   Linu x Interface
Android code, including that of the video components,
will be made available under the equally free but less
commercially restrictive Apache 2.0 license [7].

PacketVideo [8] was chosen to be the multimedia sub-                          Linu x Operating System
system provider for Android.          Thus PacketVideo
provides the multimedia framework and associated
                                                               Figure 1 – High-level view of the major OpenCORE
components (e.g audio and video codecs, file format
                                                               functional blocks.
parsing and authoring, streaming components, etc.)
which power Android’s multimedia experience and
which will be freely available to the Android developer
                                                               The audio codec block includes GSM-AMR NB and
community. These components are collectively called
                                                               WB, MP3, AAC, HE-AAC v1, and HE-AAC v2 along
OpenCORE, and include the multimedia framework and
                                                               with interfaces to integrate other formats and hardware
codecs that are the focus of the current paper.
                                                               accelerated codecs. The video codec block includes
                                                               MPEG4, H.263, and H.264 video codecs. Similarly the
                                                               video codec block includes interfaces to integrate other
2 OPENCORE MULTIMEDIA FRAMEWORK                                formats and hardware accelerated codecs. The included
OpenCORE is a modular, extensible framework for                video codecs are described in more detail in Section 3.
combining independent media processing components
such as file formats, codecs, streaming protocol               The data formats block handles the reading and writing
components, rendering components, and other elements           of file formats, as well as support for various streaming
in different ways to implement a wide variety of               protocols. Supported file formats include .mp3, .mp4,
multimedia scenarios. Figure 1 shows a very high-level         .3gp, .aac, .amr, and .wav.         Supported streaming
view of the major functional blocks of OpenCORE. A             protocols include RTSP and HTTP, the latter including
detailed description of the entire framework is beyond         both simple and “progressive” download functionality.

                         1st International Workshop on Video Coding and Video Processing                                2
                                      Shenzhen, China -- November 26-28, 2008
The multimedia engines block provides the logic for               components to be developed once and then easily
selecting and connecting components into the                      integrated across multiple operating systems and
appropriate graph structure as well as providing the high         hardware platforms. Figure 2 shows the modular
level controls to the application for a given multimedia          OpenCORE design that takes advantage of the
use-case. Finally, the Content Policy Manager block               OpenMAX IL interface.
provides a framework for content access and control,
which can be used for integration of DRM functionality.           The OpenCORE multimedia framework provides OMX
                                                                  components that are used to integrate the audio and
In a typical scenario, the Multimedia Engines block               video codecs. An OMX component serves as a wrapper
would be used to assemble a graph containing                      around the video codec, and provides compatibility to
appropriate file formats, audio and video decoders, and           the OpenMAX IL API set. Most of the APIs are
rendering interfaces in order to achieve local playback           asynchronous, meaning that the OMX component
of media files. Other graphs could be assembled to                queues the commands and input data buffers which it
achieve other typical scenarios, such as media                    receives via the OpenMAX Integration Layer, and
authoring/recording, or streaming media playback. The             processes them at a later time. As an example, consider
availability of standard components with modular graph            the video decoding case. When scheduled to run, the
assembly should allow Android application developers              OMX component may process one or more queued
to easily access and make effective use of the available          input data buffers in order to decode the associated
multimedia resources.                                             video bit stream data using the connected software
                                                                  video decoder. The OMX component then returns the
                                                                  processed input buffer back to the OpenMAX Video
3 OPENCORE VIDEO CODECS                                           Codecs Node via the OpenMAX integration layer.
The OpenCORE video codecs are listed in Table 1,
along with the supported profiles and levels. Table 1
                                                                          PV OpenMAX Video Codecs Node
also lists the maximum frame size, maximum frame rate
(FPS), and maximum bit rate (Kbps) associated with
each supported profile and level combination. Support is
provided for H.263, MPEG-4 part 2, and AVC encoders
and decoders.                                                                 OpenMAX Integration Layer

                                         Max     Max    Max           H.263 / MPEG4 Part 2             AVC
  Codec        Profile           Level   Size    FPS    Kbps            OMX Component              OMX Component

                                  10     QCIF     15     64
                  0               20     CIF      15    128
              (Baseline)          30     CIF      30    384
                                  45     QCIF     15    128           H.263 / MPEG4 Part 2               AVC
  H.263                                                                 Software Codecs             Software Codec
                                  10     QCIF     15     64
                  3               20     CIF      15    128
              (Wireless)          30     CIF      30    384
                                  45     QCIF     15    128
                                   0     QCIF     15     64
                                                                     Figure 2: OpenCORE modular design for Android
                                  0b     QCIF     15    128
 MPEG-4
                Simple             1     QCIF     15     64
  part 2                                                          Where applicable, hardware accelerated codecs may
                                   2     CIF      15    128
                                   3     CIF      30    384       also be included to speed up the video encoding or
                                   1     QCIF     15     64
                                                                  decoding process. The modular design of the Video
               Baseline           1b     QCIF     15    128       Codecs Node allows for the existence of multiple
   AVC        (constraint_        1.1    QCIF     30    192       alternate codecs to support a particular codec standard.
             set1_flag = 1)       1.2    CIF      15    384       For example, a particular mobile device may have
                                  1.3    CIF      30    768       hardware AVC decoder support, in addition to the
                                                                  optimized software AVC decoder provided as part of
       Table 1: OpenCORE video codec support                      the Android package. The Video Codecs Node may
                                                                  provide support for both decoder options, and may
3.1 Modular Design                                                allow either decoder to be loaded and used at run time.
                                                                  The choice of which underlying decoder to use may
OpenCORE has adopted the OpenMAX Integration
                                                                  depend on a use case scenario or on the availability of
Layer (IL) interface [9]. OpenMAX IL is a well-
                                                                  resources.
known cross-platform API that allows multimedia

                              1st International Workshop on Video Coding and Video Processing                           3
                                           Shenzhen, China -- November 26-28, 2008
3.2 Optimizations                                           for many operating systems, including Win32, Symbian,
The OpenCORE video codec libraries were developed           Linux, and ARM-Linux. However, the initial releases
over a period of several years, with a focus on             of Android will include only the Linux port of OSCL,
optimization for mobile embedded devices. The video         since the Android platform is Linux-based.
codec team started the development process by deeply
studying the various standards [10][11][12] and the         The OpenCORE video codecs are based on
corresponding reference code.                               PacketVideo commercial codecs that have been
                                                            deployed on nearly 250 different device models across a
The first level of implementation and optimization          wide variety of operating systems and compilers
involved the detailed design of the control and data flow   representing over 260 million shipped units.       The
architecture. The codec architectures are macroblock        environments that have been supported include:
based, allowing for an efficient pipeline as each               • Linux (using gcc compiler)
macroblock transitions through the encoding and                 • Symbian (using gcc and rvct compilers)
decoding components of the corresponding codec.                 • Brew (using ads and rvct compilers)
Additionally, various algorithms were developed to take         • Windows CE (using Microsoft compiler)
advantage of the ARM processor resulting in additional          • Nucleus (using ads compiler)
performance gains.       Examples are the use of count
leading zero (CLZ) instructions in the VLC decoding         3.3 Performance
process, the use of ARM DSP multiplications when            The OpenCORE video codecs have been optimized with
computing DCT and IDCT, and the use of saturation           ARM’s ADS/RVCT compiler and profiled on the
arithmetic for clipping.                                    ARMulator (ARM emulator). Table 2 shows the zero-
                                                            wait state performance of the codecs as profiled on the
The video codec libraries are written mainly in C. Most     ARMulator using an ARM9E processor.                The
of the performance gains are achieved through efficient     performance measure is in MHz and represents the
implementation and optimization of the C code, both at      number of megacycles consumed on the ARM9E per
the algorithmic level and the coding level. At the          second in order to perform the encoding or decoding in
algorithmic level, variable complexity algorithm            real-time.
concepts [15][16][17][18][19] are used in many areas
including DCT and IDCT computation, motion
compensation, VLC decoding, and dequantization. Fast          Codec          Resolution    fps     kbps     MHz
algorithms for DCT and motion estimation are also             Type
employed.
                                                              MPEG4 SP         QCIF          15      64       7.9
                                                              Decoder          QVGA          30     384      44.1
At the coding level, we typically follow C level
                                                              MPEG4 SP         QCIF          15      64      38.3
programming techniques which are known to perform
                                                              Encoder          QVGA          30     384     213.5
well on ARM processors [20]. We have observed that
such techniques often produce good results when the           AVC              QCIF          15      64      30.1
code is complied for non-ARM processors as well. In           Decoder          QVGA          30     384      188
addition, we supplement the code in limited key areas
using optimized inline assembly code for the ARM core
processor. The inline assembly code will be included             Table 2: ARMulator performance results for
in the initial Android open source code release, and so      OpenCORE video codecs on zero-wait state ARM-9E.
will be visible and accessible to developers.

The OpenCORE video codec libraries make use of an           Note that while H.263 results are not provided in Table
efficient internal memory management process. A pool        2, the performance of the H.263 codec is quite similar to
of memory is allocated to the library upon instantiation,   the results listed for the MPEG-4 SP codec. It should
and this memory is recycled and reused throughout the       also be noted that the MPEG-4 SP encoder performance
decoding session. This prevents the decoding process        may vary greatly depending on the selected encoding
from incurring delay due to dynamic memory allocation       options. The results shown here are based on a
requests to the OS.                                         combination of encoding tools selected by the authors in
                                                            order to maintain a reasonable balance between
The OpenCORE video codec libraries make intrinsic           embedded system performance and compression gain.
use of an OS portability layer called OSCL. Thus the        Specifically the results are based on a motion search
libraries may be easily ported to new operating systems     range of +/- 16 pixels, with INTER4v mode turned off,
by simply porting the OSCL layer. OSCL ports exist          and with the available Rate-Distortion optimized mode
                                                            selection also turned off.

                        1st International Workshop on Video Coding and Video Processing                             4
                                     Shenzhen, China -- November 26-28, 2008
                                                            3.4 Robustness, Error Resilience and Concealment
Table 3 shows an alternate result in which the              Robustness, error resilience, and concealment are
OpenCORE video codecs were compiled using arm-              extremely important for video codecs that target a
linux-g++ compiler (version 3.4.0) and were run and         mobile environment. The OpenCORE video decoders
profiled on a PacketVideo reference platform based on       are very robust to packet loss, frame loss and bit stream
OMAP 2420. Note that the OMAP 2420 contained an             errors due to streaming video over wireless networks.
ARM 11 processor running at 266 MHz, and also a             The codecs have been used extensively in commercial
C55x DSP running at 220 Mhz. The OpenCORE video             mobile handsets, both in packet-based streaming
codecs were in all cases running entirely on the ARM        applications (e.g. 3GPP Packet Switched Streaming
11.                                                         Services [13]), and in circuit switched applications (e.g.
                                                            3G-324m Mobile Video Telephony [14]). This global
Comparing the results in Table 3 to corresponding Table     test bed has allowed us to refine and improve the codecs
2 results, we see that the required MHz is higher for the   over the years, and as a result the codecs are very stable
OMAP 2420 experiment than for the ARM-9E                    and robust.
emulator. This matches our expectations, as real
hardware typically exhibits higher memory access            Error concealment is handled in the decoders by
latency than the emulator environment.                      detecting when a frame or a portion of the frame is
                                                            missing or damaged, and utilizing previously decoded
                                                            information to conceal the missing or damaged portion.
 Codec           Resolution    fps     kbps     MHz         The detection process is often handled outside of the
 Type                                                       decoder since the higher layers above the codec are
 MPEG4 SP           QCIF         15      64       10        often aware that a frame or portions of a frame have
 Decoder                                                    been dropped or damaged in transport. Similarly, the
                   QVGA          30     384       86        decoders may be capable of identifying and detecting
                                                            missing frames or corrupted bit streams without
 MPEG4 SP           QCIF         15      64       63        signalling from the higher layers. The optimized
 Encoder                                                    software codecs included with Android have such
                   QVGA          30     384      333        capability, and many alternative codecs (including
                                                            hardware-accelerated codecs) provide such capability as
 AVC                QCIF         15      64       37        well. The Video Codecs Node may provide error
 Decoder                                                    detection signalling to inform the higher layers in the
                   QVGA          30     384      252        case where bit stream corruption is detected by the
                                                            underlying decoder.

 Table 3: OpenCORE ARM11 performance data on an             There are many elaborate schemes for implementing
               OMAP 2420 platform                           error concealment, though some can be quite MIPS
                                                            intensive. The OpenCORE video decoders provide for a
                                                            simple mechanism for concealment that is very efficient
We additionally note that each scenario listed in Table 3
is able to run in real time on the 266 MHz ARM 11,          for an embedded solution. When an error is detected for
except for the QVGA MPEG4 SP encoding experiment.           a given frame, the decoders copy the co-located portions
                                                            of the previous frame. We have found this method to
Real time encoding in this case could be achieved using
                                                            produce reasonable results. Additionaly, it does not
a lower frame rate (e.g. 20 fps). Alternately it could be
                                                            cost much in terms of code size, memory usage, and
acheived on a faster processor, or by making use of
                                                            computational complexity.
hardware acceleration features of the OMAP 2420.

The performance of the OpenCORE video codec
libraries, written in C code with limited inline assembly   4 CONCLUSION
optimization, closely matches and often exceeds the         Android is the first truly complete, free, and open
performance of assembly-only implementations that are       platform for mobile devices. PacketVideo's OpenCORE
typical of other third party codecs. Maintaining most       components provide the multimedia functionality for the
of the code in C has the advantage that the code remains    Android platform. OpenCORE provides a flexible and
easily portable, and that it remains easy to work on the    modular      framework     for   assembling      useful
algorithmic and architecture elements of the code.          configurations of the provided audio and video codecs
                                                            and related multimedia elements. The audio and video
                                                            codecs included in OpenCORE are highly optimized,



                        1st International Workshop on Video Coding and Video Processing                             5
                                     Shenzhen, China -- November 26-28, 2008
configurable, and easily adaptable to             many      [11]   ISO/IEC         14496-2:2004:   “Information
combinations of OS, processor, and compiler.                       Technology – Coding of audio-visual objects –
                                                                   Part 2: Visual”
There will be many interesting devices based on the
Android platform in the near future. The OpenCORE           [12]   ITU-T Recommendation H.264, “Advanced
video components should increase the appeal and value              video coding for generic audiovisual services,”
of these devices by making it easy to develop innovative           November 2007.
and exciting multimedia applications.
                                                            [13]   3GPP TS 26.234, “Transparent end-to-end
                                                                   Packet-switched Streaming Service (PSS);
5 ACKNOWLEDGMENTS                                                  Protocols and codecs,” Release 7.
The authors wish to thank Muhammed Coban, a former
member of PacketVideo’s codec team, for his efforts in      [14]   3GPP TS 26.111, “Codec for Circuit-Switched
the development and optimization of the OpenCORE                   Multimedia Telephony Service, Modifications to
video codecs.                                                      H.324,” Release 7.

REFERENCES                                                  [15]   K. Lengwehasatit and A. Ortega, "DCT
                                                                   computation with minimal average number of
[1]    “Industry Leaders Announce Open Platform for                operations," in Proceedings of VCIP 1997, San
       Mobile Devices,” Press Release from the Open                Jose, California, February 1997.
       Handset Alliance, November 5, 2007.
                                                            [16]   K. Lengwehasatit and A. Ortega, "Probabilistic
[2]    Open Handset Alliance web site,                             Partial Distance Fast Matching for Motion
       http://www.openhandsetalliance.com                          Estimation", IEEE Transactions on Circuits
                                                                   and Systems for Video Technology, Vol. 11,
                                                                   No. 2, pp. 139-152, February 2001.
[3]    Announced on the Android Developer’s Blog
       site, April 17, 2008.
                                                            [17]    I.-M. Pao and M.-T. Sun, “Modeling DCT
        http://android-developers.blogspot.com/
                                                                    coefficients for fast video encoding,” IEEE
                                                                    Transactions on Circuits and Systems for Video
[4]    Android Developers Challenge, Winner’s
                                                                    Technology, volume 9, pp. 608-616, June 1999.
       Gallery, August 28, 2008.
       http://code.google.com/android/adc_gallery/
                                                            [18]    Z. Wang, “Fast algorithms for discrete W
                                                                    transform and for the discrete Fourier
[5]    T-Mobile Press Release, “T-Mobile Unveils the
                                                                    transform,” IEEE Transactions on Signal
       T-Mobile G1 – the First Phone Powered by
                                                                    Processing, volume ASSP-32, pp 803-816,
       Android,” September 23, 2008.
                                                                    August 1984.
[6]    GNU Public License v2.0, June 1991. Available
                                                            [19]   Y. Arai, T. Agui, and M. Nakajima, “A Fast
       at http://www.gnu.org/licenses/old-licenses/gpl-
                                                                   DCT-SQ Scheme for Images,” Transactions of
       2.0.html
                                                                   the IEICE, E 71(11):1095, November 1988.
[7]    Apache Software License, Version 2.0, January
                                                            [20]    ARM, Application Note 34, “Writing Efficient
       2004. Available at:
                                                                    C for ARM”, issued January 1998 . Available at
       http://www.apache.org/licenses/LICENSE-
                                                                    http://www.arm.com/.
       2.0.txt

[8]    PacketVideo web site, http://www.pv.com/

[9]    OpenMAX IL specification and tests, available
       at http://www.khronos.org/openmax/

[10]   ITU-T Recommendation H.263, “Video Coding
       for Low Bit Rate Communication,” January
       2005.



                        1st International Workshop on Video Coding and Video Processing                            6
                                     Shenzhen, China -- November 26-28, 2008