Video Compression Standards
Wen Gao
Peking University
Outline
• Technology issues
• IPR issues
• Effort in RF or low price IPR
• Future standards
• Summary
2011-11-10 http://www.avs.org.cn 2
Video coding standards – how many
aspects need to look at?
• At least two
– Technical achievement on removing
redundancy
– IPR policy movement
2011-11-10 http://www.avs.org.cn 3
Three issues on video coding
technology
1. Where the redundancy come from, how
many kind of redundancy can be removed
2. What is the up-bound for video coding
(lossless/lossy)
3. Why standard progress is always in
traditional way (keeping in hybrid coding)
2011-11-10 http://www.avs.org.cn 4
Answer to issue 1
• Spatial redundancy
– From over sampling
– Can be removed by
• Transform/prediction
• Analysis/reconstruction (inpainting, texture coding, …)
• Temporal redundancy
– From duplication of no change region
– Can be removed by
• Prediction (motion) coding
• Coding redundancy
– From normal bit allocation for sparse data
– Can be removed by entropy coding
• VLC/arithmetic coding
2011-11-10 http://www.avs.org.cn 5
Major tools in hybrid video coding
Input Coder
Video Control
Control
Signal
Transform coding Data
Transform/
coeffs
Quant
-
Split into Inv Quant/
Macroblocks Inv Transform
16x16 pixels Entropy
Entropy
Coding
De-blocking coding
Intra-frame Filter
Prediction
Prediction coding Output
Motion-
Video
Compensation
Intra/Inter Signal
Motion
Data
Motion
2011-11-10 Estimation
http://www.avs.org.cn 6
6
Spatial Coding Tools
1960s 1970s
Andrews, 1968 Pratt, 1969 Chen, 1973 Ahmed, 1974
Fourier Hadamard Slant Cosine
Transform Transform Transform Transform
Huang 1966 Pratt 1969
Statistical Zonal
Coding Coding Habibi, 1971
Block Tasto 1971
Max, 1960 Huang 1963
Classification
Adaptive
Non- Block Block
Uniform Quantization Coding
Quantizer
Fano 1949
Huffman 1952 Seyler 1962 Chen 1981
Variable- Scene
Length VLC Video Adaptive
Coding VLC Woods 1969
Coder
Uniform Dead-Zone
Quantizer Quantizer
Pratt 1969
Tescher, 1976
Threshold Pratt, 1970
Coding ZZ-Run-
Run-
Tescher, 1975 Length
Length
Coding
Coding Zigzag
Scan Tescher, 1977
C Reader, 2002
Rate
Buffer
2011-11-10 http://www.avs.org.cn 7
Temporal Coding Tools
1960s 1970s
Seyler 1962 Limb 1975 Netravali 1979
Brofferio 1970
Temporal Motion Motion PRA Motion
Statistics Modeling Measurement Estimation
Seyler 1962 Candy 1971 Netravali 1979
Conditional 3D Motion
Update (Temporal) Compensated
(Spatial; Pixel) DPCM DPCM (Pixel)
Harrison 1952 Rocca 1969 Koga 1981 H.120
Conditional Motion
DPCM Update Compensated
(Spatial; Block) Taki 1974 Giorda 1975 DPCM (Block)
BMA Variable –
Motion Size BMA
Estimation Motion Est.
Jain 1979 H.261
Schroeder 1970
Transform in
Hybrid
DPCM Loop
MPEG1
Coding
with MC MPEG2
Etc.
Pratt 1968 Reader 1973 Roese 1975 Netravali 1979
Transform Conditional Hybrid Motion
Coding Update Coding Compensated
(Intra) (Transform) (Transform) Transform
Jones 1978
Interlaced
C Reader, 2002
Field/Frame MPEG2
Coding
2011-11-10 http://www.avs.org.cn 8
Answer to issue 2
• Given a 1080P(30fps) video data, what is the up-
bound of lossless compression?
• Typical MPEG-2 compression is in 20Mbps
– 1080x1920x8x3x30 ≈ 1500Mbps
– 150 : 20 ≈ 75 : 1
• Typical AVC/H.264 compression is in 10Mbps
– 1080x1920x8x3x30 = ≈ 1500Mbps
– 1500 : 10 ≈ 150 : 1
• Can HVC/H.265 do at 300:1?
• Can people achieve at 1000:1 or even higher?
2011-11-10 http://www.avs.org.cn 9
If we can find a ideal transform
X 1920X 1080 X 1920X 1920 X
X X Y
y11 0 ... 0
0 y 22 ... 0
X
Y ... ... ... ...
0
0 ... y NN
• If someone find an ideal transform, then we can get 1920:1
2011-11-10 http://www.avs.org.cn 10
Up-bound?
compression
2000
One possible
1500 up-bound
AVS/H.264
1000
MPEG-1/MPEG-2
500
VCD DVD HDTV
2011-11-10 http://www.avs.org.cn 11
Real up-bound？
• By that up-bound, we can have
– DVD(720x576), 720 : 1
– HDTV(1920x1080), 1920 : 1
– XDTV(4000x2000), 4000 : 1
• If that is true, considering temporal factor,
then we should have at least 1 order higher
that just deal that in one frame
• Please let me know, if you get the real up-
bound
2011-11-10 http://www.avs.org.cn 12
Standards still have enough space to
work
• ISO/IEC MPEG
– MPEG-1/2/4/AVC/HVC
• ITU-T VCEG
– H.261/2/3/4/5/???
Next G
AVS
MPEG-4 AVC
MPEG-4
MPEG-2
MPEG-1
2011-11-10 http://www.avs.org.cn 13
1989 1993 1997 2001 2005 2009 2013 2017
Answer to issue 3
• After big success of MPEG-2, we do need
to face legacy resource (content, product in
use, technology, …)
• In simple word, industry want to keep their
early investment valuable
2011-11-10 http://www.avs.org.cn 14
History of Video coding Standards
• ISO/IEC recommended MPEG series standards
– MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC
• ITU-T recommended H.26X series standards
– H.261, H.262, H.263, H.263+, H.263++, H.264
ITU-T H.261 H.263 H.263+ H.263++
Joint H.262/ H.264 H.264 H.264
ITU-T/MPEG MPEG-2 BP/MP HP SVC
MPEG MPEG-1 MPEG-4
MPEG-China / AVS AVS
AVS 基准 加强
1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
2011-11-10 http://www.avs.org.cn 15
Tools in H.261, MPEG-1, MPEG-2
Input Coder
Video Control
Control
Signal
Data
Transform/
coeffs
Quant
-
Split into Inv Quant/
Macroblocks Inv Transform
16x16 pixels Entropy
Coding
De-blocking
Intra-frame Filter
Prediction
Output
Motion-
Video
Compensation
Intra/Inter Signal
Motion
Data
Motion
2011-11-10 Estimation
http://www.avs.org.cn 16
16
Tools in AVC/H.264, AVS
Input Coder
Video Control
Control
Signal
Data
Transform/
coeffs
Quant
-
Split into Inv Quant/
Macroblocks Inv Transform
16x16 pixels Entropy
Coding
De-blocking
Intra-frame Filter
Prediction
Output
Motion-
Video
Compensation
Intra/Inter Signal
Motion
Data
Motion
2011-11-10 Estimation
http://www.avs.org.cn 17
17
Main Tools & Standards
1950s 1960s 1970s 1980s 1990s 2000s
Entropy Coding 1949-1976
DPCM 1952-1980
Motion Compensated Prediction 1972-1989
Transform Coding 1965-1980
H.261 1984-1990
JPEG 1984-1992
MPEG1
MPEG2
H.263
MPEG4
AVC/H.264
VC-1
AVS
Modified From C Reader
2011-11-10 http://www.avs.org.cn 18
Video coding standards IPR
issue
• What base to pay
– Device base by end user/manufacture/chip
company/software company
– Usage base by content provider/operator
• How much to pay
– Pricing, by percentage/fixed rate
• How long have to pay
– Patent life time, 17year/20year
2011-11-10 http://www.avs.org.cn 19
Who control MPEG IPR policy
• Non-MPEG organization
• Profitable company
• Maintain MPEG-2, MPEG-4, MPEG-4
AVC, VC-1,…
2011-11-10 http://www.avs.org.cn 20
MPEG-X/H.26x IPR policy
• MPEG-2 licensing term weakness – fixed price
– IPR policy for patents in MPEG-2 is good in beginning, but not
good now
– 4.0$/device => 2.5$/device (after 2002)
– Now cost for MPEG-2 STB is under 15$, IPR charge is over 14%
– In normal sense, patent charge should be within 5-10%
• MPEG-4 licensing term, charge to operator
– Against by broadcasting operators from Japan, USA, Europe, …
– IPR policy for patents in MPEG-4 is bad, almost kill MPEG-4
• IPR policy for patents in MPEG-4 AVC/H.264 is still with
question, maybe have problems later on
2011-11-10 http://www.avs.org.cn 21
AVC/H.264 licensing structure
Participation Fees
Where End User pays
Subscription
Title-by-Title
Codec
Manufacturers
Internet
Broadcast
Participation Fees where
Free Remuneration is from
Television Other sources
Source:AVC/H.264 license terms
2011-11-10 http://www.avs.org.cn 22
Case estimation for operator
• C1 (Beijing Cable TV, a pay TV operator)
– They have 3M users, if every user watch 2 title each day
– Then they might need to collect from end-users
• 3M * (0.02 * 2 * 365)/year = 43.8M USD/year
• Paying in Cap : 4.25M(in 2009); 5.00M(in 2010)
• C2(China mobile, a mobile operator with 31 subsidiary company)
– If each subsidiary has over 100K VOD/TimeShifting service subscribers
– Then they may need to pay amount of cap
• 4.25M*31= 131.75M USD/yr, in 2009
• 5M*31=155M USD/yr, after 2009
• C3(CMMB operator, a mobile TC operator in China, with 150
subsidiary company now, maybe over 600 after 2011)
– Paying in Cap
• 4.25M*150= 627.5M USD/yr, in 2009
• 5M*150=750M USD/yr, after 2009
• 5M*600=3B USD/yr, after 2011
2011-11-10 http://www.avs.org.cn 23
Solution?
• Industry looking for more selection/replacement
• RF MPEG-4 AVC/H.264 baseline
• VC-1
• AVS
• OMS
• With same/better coding performance than
H.264/AVC
• Suitable for big operator like China Telecom,
China Netcom, China Mobile, CMMB, Telefonica
2011-11-10 http://www.avs.org.cn 24
Periods of key video coding tools
contributed
Video
Variable Block-size
Object
Motion
Transform Macroblocks Planes
Compensation
Coding Multiple
DPCM
P-frames B-frames Reference
DCT Generic
B-pictures
1950 ~1989 1999
Hybrid Advanced
Scene Integer Deblocking
Coding Interlace
Adaptive Transform Filter
Huffman
Coder
Coding
Block Motion Object-
Motion Vector Based Error
Estimation Prediction Scalability Resilience
Modified from C Reader, 2002
Using for Need to pay
free
2011-11-10 http://www.avs.org.cn 25
Effort in RF or low price IPR
• FR baseline @ MPEG-4 AVC/H.264
• VC-1 @ Microsoft
• AVS @ China
• RF video codec @ Sun Microsystems
2011-11-10 http://www.avs.org.cn 26
MPEG-4 AVC/H.264 FR baseline
• The foundation of AVC/H.264 IPR policy was laid in early 2001
• Inherited by JVT when it was formed in December 2001
– JVT-C110 [Sullivan, Microsoft] JVT IPR Status Report
– JVT-C085 [Kogure+, Matsushita] IPR WG establishment for JVT
– JVT-C123 [van der Meer, Philips] JVT Licensing issue
– JVT-C124 [van der Meer, Philips] Information on JVT patents
– JVT-C149 [Yagasaki, Sony] Licensing issues on JVT
– JVT-C150 [Lindbergh, Polycom] Support for JVT Royalty Free Baseline
• By May 2002, FR Baseline supported by companies including
– Apple Computer, Inc. (USA), British Telecommunications plc (UK), Broadcom Corp. (USA), Cisco Systems,
Inc. (USA), Conexant Systems, Inc. (USA), Deutsche Telekom AG (Germany), FastVDO LLC (USA), Nokia
Corp. (Finland), Polycom, Inc. (USA), RADVISION, Inc. (Israel), SANDVIDEO (USA), Siemens AG
(Germany), Sun Microsystems, Inc. (USA), Tandberg (Norway), Telenor (Norway), Teles AG (Germany),
Texas Instruments, Inc. (USA), UBVideo (Canada), VCON (Israel)
VideoLocus Inc. (Canada), ViXS Systems Inc. (Canada), VWeb Corp. (USA)
• Activity slow down in 2003
– some key contributors to H.264 that supported the RF baseline had donated their IP to the standard, so it could
not be used defensively against the IP holders that opposed the RF baseline.
– The promise to license RF that accompanied every contribution had a key weakness – the promise was
conditional on all others also licensing RF
– MPEGLA licensed H.264 with a single license for all profiles, not distinguishing the baseline in any way. The
fee however was far lower than the MPEG2 fee
2011-11-10 http://www.avs.org.cn 27
VC-1
• In 2001, Microsoft create VC-9
• Microsoft promise in low IPR charge
• VC-9 approved and recommended by
STMPT, becoming a US standard
• After VC-9 open to public, Microsoft faced
a hard time
• In the end, Microsoft agreed to license their
patents via same channel as AVC/H.264
2011-11-10 http://www.avs.org.cn 28
AVS
• Starting in 2002
• Promise in very low IPR price
2011-11-10 http://www.avs.org.cn 29
Making a good AVS for industry
• We need
– Good IPR policy
– Good coding technologies
– Good industry promotion
• Three horses cart
– AVS WG
– AVS PPC
– AVSA
2011-11-10 http://www.avs.org.cn 30
30
AVS Working Group
• Set up by MII (Ministry of Information
Industry) in June of 2002
• In charge of national standardization of
audio-video coding for Chinese A/V
industry
• Also in charge of organizing the delegation
of China National Body to MPEG(ISO/IEC
JTC1/SC29/WG11)
2011-11-10 http://www.avs.org.cn 31
AVS members
• Total 192 members (by Dec. 30, 2007)
– CE, 12%；
– Com, 12%；
– Comp and Soft, 44%；
– IC, 14%；
• Local research entities, 25%
• Worldwide research entities, 30%
2011-11-10 http://www.avs.org.cn 32
AVS Multinational members
2011-11-10 http://www.avs.org.cn 33
AVS Local Members
2011-11-10 http://www.avs.org.cn 34
AVS industry Alliance
May 25, 2005, announced at People’s Great Hall
2011-11-10 http://www.avs.org.cn 35
AVS Industry Alliance (32 by 2008)
– Amoisonic – Panstar Semi
– Broadcom – SVA
– Changhong – Skyworth
– Haier – Spectrum
– Highsense Communication
– Huaju Foundation – Sunnywell
– Huawei – TCL
– Langchao – USstarcom
– NSCC – ZTE
– … – …
2011-11-10 http://www.avs.org.cn 36
AVS IPR Experts Group
• Late 2003 the President of AVS asked that a group of
experts be assembled to create a recommendation for an
IPR policy for AVS
• Create policies that:
– reflect and respect China law and culture
– reflect WTO requirements
– reflect successful global practices
– balance the rights of the inventor and needs of the implementers
– innovative and forward looking
– evolves with practice and law in China
2011-11-10 http://www.avs.org.cn 37
AVS IPR Experts Group
• Experienced members:经验丰富的成员
– Patent Attorneys专利律师
– Technical Experts技术专家
– Contract Attorneys处理合同的律师
– Previous experience in setting up patent pools具有建立专利池相关经验
的成员
• Members are from:成员来自
– Asia 亚洲
– Europe 欧洲
– North America 北美
• Members represent: 成员代表了
– Consumer Electronics 消费电子产品界
– Information Technology 信息技术界
– Microelectronics微电子产业界
2011-11-10 http://www.avs.org.cn 38
AVS IPR Experts group
• Key members
– BroadCom
– IBM
– Intel
– Matsushita/Panasonic
– Microsoft
– Nokia
– Sony
– Sun Microsystems
2011-11-10 http://www.avs.org.cn 39
Licensing Obligations
Non-Participant: Participant:
Join AVS • RAND RF • RAND RF
• Sign Member Agreement • POOL • POOL
• RAND • RAND
•Declare default option • No License
Participate in Subgroup • RAND RF
• POOL
• Commit to license • RAND
• License options
Submit Contribution China Patent: Worldwide Patent:
• RAND RF • RAND RF
• Disclose patents • POOL • POOL
•Declare license option • RAND
2011-11-10 http://www.avs.org.cn 41
Disclosure Obligations
• When making a contribution
– Unpublished, published and granted patents
• Ongoing
– When a patent is discovered
– When a Patent Application is published
• During Final Draft Standard Review
– 90 days to declare licensing option, OR:
– Default applies
2011-11-10 http://www.avs.org.cn 42
AVS Licensing
• 1 RMB concept
– Only one charge, to device/software/chip
– No charge to operator
– No charge to end-user
• Will be handled by AVS PPA
– PPA is a non-profit organization
– Founded By #1 institute of MII and ICT of
CAS, in 2005
2011-11-10 http://www.avs.org.cn 43
AVS Standardization Roadmap
Part2: Video
HDTV/SDTV
broadcasting Part1: system
and Storage
applications
Part3: Audio
Part4: DRM
Mobile Part7: Mobile Video
Applications Part7: Mobile Video X profile
2011-11-10 http://www.avs.org.cn 44
AVS Video Technical Aspect
• High efficiency
Similar with H.264/MPEG-4 AVC
2 to 3 times higher than MPEG-2
• Lower complexity
– 2 reference frames
– 8*8 block but not 4*4
– No useless tools for SD and HD video coding
– 70% for decoder as that of H.264
– 30% for encoder
• Compatible with MPEG-2 at system level
2011-11-10 http://www.avs.org.cn 45
AVS1-P2 diagram and special
techniques
Input Coder
Video Control
Control
Signal
Data
Transform/
coeffs
Quant
-
Split into Inv Quant/
Macroblocks Inv Transform
16x16 pixels Entropy
Coding
De-blocking
Intra-frame Filter
Prediction
Output
Motion-
Video
Compensation
Intra/Inter Signal
Motion
Data
Motion
2011-11-10 Estimation
http://www.avs.org.cn 46
46
Performance- HD (1280x720)
Flamingo
Harbour
41
40
39
38
PSNR(dB)
37
36
PSNR(dB)
35
34
33 H.264 Main
32
31 H.264 Baseline
AVS
30
29 H.264 MainMPEG-4
MPEG-2
28 1000 AVS
6000 11000 16000 21000 26000 31000 36000
26 Bitrate(kbits/s) MPEG2
10000 40000 70000 100000 130000
Bitrate(kbits/s)
2011-11-10 http://www.avs.org.cn 47
Performance – SD (720x576)
Flowerguarden
Basketball
38
38
35
35
PSNRY(dB)
PSNRY(dB)
32
32
29
29
H.264 Main
26
26 H.264 Main
AVS
23 AVS
23 MPEG-2
4000 9000 14000 19000 24000 MPEG-2
29000 34000
1900 4900 7900 10900 13900
Bitrate(kbits/s)
Bitrate(kbits/s)
2011-11-10 http://www.avs.org.cn 48
Performance – CIF (352x288)
Mobile
News
40
38
37
36
PSNR(dB)
PSNRY(dB)
34
34
31 32
28 30 H.264 BaseLine
H.264 BaseLine
H.264 Main
H.264 Main
MPEG-4
25 28 MPEG-4
AVS
AVS
0 55 105
1000 155
2000 205 3000 255 4000305 5000
355 6000
Bitrate(kbits/s)
Bitrate(kbits/s)
2011-11-10 http://www.avs.org.cn 49
AVS1-P2 Subjective Testing
• April-Sept., 2005, AVS subjective testing has been
executed by SARFT testing center
• The testing is based on GY/T 134-1998 standard as
well as related ITU testing standard
Compressed testing video SD resolution HD resolution
stream bit-rate (625/50i) (1125/50i)
MPEG-2 typical bit-rate 4~6 20
(Mbps)
AVS bit-rate (Mbps) 3 1.5 10 6
2011-11-10 http://www.avs.org.cn 50
Testing result A B A A-
AVS video standard
• In Feb. 2006, AVS video
issued as national
standard
– GB/T 20090.2-2006
– Effective from March
2006
2011-11-10 http://www.avs.org.cn 51
51
AVS Encoder vendor
• NSCC
– AE100S，实时标清编码器
– AE200S，专业级的实时标清编码器
• SVA
– Codestar AC1001标清转码器
– Codestar AE1001标清编码器
• Envivio
– 4caster C4 trascode，四路输入，四路输出
– 4caster B3 Encoder
2011-11-10 http://www.avs.org.cn 52
AVS Chip
• Shanghai Spectrum Communication (展讯)
– SV 6100, HD, AVS
– SV 6111, HD, AVS/MPEG-2
• Shanghai PenStar Semi.(龙晶)
– DS1000, HD, AVS-core
• Broadcom (博通)
– BCM 7405B, HD, AVS/MPEG-2/H.264/VC-1
• Sigma Design
– SGM 8654, HD, AVS/MPEG-2/H.264/VC-1
• STMicro
– 710x, HD
• NXP
– STB 222, HD
• Hangzhou Guoxin（杭州国芯）
2011-11-10 http://www.avs.org.cn 53
AVS adopted by operator in China
• In service
– China Netcom
• Dalian city(57 channels, soon up to 100)
• Other cities
– Hangzhou Terrestrial DTV (21 channels)
– Shanghai Terrestrial DTV (25 channels)
• On going
– Baoding Terrestrial DTV
– Taiyuan Terrestrial DTV
– Sichuan703 Terrestrial DTV
– …
2011-11-10 http://www.avs.org.cn 54
OMS Video
• In Aug. 2005, Sun Microsystems Inc. announced
Open Media Stack (OMS),
• OMS Video Specification has been developed
based on the principles of the Open Media
Commons (OMC)
(http://www.openmediacommons.org)
• Using non-patent/expired patent technology
• Performance close to MPEG-2 video part
• On-going
2011-11-10 http://www.avs.org.cn 55
Influence factor in future standards
• Technology
– Removing redundancy
• IPR
– Maybe three layer/profile? HVC
• RF baseline LP Main
• Low IPR main
RF baseline
• High
2011-11-10 http://www.avs.org.cn 56
HVC/AVS2.0 structure
• New coding framework
– Texture analysis/synthesis
– Super-resolution based video coding
– Learning based video coding
– Visual based video coding
– Traditional coding tools improvement
Super-macroblock prediction
Adaptive Block-size Transform (ABT)
Directional transform
Advanced motion vector prediction
Rate Distortion Optimization Quantization (RDOQ )
2011-11-10 http://www.avs.org.cn 57
Texture Analysis/Synthesis
Texture masking effect
2011-11-10 http://www.avs.org.cn 58
Image Compression with
Inpainting
2011-11-10 http://www.avs.org.cn 59
Super-resolution Based Video Coding
Reconstruct high resolution image from multiple sequential low
resolution images
High frequency modeling
Spatial-temporal interpolation
High Frequency Low Frequency
Spatial-temporal Interpolation
2011-11-10 http://www.avs.org.cn 60
Learning based Video Coding
• Model based video coding
• Construct the coding model in the learning process
• Model parameters give a description for edge,
texture, motion….
Encoder Decoder
Learning
Learning
Model Model
2011-11-10 http://www.avs.org.cn 61
Super Macroblock Prediction
• Macroblock size is defined as N×N (N>16, e.g. 32×32 or
64×64)
• Super macroblock partitions
– N×N，N×hN， hN×N，pQuar
• RDO mode decision
– Select the mode whose RDcost is the smallest
N ×N N×hN hN×N pQuar
2011-11-10 http://www.avs.org.cn 62
Motivation of ABT
• Bigger transforms allow to better capture
the usually smoother content in high
definition video
• Smaller transforms is more helpful for
regions with high texture information
2011-11-10 http://www.avs.org.cn 63
Proposed Scheme for HVC
• More flexible ABT
– transform_size_flag = 0
• 4×4: all kinds of motion partition
– transform_size_flag = 1
• 8×8: motion partition bigger than or equal to 8×8
– transform_size_flag = 2
• 16×8: motion partition equals to 16×8
• 8×16: motion partition equals to 8×16
• 16×16: motion partition equals to N×hN/hN×N (N>16), N×N (N>=16)
transform_size_flag Transform size Motion partition size
0 4×4 4×4 and above 4×4
1 8×8 8×8 and above 8×8
16×8 16×8
2 8×16 8×16
16×16 N×hN, hN×N (N>16)
N×N (N>=16)
2011-11-10 http://www.avs.org.cn 64
Directional Transform and
Adaptive Coefficient Scanning
• Adaptive coefficient scanning is used to capture the
distinct coefficient statistics between different
prediction modes.
H.264 Improved
statistics scanning order
80 66 47 27
46 35 23 8
28 21 13 4
14 11 5 2
2011-11-10 http://www.avs.org.cn 65
Uniform Reconstruction Quantizer
• Uniform-reconstruction quantizer
– Input : ci , where i is the coefficient index
– Output : C[x]
– Quantization step : QP
– Multiplication factor matrix : Q
C[ x] sign(ci ) | ci | Q QP%6, i / 215QP / 6
– Disadvantage
• No RD optimization is performed for each DCT block
2011-11-10 http://www.avs.org.cn 66
Rate Distortion Optimized Quantization
• For a given coefficient position k, assume that coefficient
ck is the last significant coefficient in the block
– For each coefficient, i=k-1,…,0, calculate its Lagrangian cost
when the quantized value li is equal to 0, lceil and lfloor:
J k ,i , l i err c i , l i bitsl i
with l floor floor (| ci | QQP%6, i / 215QP / 6 ) lceil lfloor 1.
lk-1=0 Jk, k-1(λ, 0)
min Jk,, k-1
lk-1=lceil Jk, k-1(λ, lceil) Jk, k-1(λ, lk-1)
lk-1=lfloor Jk, k-1(λ, lfloor)
c0 c1 ck-1 ck
– Let the final quantized level and update
cost Jk(λ) using Jk, i(λ, li, opt) l i ,opt arg min Jk ,i , l i
li
2011-11-10 http://www.avs.org.cn 67
Competition based MV Coding with
Optimal Predictor
• Non-competition based MV coding
– MV predictor is chosen by certain fixed rule, such as median
prediction for H.264
• Competition based MV coding
– MV predictors are chosen from a candidate set
– Selected by RD-criterion that considers the cost of the residual and
the index for the prediction
2011-11-10 http://www.avs.org.cn 68
Decoder Side MV Derivation using
Template Matching
• Template matching (TM) is performed at the
decoder side to derive motion vectors from the
already decoded signal
• The size of target and template should designed
carefully according to the block size and mode
2011-11-10 http://www.avs.org.cn 69
JND based Video Coding
• HVS property
– We can not sense any change in the image that below
the JND (Just Noticed Distortion)
– Two components in spatial domain
– Luminance masking
– Texture masking
• Residual pre-filtering based on JND
– For each pixel，compute its JND value
– Only encode the pixels, whose residual is larger than
JND
– For the pixels whose residual is smaller than JND, set it
to be 0.
2011-11-10 http://www.avs.org.cn 70
New/Improved coding techniques
Evolutionary efforts (ITU/MPEG)
Work within ITU and MPEG
NGVC: Next Generation Video Coding
(An improvement of H.264 High Profile)
H.261
MPEG-1 Video
HVC: High performance Video Coder
(for high-quality applications)
H.262/MPEG-2 Video
H.265: Not yet formalized
(Focus on Computational efficiency) H.263
MPEG-4 Visual
H.264/AVC/SVC/MVC RVC
NGVC ? HVC ?
Source: F. Pereira, IST, Lisbon
Source: Robert Forchheimer, LiU
2011-11-10 http://www.avs.org.cn 71
PCS 2009, May 6-8, Chicago
Working in plan
• MEPG Xi’an meeting, Oct. 2009
• JPEG meeting will be there
• ITU SG16 meeting may be there too
• “Xian is going to be a watershed in ICT
standardisation”
– personal communication with Leonardo
Chiariglione
2011-11-10 http://www.avs.org.cn 72
Summary
• Video standards influence by not only
technical, but also IPR policy
• Efforts on balancing two aspects is endless
• Considering all patents will be expired in
2014, the way of standards might change by
this factor
• Next generation of video coding standard
will be coming out within 5 years
2011-11-10 http://www.avs.org.cn 73
Thanks