Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Derek Pang, Xiaoyu Xiu, Jie Liang
IEEE ICME 2009 New York, USA June 30th 2009
School of Engineering Science Simon Fraser University
Introduction
Proposed MVC Scheme
Result
Conclusion
Outline
Constraints in existing MVC prediction techniques Our new solution to MVC prediction Experimental result of our solution Future research and concluding remark
Slide 2
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
INTRODUCTION
School of Engineering Science Simon Fraser University
Introduction
Proposed MVC Scheme
Result
Conclusion
Multiview Video Coding (MVC)
Current MVC Standard
MVC Standard – Multiview extension to H.264/AVC (Annex H) [JVT, 2008]
Taken from Coding Algorithm for 3DTV- A Survey [Smolic et al, 2007]
Slide 4
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Inter-view Prediction
View Synthesis Prediction (VSP)
Traditional motion compensation is ineffective for inter-view prediction. View Synthesis Prediction (VPS) is proposed to utilize the geometric relationship between camera views and generate a virtual reference view for prediction.
Two common techniques: Disparity-based view interpolation Depth-based 3-D warping
Slide 5
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
View Synthesis Prediction (VSP)
Our Motivation
View interpolation [Yamamoto et al,2007; Lee et al, 2007] can only be applied to about half of the views does not consider complex camera settings
3-D warping [Martinian et al, 2006; Ekmekcioglu et al, 2007] requires additional overhead for estimating and coding depth information Our objective: To overcome above constraints by introducing a new VSP-based MVC scheme
Slide 6
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
PROPOSED MVC SCHEME
School of Engineering Science Simon Fraser University
Introduction
Proposed MVC Scheme
Result
Conclusion
Overview
The proposed VSP method
Based on the rectification-based view interpolation method for MVC in [Xiu, PCS’09+. Five Major Steps:
View #1
new
View #2
Projective rectification
Disparity estimation
View extrapolation
new
Virtual Reference (View #3)
Synthesis Bias Correction
Unrectification
Slide 8
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Overview
The proposed VSP-based MVC Scheme
T0 S1 T1
T2
T3
T4
T5
T6
T7
T8
I0
B3
B2
B3
B0
B3
B2
B3
I0
S2
P0
Vp
B3
Vb
B2
Vb
B3
Vb
B0
Vb
B3
Vb
B2
Vb
B3
Vb
P0
Vp
V3
S3 V4
P0
Vp
B3
Vb
B2
Vb
B3
Vb
B0
Vb
B3
Vb
B2
Vb
B3
Vb
P0
Vp
S3
P0 . . .
B3 . . .
B2 . . .
B3 . . .
B0 . . .
B3 . . .
B2 . . .
B3 . . .
P0
. . .
Slide 9
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #1 and #2
Projective Rectification & Disparity Estimation
Step #1 Projective Rectification Step #2 Disparity Estimation
To rectify two adjacent camera views such that their epipolar lines are horizontal using Xiu’s method [PCS, 2009].
To find the disparity between each pair of point correspondence on the two camera views.
Slide 10
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #3
View Extrapolation
To utilize the disparity between two adjacent camera views for synthesizing a virtual reference view.
Allows VSP to be applicable to all camera views beside the first two. Does not require explicit coding of depth map Needs to consider four cases: 1. A pixel is visible in all views. 2. A pixel is only visible in the right view 3. A pixel is only visible in the left view 4. Occluded pixel
Slide 11
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #3
View Extrapolation
Case 1 A pixel is visible in all views.
α
Rectified virtual cameras
cα cd d
Left View Right View
Pixel to be extrapolated
=
Extrap. View
1 2
1 2
+
Slide 12
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #3
View Extrapolation
Case 2 A pixel is visible in right view. Assume pixel is visible in the extrapolated view.
Extrapolated view
Right View
Left View
Case 3 A pixel is visible in left view. Assume pixel is invisible in the extrapolated view.
Slide 13
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #3
View Extrapolation
Case 4 Occluded Pixel
Previous frame #1
Previous frame #2
Extrapolated View with occluded regions
Slide 14
Final Extrapolated View
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #4
Un-rectification
To project the virtual reference back to the original format.
The un-rectification transformation can be found by the direct linear transform algorithm.
[Xiu, 2009; Hartley et al, 2003]
Slide 15
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #5
Synthesis Bias Correction
To reduce the biases originated from the view synthesis and the illumination variations between cameras.
A simple correction technique that estimate bias based on the past errors made in the previously coded frames. Two Types of Biases: 1. Disparity estimation bias 2. Pixel synthesis bias
Slide 16
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Synthesis Bias Correction
Disparity Estimation Bias
Previous frame #1
Reconstructed frame Virtual reference frame Corrected by disparity bias
Previous frame #2
Current Frame
Virtual reference frame
Disparity bias
Slide 17
+
=
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Synthesis Bias Correction
Pixel Synthesis Bias
Previous frame #1
Previous frame #2
Current Frame
Slide 18
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
RD PERFORMANCE RESULT
School of Engineering Science Simon Fraser University
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
Test Setup
Used the eight standard MV test sequences
Benchmark: 1. JMVC 3.0 [JVT, Nov 2008]
Slide 20
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
Average Coding Gains over JMVC
1.4
1.2 1
Proposed Method
dPSNR (dB)
0.8
0.6 0.4 0.2 0
Test Sequeces
Slide 21
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
Average Coding Gains over JMVC (Rena)
1.4
Proposed Method
1.2 1
dPSNR (dB)
0.8
0.6 0.4 0.2
0
3
Slide 22
5
7
9 Number of Views
11
13
15
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
CONCLUSION
School of Engineering Science Simon Fraser University
Introduction
Proposed MVC Scheme
Result
Conclusion
Conclusion
Introduced 3 unique techniques to VSP: 1. projective rectification 2. view extrapolation 3. synthesis bias correction Achieved significant coding gains (up to 1.6 dB) for video sequences with large number of views
Future researches 1. Complexity reduction 2. Exploitation of parallel computing 3. Improvement to synthesis techniques used
Slide 24
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Q&A
Thank You.
Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction IEEE ICME 2009 Derek Pang
dcpang@sfu.ca http://www.sfu.ca/~dcpang/ School of Engineering Science Simon Fraser University
ADDITIONAL SLIDES
School of Engineering Science Simon Fraser University
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method
Synthesized Reference
Our experiment shows that our SBC can offer an average of 2.74 dB improvement in terms of synthesis quality
Original
Synthesized Virtual Reference (34.14 dB)
Slide 27
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
View Synthesis Quality
Average: 31.94 dB with SBC vs 29.20dB without SBC
40 With Synthesis Bias Correction 35 Without Synthesis Bias Correction
30
PSNR-Y (dB) 25 20
15
10 5 0 0 5 10 15 20 25 30 35 40 45 50 55 Frame Index
Slide 28
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
RD Performance Comparison (Rena)
40 39 38
PSNR (dB)
0.5 dB
37 36 35 34
1.6 dB
Proposed Method JMVC 3.0 Modified AS_IPP 64 128 192 256
33
Avg. Rate [kpbs]
Slide 29
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
Test Setup
Benchmark: 1. Modified AS_IPP structure
from Efficient prediction structures for MVC [Merkle et al, 2007]
Slide 30
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
Average Coding Gains over JMVC
1.4 1.2 1 0.8
Modified AS_IPP
Proposed Method
dPSNR (dB)
0.6 0.4 0.2 0 -0.2 -0.4
Test Sequeces
Slide 31
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
Experimental Result
Average Coding Gains over JMVC (Rena)
1.4
Modified AS_IPP
1.2 1
Proposed Method
dPSNR (dB)
0.8
0.6 0.4 0.2
0
3
Slide 32
5
7
Number9 Views of
11
13
15
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Problem
Proposed MVC Scheme
Result
Conclusion
View Synthesis Prediction (VSP)
View Interpolation
Right View
Virtual interpolated view
Related Work: MVC using view interpolation and color correction [Yamamoto et al, 2007] An efficient view interpolation scheme and coding method for MVC [Lee et al, 2007]
Left View
Slide 33
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
View Synthesis Prediction (VSP)
3-D Warping
3-D Warping
Left View
Right View
Virtual Middle View
Slide 34
Related Work: Extensions of H.264/AVC for MVC [Martinian et al, 2006] MVC via virtual view generation [Ekmekcioglu et al, 2007]
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #1
Projective Rectification
To rectify two adjacent camera views such that their epipolar lines are horizontal.
1. Estimate the fundamental matrices of both views 2. Determine rectification transformation of the right view using Hartley’s method. [Intl. Jr. Of Comp. Vision, 1999] 3. Find rectification transformation of the left view using Xiu’s method. [PCS, 2009] 4. Apply rectification to both views. 5. Up-sampled the rectified images.
Slide 35
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #2
Disparity Estimation
To find the disparity between each pair of point correspondence on the two camera views.
Used a graph-cut based disparity estimation technique proposed by Kolmogorov and Zabih [Intl. Conf. Of Comp. Vision, 2001]
Slide 36
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
Introduction
Proposed MVC Scheme
Result
Conclusion
The New VSP Method Step #1
Projective Rectification
Projective rectification reduced disparity estimation into a 1-D problem.
Epipolar line
Rectified Left View
Slide 37
Rectified Right View
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
References
[1] P.Merkle, A. Smolicand, K.Muller, and T.Wiegand, “Efficient prediction structures for multiview video coding,” IEEE Trans. CSVT, vol. 17, no. 11, pp. 1461–1473, Nov. 2007 [2] K.Yamamoto, M. Kitahar, H. Kimata, T.Yendo, T. Fuji, M. Tanimoto, S. Shimizu, K. Kamikura, and Y. Yashima, “Multiview video coding using view interpolation and color correction,” IEEE Trans. Circ. Syst. for Video Tech., vol. 17, no. 11, pp. 1436–1449, Nov. 2007. [3] C. Lee, K.-J. Oh, S.-H. Kim, and Y.-S Ho, “An efficient view interpolation scheme and coding method for multi-view video coding,” in Proc. International Conference on Systems, Signals and Image Processing (IWSSIP), Jun. 2007, pp. 27–30.
Slide 38
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
References
[4] E. Martinian, A. Behrens, J. Xin, and A. Vetro, “Extensions of H.264/AVC for multiview video compression,” in Proc. ICIP, Oct. 2006, pp. 2981–2984. [5] E. Ekmekcioglu and A. M. Kondoz S. Worrall, “Multi-view video coding via virtual view generation,” in Proc. Picture Coding Symp. (PCS), Nov. 2007. [6] X. Xiu and J. Liang, “Projective rectification-based view interpolation for multiview video coding and free viewpoint generation,” Proc. Picture Coding Symp. (PCS), May. 2009.
Slide 39
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
References
[7] A. Smolic, K. Mueller, N. Stefanoski, J. Ostermann, A. Gotchev, G.B. Akar, G. Triantafyllidis, and A. Koz, “Coding algorithms for 3DTVA survey," Circuits and Systems for Video Technology, IEEE Transactions on, vol. 17, no. 11, pp.1606-1621, Nov. 2007. *8+ “Joint multiview coding (JMVC) 3.0,” garcon.ient.rwthaachen.de, Nov. 2008. *9+ R. Hartley, “Theory and practice of projective rectification,”Int. J. of Comp. Vision, , no. 2, pp. 115–127, 1999. [11] V. Kolmogorov and R. Zabih, “Computing visual correspondence with occlusions using graph cuts,” International Conference on Computer Vision, vol. 2, pp. 508–515, 2001.
Slide 40
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction
References
[12] R. Hartley and A. Zisserman, “Multiple view geometry in computer vision,” Cambridge Univ. Press, 2003.
Slide 41
D. Pang: Multiview Video Coding Using Projective Rectification-based View Extrapolation and Synthesis Bias Correction