The Future of Interactive Television
(August 30, 2000)
Computer Science Division - EECS University of California at Berkeley http://bmrc.berkeley.edu/~larry
Lawrence A. Rowe
Outline
m m m m m
Conventional Broadcast Television Interactive Television Internet Webcasting Production Systems Research Issues
Future TV (9/00)
2
1
Conventional Television Broadcasting System
Transmitter
Transmission: wireless, satellite, or cable
Future TV (9/00)
Receiver
3
Conventional Television
Produce the best signal possible given the NTSC signal and transmission standards
m
Single audio/video stream
May composite several streams into one
m m m
Fixed image size (720x485) Constrained image quality No interactivity
Future TV (9/00)
4
2
Television Production
tv studio tv studio edit bay edit bay edit bay ad insertion master control
Routing Switcher
transmission
Future TV (9/00)
5
Television Studio
studio audio production
audio tape
audio mixer/control line output master clock
TBC
live rem ote
production switcher VTR
video production
video audio clock
Future TV (9/00)
6
3
Television Distribution
cable headend
TV Station
m m
Limited number of channels Maximum audience
7
Future TV (9/00)
Trends
m
Majority of homes have cable
70%+ homes in US have cable 25% homes have neither cable nor satellite
m
Adding channels makes money
Typical cable systems have 50-200 channels Digital cable subdivides 6 Mhz channel into multiple subchannels (a.k.a. “multicasting”) New channels and pay-per-view offerings
Future TV (9/00)
8
4
Trends (cont.)
m
Video-on-Demand (VOD) not economical
Users love it, but too costly Simulate with scheduled play over several channels
m
HDTV is wonderful
1
Still 2-3 years before reasonable-priced receivers HDTV wireless broadcast does not make sense!
m
Current products being pushed by industry
Personal TV Receivers (PTRS) like Tivo and ReplayTV Home theathre: DVD and 5.1 surround sound audio
1
“Defining Vision: The Battle for the Future of Television” by Joel Brinkley is a wonderful story about the development of the HDTV standard
9
Future TV (9/00)
Looking for New Business
m
Interactive television (ITV)
WebTV (ATVEF) – push remote to communicate with settop box (e.g., play games, browse web pages, display information, etc.) Many other unsuccessful approaches to ITV
m
Telephone service
Buy phone service from your local cable company – they must be crazy!
m
But, what about the Internet?
Future TV (9/00)
10
5
ATVEF Examples
(Images thanks to RespondTV)
Order products
Enter personal info
Other examples: ask for additional information, watch different camera views, play games, etc.
Future TV (9/00) 11
Internet Television Distribution
TV Station
Video Gateway Computer
Content Distribution Network (CDN)
m
CDN is 1-n virtual subnetwork
èjust another distribution channel for conventional TV! èinteraction using ATVEF style interaction
Future TV (9/00)
12
6
Why this Solution?
m
Efficient resource utilization
Broadband connection to the home can carry content the user is watching rather than mass market content controlled by media conglomerates Allocate bandwidth based on viewers or webcasters willingness to pay
m
Proliferation of program content
Trend from few network channels to everyone is a broadcaster (analogy to the web) People want to communicate - Internet provides an outlet not available with conventional distribution technology
Future TV (9/00)
13
Consider Resource Usage
Assume 100 Channel Cable System
m
m
27 Mbs data bandwidth per channel implies 2.7 Gbs shared bandwidth for a neighborhood 6 Mbs for conventional channel implies 100 channels plus 2.1 Gbs for other applications How much would you pay for a 100 Mbs shared ethernet connection from home?
Of course some programs might use less, more, or vary dynamically
m
Future TV (9/00)
14
7
What Problems?
m
Economic: ubiquitous broadband IP communications
Fiber to the home!
m
Technical: fix multicast protocols
IETF developing source-specific multicast (SSM)
m
Social: overcome powerful media companies and political impediments to reallocating bandwidth
Why consume wireless bandwidth for <25% of the homes? Put all HDTV broadcasts on cable/satellite
Future TV (9/00)
15
Source-Specific Multicast
m
Service models
IP Multicast: n-way unreliable communication SSM: 1-n unreliable communication (e.g., ATM multipoint circuit with dynamic join/leave)
m
SSM works perfectly for Internet distribution of conventional TV
Many companies developing/deploying technology to support this new distribution channel
Future TV (9/00)
16
8
Isn’t the Future Wonderful?
Well, maybe…
m
Need more innovative ITV programs
More TV channels like we currently watch will not work economically – content matters User-controlled content…
m
Still need n-way communication
Small group interaction (up to 10-20?) Large virtual spaces – few people per room Moderated discussions (e.g., townhall meetings and talk radio)
Future TV (9/00)
17
Internet Webcasting
Interesting experiences given unlimited bandwidth
m m m m
Multiple video streams Varying picture size and quality Interactivity between subsets of viewers Improve quality by using intelligent systems
Examples:
Lecture/seminar webcasts - speaker and content streams Live events where all video sources are available Collaborative experiences (“places all over the world,” etc.)
Future TV (9/00)
18
9
Webcasting Architecture
colab studio
Multicast
m m m m
N-way communication Multiple streams and discussion threads User control of content (e.g., select streams) Variable bit rates streams
19
Future TV (9/00)
Examples
m
Edutainment
Distance/asynchronous learning Games
m
Live events (e.g., sports, concerts, etc.)
User should control what he/she views – conventional production transmission plus choice of other camera views
m
And programs we can’t even imagine!
Note: most can be implemented by applications using SSM
Future TV (9/00)
20
10
Berkeley Multimedia, Interfaces, and Graphics (MIG) Seminar
m
Regularly scheduled seminar webcast worldwide on the Internet since Jan 1995
Early days averaged 30-40 remote viewers with high of 200 Network problems reduced viewers to zero
m
Program and webcast production
Two streams – speaker and content Simulcast different transmissions (varying bit rate and technology) Remote questions and speakers
Future TV (9/00)
21
Seminar Announcements
Future TV (9/00)
22
11
Participants View
Future TV (9/00)
23
Studio Classroom
mic wireless control Video Gateway PC Elmo audience mic speaker camera video projector
computer
Video Gateway
computer
matrix switcher
VCR
video audio control
Future TV (9/00) control switcher
audience camera
24
12
A MIG Seminar Webcast
Wireless Mic Audience Mic Audience Mic
FX Processors
Mars Server
FX Server
Future TV (9/00)
25
Webcast Production
m
Streaming media tools
Standard MBone tools (e.g., vic, vat, etc.) Real Networks Producer Mbone2Real transcoder
m m
m m
Broadcast Manager for launching webcasts Question Board for floor control and remote questions Video Effects System to improve visual quality Director’s Console for live production control
Future TV (9/00)
26
13
iWebcast Architecture
Low Bitrate rtpgw
Medium Bitrate
Future TV (9/00)
27
Broadcast Problems
m
Using tools is easy, but configuring is not
Learning curve too high Tedious to experiment with new configurations
m
Too many tools
Seminar requires over ten applications started on different hosts
m
Configuration problems
Example: low bit-rate video archive process and low bit-rate video transcoder must use the same multicast address
m
Time sensitive pressures
Live broadcasts must be done on time Time to restart crashed tools means loss of air time
Future TV (9/00)
28
14
Broadcast Problems
Future TV (9/00)
29
Broadcast Manager
m
Provide user interface for configuring tools Store configuration in database for reuse Provide automation
One click to start a broadcast stored in database Send SAP announcements out to advertise program Start broadcast automatically on time Reduces learning curve Encourages broadcast configuration experimentation
m
m
Future TV (9/00)
30
15
Launch Broadcast
Future TV (9/00)
31
Experience
m
Reduced complexity and learning curve
More services added with little impact on manageability Undergraduates quickly learned to produce broadcasts
m
Setup time shortened Restart time shortened
Previously required 30 to 45 minutes preparation, now only 5-10 Recover from hardware and software crashes quicker
m
m
More simultaneous broadcasts
But only one operator required
Future TV (9/00)
32
16
Webcast Production
m m m
m
m
Streaming media tools Broadcast Manager for launching webcasts Question Board for floor control and remote questions Video Effects System to improve visual quality Director’s Console for live production control
33
Future TV (9/00)
Question Board (qb)
m
Lecture style floor control tool
Participants indicate they have a question and desired media (e.g., text, audio-only, or audio and video) Lecturer or moderator controls floor
m
Enhanced services
All participants see questions (optional) Private questions only seen by speaker Anonymous questions
Future TV (9/00)
34
17
qb User Interface
Future TV (9/00)
35
Moderator User Interface
• Participants request follow-up questions • Moderator switches rapidly between speakers
Future TV (9/00)
36
18
Discussion
m
Questions distributed using multicast Floor control separates audio and video grant messages Have used in multimedia seminar broadcast
Control messages sent to participants using control bus – another multicast session between all participants Received audio and text questions Need moderator in room because most speakers are unfamiliar with tool Reliable/unicast to moderator who multicasts to others Use SRM/NACK protocol for reliable multicast
m
m
Future TV (9/00)
37
Webcast Production
m m m
m
m
Streaming media tools Broadcast Manager for launching webcasts Question Board for floor control and remote questions Video Effects System to improve visual quality Director’s Console for live production control
38
Future TV (9/00)
19
Conventional Production Switcher
m
Produces high quality visual images
N input streams and 1 output stream
m
Special effects
Titles, transitions, picture-in-picture, compositions, chromakey, …
m
Implementation
Conventional solutions use custom-designed hardware
Future TV (9/00)
39
Video Effects Processing
Use commodity processors - Networks of Workstations
m
Processing required
CCIR 601 frames (e.g., 720 x 480, 4:1:1 subsampling) Uncompressed 50% fade (multiply pixels by 0.5 then add together) Pentium-Pro 200 requires ~500 milliseconds
m
Must exploit
Parallel processing Special-purpose instructions (e.g., Intel MMX, etc.)
Future TV (9/00)
40
20
Parallel Processing
m
Temporal Decomposition
Distribute frames to different processors
m
Spatial Decomposition
Partition single frame image into regions and distribute to different processors
m
Functional Decomposition
Assign functions to different processors (e.g., decode, pip, titling, etc.)
Future TV (9/00)
41
System Architecture
Effects Processor Effects Processor
Effects Server
Local Network
Processed Video Output
Video Archive Director’s Console
Video Control
Live Video Sources
Future TV (9/00)
42
21
System Operation
m
Hierarchical decomposition of effects graph Difficult control issues
May require subgraphs to be optimized differently Want effect graphs to work either as application interface or as subgraph – use SNAP to control multicast message filtering Must handle several broadcasts at the same time May want to reallocate processing dynamically
m
m
Processor allocation
Future TV (9/00)
43
Discussion
m
Developed prototype parallel video fx system
Runs on SparcNow – supports titling, fades, chromakey, etc.
m
Incorporated into production control system
Service setup and application interface controls Used in MIG Seminar (once!)
m
Much research remains…
Effect parallelization optimization – similar to query optimization High-level language specification of effects, interface controls, and application services
Future TV (9/00)
44
22
Webcast Production
m m m
m
m
Streaming media tools Broadcast Manager for launching webcasts Question Board for floor control and remote questions Video Effects System to improve visual quality Director’s Console for live production control
45
Future TV (9/00)
iWebcast Architecture
Low Bitrate rtpgw
Medium Bitrate
What sources are included in webcast and how are they controlled?
Future TV (9/00) 46
23
DC Model
Sources – streams available for inclusion in program Broadcast – stream(s) in webcast Transmission – different transmissions of webcast
Future TV (9/00) 47
Director’s Console
• Lists available sources • Shows current stream(s) • Controls to manipulate stream (e.g., move camera or position stored media) • Extensible interface to support new sources and controls • Eventual goal is to automate this process
Future TV (9/00)
48
24
Systems Architecture
m
Client/server system
dc is client and servers provide interface to various services which can produce media streams or support control (e.g., turn on lights, lower screen/projector, etc.)
m
m
Service discovery protocol to select services for particular webcast Extensible architecture so new services can be added to the system
m
2800 lines of code (400 C++ and 2400 Tcl/Tk)
Services provide a UI code segment to dc for controlling the service - takes two args: socket to access service process and window in which to instantiate/display control interface
Future TV (9/00)
49
Experience
m
Fabulous tool for producing webcasts
Incorporated into MIG Seminar in fall 1999, now indispensable Other researchers and webcast producers have asked about modifying dc for use in other webcasting applications Add services and improve user interfaces Automate control decisions Develop scripting language for webcast program
m
Many directions for improvement
Future TV (9/00)
50
25
Automated Control
m
Speaker switches source Audience question sensed on mic’s Script for webcast
implies content stream should be switched implies change content to audience cam modify audio levels openning, speaker+content, speaker titling at f(curtime), periodic switch from wide shot to speaker closeup switch content and local projection to remote speaker
m
m
m
Speaker/moderator recognizes remote question Speaker talks to room/webcast control system etc.
“Please show the whiteboard….”
m
m
Future TV (9/00)
51
Future Plans
m
Improved production
Want all audio/video sources on network Higher bandwidth streams Control automation
m
Improved studio classroom
Improved “sense of presence” for remote participants Improved instructor interface – ad hoc content display More displays in room – multiple slides projected Improved audio and presenter control
m
Wider deployment
Future TV (9/00)
52
26
Berkeley Internet Broadcasting System (BIBS)
m
Webcast class lectures
Live webcasts for remote viewing On-demand replay for review and watching missed lectures
m
Fall 1998 and spring 1999 small experiments
Approximately 5 classes with one hundred students
m
Scaled up in fall 1999 – continues today
Webcasting 10 classes with enrollment of 3,000 Using Real Networks SureStream (50/200 Kbs)
m
Adding functionality to Real Webcasts
2-stream programs (speaker + content) Lecture browser (video + synchronized slides + …)
Future TV (9/00)
53
BIBS Program Guide
Future TV (9/00)
54
27
Sample Lectures
Future TV (9/00)
55
Campus Infrastructure
Future TV (9/00)
56
28
Research Problems
m
Automation and control of webcasts Compilation and planning of video effects
Given high-level effect description and machine resource description, produce plan to execute effect within required performance constraints Dynamic optimization of effects processing across many users Sense of presence for remote participants Improved interface for ad hoc presentations (e.g., tablet, display control, etc.) Need better tools for remote participation and ad hoc presentation Capture more than just slides and video (e.g., notes, links to related material, drawings on whiteboard, search indexes, etc.) Develop heuristics to automatically control broadcasts
m
m
User-interface for presenter
m
Lecture performance and capture
Future TV (9/00)
57
Summary
m
Internet webcasting will enable important applications in the future
Many problems and opportunities remain to be explored
m
Distributed collaboration will happen
First promised in early 1960’s, …
m
Issues/opportunities
Experiment with physical spaces Continue to develop and experiment with system architecture Problems with multicast network protocols Must be low cost and easy to operate
Future TV (9/00)
58
29
More Information
m
BMRC website
http://bmrc.berkeley.edu
m
Specific links
…/bibs – class lectures …/mig – Berkeley MIG Seminar …/papers – research publications …/projects – other multimedia content …/projects/sigchi – ACM SIGCHI Video DB …/projects/lb – Lecture Browser
Future TV (9/00)
59
References
m
Broadcast Manager
An Internet MBone Broadcast Management System by D. Wu, A. Swan, and L.A. Rowe, SPIE Multimedia Computing and Networking , January 1999. http://bmrc.berkeley.edu/research/publications/1999/151/
m
Director’s Console
dc: A Live Webcast Control System by T.P. Yu, D. Wu, K. Meyer -Patel, and L.A. Rowe, submitted for publication, January 2000. http://bmrc.berkeley.edu/research/publications/2000/157/
m
QuestionBoard
Floor Control for Large-Scale MBone Seminars by R. Malpani and L.A. Rowe. Proc. of The Fifth Annual ACM Intl. Multimedia Conf., November 1997. http://bmrc.berkeley.edu/research/publications/1997/137/
m
Parallel Video Special Effects
Exploiting Temporal Parallelism For Software-only Video Effects Processing by Ketan Mayer-Patel and L.A. Rowe. Proc. of The Sixth Annual ACM Intl. MultimediaConf., September 1998 http://www.bmrc.berkeley.edu/research/publications/1998/149/149.html .
Future TV (9/00)
60
30
References (cont.)
m
Parallel Video Special Effects (cont.)
Exploiting Temporal Parallelism For Software-only Video Effects Processing by Ketan Mayer-Patel and L.A. Rowe. Proc. of The Sixth Annual ACM Intl. MultimediaConf., September 1998 http://bmrc.berkeley.edu/research/publications/1998/149/149.html . Exploiting Spatial Parallelism For Software-only Video Effects Processing by K. Mayer -Patel and L.A. Rowe, SPIE Multimedia Computing and Networking , January 1999. http://bmrc.berkeley.edu/research/publications/1999/150/ Multicast Control Scheme For Parallel Software-only Video Effects Processing by K. Mayer -Patel and L.A. Rowe. Proc. of The Seventh Annual ACM Intl. Multimedia Conf., October 1999. http://bmrc.berkeley.edu/research/publications/1999/154/ A Parallel Software-Only Video Effects Processing System by K. Mayer -Patel. PhD Dissertation, Computer Science, U.C. Berkeley, December 1999. http://www.bmrc.berkeley.edu/research/publications/1999/155/index .html
m
Layered Source/Channel Transmission
Layered Transmission and Caching for the Multicast Session Directory Service by Andrew Swan, Steven McCanne, and L.A. Rowe. Proc. of The Sixth Annual ACM Intl. Multimedia Conf., September 1998. Best student paper award. http://bmrc.berkeley.edu/research/publications/1998/148/148. html
Future TV (9/00)
61
31