Docstoc

Media Plane Programming

Document Sample
Media Plane Programming Powered By Docstoc
					Chapter 11. Media Plane
    Programming
Objects in this chapter


introduce the Java Media Framework (JMF), a simple yet powerful tool
for handling media in Java applications.
Furthermore, high-performing applications may require lower-level APIs
than the one provided by JMF. For instance, another option to develop
voice application could be to use the lower-level Java Sound API.
Our sole interest is just to focus on some of the key functionalities offered
by the API, and let the reader gain an understanding of its scope of
applicability and potential.
Additionally, we will build a simple program that is able to:
    capture media and transmit it over the network
    receive media over the network and render it to the user




                                     (2)
Overview-(1)
The JMF allows programmers to develop Java code to capture, present, store, and
process time-based media. Moreover, it can be extended to support additional
media types and perform custom processing.
Additionally, JMF defines an optional RTP API to enable the transmission and
reception of RTP streams.
The fundamental data-processing model offered by the JMF API.




                                      (3)
Overview-(2)
The model considers three stages in a data-processing flow: input, processing, and output.
     The input stage is meant to acquire the media data. The media data can be obtained from different
     sources.
        From capture device (e.g., from microphone or camera)
        From file (e.g., music.wav)
        From the network (e.g., from received RTP stream)
     The processing stage takes the data obtained at the input stage and applies some processing to it,
     such as:
        Multiplexing/demultiplexing
        Encoding/decoding
        Packetizing/depacketizing
     The output stage is responsible for sending the media data to the destination. Possible destinations
     are:
        A presentation device (e.g., soundcard and loudspeakers)
        A file
        The network (e.g., transmit the media data as an RTP stream)




                                                  (4)
Overview-(3)
Time-based media takes the form of a media stream. The aim of the input stage is
to obtain a media stream. The processing stage also results in a new media stream,
which can then be fed into the output stage for presentation and so forth. In order to
obtain a media stream at the input stage, we can programmatically specify its
location and the protocol used to access it. In order to represent the location and
the protocol, sometimes a URL or a media locator1 format is used. For example:
     A media stream obtained from a local fi le could be identified by a “file://”
     URL.
     A media stream obtained from a fi le in a web server might be identified by
     an “http://” URL.
     A media stream obtained from the network could be represented by an
     “rtp://” media locator.
     A media stream captured from the soundcard could be represented by a
     “dsound://” media locator.
Media streams can also contain multiple channels of data called tracks. For
example, a media stream might contain both an audio track and a video track.
A media stream that contains multiple tracks is said to be multiplexed. The process
of extracting the individual tracks is called demultiplexing.



                                         (5)
JMF Entities-(1)
 The JMF API defines several entities that model media processing. The
 main entities are:
    Managers
    Data source
    Player
    Processor
    Data sink
    Session manager




                                 (6)
JMF Entities-(2)
 Managers
    In order for an application to obtain
    instances of objects that represent
    the main JMF entities (such as
    datasources, players, processors and
    datasinks), the application uses
    intermediary objects called
    managers.
    JMF uses four managers:
          Manager
          CaptureDeviceManager
          PackageManager
          PlugInManager
    Manager Class
    CaptureDeviceManager Class




                                            (7)
JMF Entities-(3)
 Data Source
    A data source is an entity that encapsulates a media stream. During the media
    handling process, different data sources may represent the underlying media
    streams at different stages of the process, as shown in Figure




  For example,
MediaLocator ml=new MediaLocator("fi le://c:\\music.wav");
DataSource ds= Manager.createDataSource(ml);




                                     (8)
JMF Entities-(4)
The Format Class
   The JMF API defines the Format class that represents a media format. It is
   extended by the AudioFormat and VideoFormat classes.
   For instance, in order to create an AudioFormat object, we would specify the
   following parameters:
         type of encoding (e.g., LINEAR, GSM, G723, etc.)
         sample rate
         number of bits per sample
         number of channels
   The following line of code creates an AudioFormat object that represents a
   GSM media format with sampling rate of 8,000 samples per second, 8 bits per
   sample, and just one channel:
 AudioFormat af=new AudioFormat(AudioFormat.GSM,8000,8,1);




                                         (9)
JMF Entities-(5)




                   (10)
JMF Entities-(6)




                   (11)
JMF Entities-(7)
 Player
    An entity responsible for processing and rendering a media stream. It is
    modeled by the Player interface. The media stream is conveyed to the Player
    as a DataSource.
    For instance, the following line of code creates a Player for the DataSource
    ds:
  Player p=Manager.createPlayer(ds);
    In order to start a player, we can invoke the start() method:
  p.start();




                                      (12)
JMF Entities-(8)




                   (13)
JMF Entities-(9)
 Processor
    A specialized type of player that provides control over media stream
    processing. It is modeled through the Processor interface that extends the
    Player interface.
    The Processor can multiplex, demultiplex, encode, decode, and apply effect
    filters over a media stream. A Processor can also render a media stream to a
    presentation device.
    The following code would create a Processor object for the DataSource ds:
  Processor p=Manager.createProcessor(ds);




                                     (14)
JMF Entities-(10)




                    (15)
JMF Entities-(11)
 Data Sinks
     gets a media stream as input, and renders the data to some destination
     (typically different from a presentation device).




  Create a DataSink for the specified input
    DataSink dsink=Manager.createDataSink(ds,ml);
  In order to start transferring data to the destination, two steps are needed:
      First we need to call open() on the DataSink in order to open a connection to the output
      destination identified by the MediaLocator.
      Next we need to invoke start() to actually initiate the data transfer:
    dsink.open();
    dsink.start();


                                            (16)
JMF Entities-(12)
 SessionManager
   may be used instead of a DataSink.
   A SessionManager offers an enhanced degree of control over RTP sessions
   compared to a DataSink (which offers almost no degree of control).




     The SessionManager represents an entity that is used to manage and
     coordinate an RTP session. It keeps track of the participants in the media
     session and keeps track of the media being transmitted. It also handles the
     RTCP control channel. Thus, it offers methods to:
           start and close an RTP session
           create RTP streams to be sent
           add and remove peers
           obtain session statistics
           etc.
                                        (17)
JMF Entities-(13)
 RTP Streams
    The RTPStream class, which represents an RTP stream. There are two types
    of RTP streams:
            ReceiveStream: represents an incoming RTP stream.
            SendStream: represents an outgoing RTP stream.
 Listeners
 A SessionManager can send session-related events to objects that implement
 specifc listener interfaces. Four types of listener are defined for the
 SessionManager:
    SessionListener: Receives notifications of changes in the state of the session.
    SendStreamListener: Receives notifications of changes in the state of the
    stream that is being transmitted.
    ReceiveStreamListener: Receives notifi cations of changes in the state of the
    stream that is being received.
    RemoteListener: Receives notifi cations of control messages from a remote
    participant.

                                          (18)
JMF Entities-(14)
 In order to use a SessionManager, fi rst we have to create an instance of it.
    RTPSessionMgr sm=new() RTPSessionMgr;
 Next we would need to initialize the SessionManager by calling its initSession() method and passing
 some confi guration parameters such as the local session addresses and so forth.
   sm.initSession(localAddress,.......);
 sm.startSession(receiver address,......,destination address,....);
 SendStream ss=sm.createSendStream(ds,1);
 And then we could start actual transmission of the stream:
    ss.start();
 In order to receive a media stream, as soon as this is detected by the SessionManager, it would fire a
 ReceivedStreamEvent event to our listener, which would then obtain a reference to the ReceivedStream:
    ReceiveStream rs= event.getReceiveStream();
 And next we would obtain a DataSource from the ReceiveStream;
       DataSource ds=rs.getDataSource();
 JMF defines the SessionAddress class that encapsulates a session address. It comprises four
 pieces of information:
      IP address for RTP
      Port for RTP
      IP address for RTCP
      Port for RTCP
 Example:
   InetAddress addr=InetAddress.getByName("1.2.3.4");
   SessionAddress sa=new SessionAddress(addr, 50000, addr, 50001);



                                                (19)
JMF Operation-(1)
 How the API is used in order to implement the following operations:
    capture live media
    capture media file
    present media
    send media to file
    process media
    receive media from network
    send media over network




                                     (20)
JMF Operation-Capture Live Media



 Use the Manager to create the DataSource. JMF provides two ways to obtain the
 DataSource from a capture device:
     If we know the media locator of the capture device, we can directly obtain the
     DataSource from it. In the following example, “dsound://8000” represents an
     audio card that samples voice at 8,000 Hz:
   MediaLocator ml=new MediaLocator("dsound://8000");
   DataSource ds= Manager.createDataSource(ml);
     Obtain the CaptureDeviceInfo corresponding to a capture device that supports
     a specified format. As we saw in previous sections, we can invoke the method
     getDeviceList on the CaptureDeviceManager, passing the specification of the
     desired format. Once we have the CaptureDeviceInfo, we can obtain a media
     locator from it:
   AudioFormat df=new AudioFormat(AudioFormat.LINEAR,8000,8,1);
   Vector devices=CaptureDeviceManager.getDeviceList(df);
   CaptureDeviceInfo di=(CaptureDeviceInfo) devices.elementAt(0);
   DataSource ds=Manager.createDataSource(di.getLocator());


                                         (21)
JMF Operation-Capture Live Media



 Use the Manager to create the DataSource. JMF provides two ways to obtain the
 DataSource from a capture device:
     If we know the media locator of the capture device, we can directly obtain the
     DataSource from it. In the following example, “dsound://8000” represents an
     audio card that samples voice at 8,000 Hz:
   MediaLocator ml=new MediaLocator("dsound://8000");
   DataSource ds= Manager.createDataSource(ml);
     Obtain the CaptureDeviceInfo corresponding to a capture device that supports
     a specified format. As we saw in previous sections, we can invoke the method
     getDeviceList on the CaptureDeviceManager, passing the specification of the
     desired format. Once we have the CaptureDeviceInfo, we can obtain a media
     locator from it:
   AudioFormat df=new AudioFormat(AudioFormat.LINEAR,8000,8,1);
   Vector devices=CaptureDeviceManager.getDeviceList(df);
   CaptureDeviceInfo di=(CaptureDeviceInfo) devices.elementAt(0);
   DataSource ds=Manager.createDataSource(di.getLocator());


                                         (22)
JMF Operation-Capture Media File




 The best way to do that is through a URL that represents the local file.
 For instance, in order to obtain the media stream from the file music.wav,
 we could do the following:
  MediaLocator ml=new MediaLocator("file://c:\\music.wav");
  DataSource ds=Manager.createDataSource(ml);
 If the media stream were stored in a remote file in a web server, we could
 obtain it by using an HTTP URL.



                                  (23)
JMF Operation-Present Media




 The most common way to do so is by using a Player. The following example
 represents the simplest way to play the media stream contained in DataSource ds:
   Player player = Manager.createPlayer(ds);
   player.start();
 The start() method attempts to transition the Player to the Started state as soon as
 possible. Therefore, it automatically tries to move the Player to the
 Realized state, then to the Prefetched state, and finally to the Started state. One
 way to do that is by implementing the ControllerListener interface and explicitly
 invoking the realize() and prefetch() methods when appropriate.



                                       (24)
JMF Operation-Send Media to File




 The simplest way to send media to a file is to create a DataSink object
 that points to the file URL, and pass the input DataSource in the creation
 method. Once created, we just open and start the data sink. MediaLocator
 ml=new
      MediaLocator("fi le://c:\\oo.wav");
      DataSink sink=Manager.createDataSink(ds,ml);
      sink.open();
      sink.start();
 It is important to note that, in this case, the DataSource ds represents the
 input media stream to the DataSink, whereas the MediaLocator ml is used
 to determine the file acting as sink for the media.


                                   (25)
JMF Operation-Process Media




 The first step is to create the Processor from the input DataSource iDS:
  Processor p=Manager.createProcessor(iDS);
 The next step would be to instruct the Processor to transit to the
 Configured state:
   p.configure();
 The configure method is asynchronous, therefore we need to wait until the
 Configured state is reached in order to set up the processing rules.
   while (p.getState()!=Processor.Confi gured) {
   Thread.sleep(20);}
 Using the loop approach is not recommended for commercial code. A commercial
 product might want to implement the ControllerListener interface and set the rules
 when a transition event to Configured is fired. Another possible option is to use
 the StateHelper class included in the JMF package.

                                      (26)
JMF Operation-Process Media
 Next we set the processing rules by defining which is the desired format of the first and only track in the
 input media stream. To do so, f rst we create an AudioFormat object that represents the desired GSM
 format with a sampling rate of 8,000 samples per second and 4 bits to represent each sample. The last
 argument represent the number of audio channels; in our case, just one:
       AudioFormat af=new AudioFormat(AudioFormat.GSM,8000,4,1);
 Then we get a TrackControl object that allows us to invoke the setFormat() method:
       TrackControl track[]= p.getTrackControls();
       track[0].setFormat(af);
 Once the output format is defined in the Processor, we move it to the Realized state and wait for the
 Processor to become Realized:
       p.realize();
       while (p.getState() != Processor.Realized) {
       Thread.sleep(20);}
 Then we obtain the output DataSource and invoke start() on the Processor:
       DataSource oDS = p.getDataOutput();
       p.start();
 If we wanted to send the output media stream over the network, we should have asked the Processor to
 not only encode the input stream, but also to perform packetization. The way to indicate to the Processor
 that the stream needs to be packetized for sending it in an RTP session is just to append “_RTP” to the
 desired media format that is passed as a parameter to the setFormat() method:
       AudioFormat af=new AudioFormat(AudioFormat.GSM_RTP,8000,4,1);


                                                 (27)
JMF Operation-Receive and Send Media from/over the
Network
 The JMF RTP API offers two ways to receive and send RTP media from the network. The first way uses
 just RTP media locators, whereas the second one implies using a SessionManager.
 Approach 1: Media Locators
 For the receiving case, let us imagine that the IP address and port where our receiver application is
 expecting the media is 1.2.3.4:40000. In the simplest approach, we just create a DataSource from an RTP
 media locator:
    MediaLocator ml=new MediaLocator("rtp://1.2.3.4:40000/audio/1");
    DataSource rDS=Manager.createDataSource(ml);
      The last “1” in the RTP media locator represents the time to live (TTL) in RTP packets.
 For the sending case, in its simplest form, in order to send a media stream over the network using RTP,
 we just need to create a DataSink object and pass two arguments to it:
      The DataSource object that represents the media stream that we want to send over the network.
      A RTP media locator that identifies the destination of the stream. Let us assume that the address of the destination is
      5.4.3.2:50000:
      MediaLocator ml=new MediaLocator("rtp://5.4.3.2:50000/audio/1");
      DataSink sink=Manager.createDataSink(tDS,ml);
 Once the data sink has been created, we just need to open and start it:
      sink.open();
      sink.start();




                                                          (28)
JMF Operation-Receive and Send Media from/over the
Network
 The JMF RTP API offers two ways to receive and send RTP media from the network. The first way uses
 just RTP media locators, whereas the second one implies using a SessionManager.
 Approach 1: Media Locators
 For the receiving case, let us imagine that the IP address and port where our receiver application is
 expecting the media is 1.2.3.4:40000. In the simplest approach, we just create a DataSource from an RTP
 media locator:
    MediaLocator ml=new MediaLocator("rtp://1.2.3.4:40000/audio/1");
    DataSource rDS=Manager.createDataSource(ml);
      The last “1” in the RTP media locator represents the time to live (TTL) in RTP packets.
 For the sending case, in its simplest form, in order to send a media stream over the network using RTP,
 we just need to create a DataSink object and pass two arguments to it:
      The DataSource object that represents the media stream that we want to send over the network.
      A RTP media locator that identifies the destination of the stream. Let us assume that the address of the destination is
      5.4.3.2:50000:
      MediaLocator ml=new MediaLocator("rtp://5.4.3.2:50000/audio/1");
      DataSink sink=Manager.createDataSink(tDS,ml);
 Once the data sink has been created, we just need to open and start it:
      sink.open();
      sink.start();




                                                          (29)
JMF Operation-Receive and Send Media from/over the
Network
 The JMF RTP API offers two ways to receive and send RTP media from the network. The first way uses
 just RTP media locators, whereas the second one implies using a SessionManager.
 Approach 1: Media Locators
 For the receiving case, let us imagine that the IP address and port where our receiver application is
 expecting the media is 1.2.3.4:40000. In the simplest approach, we just create a DataSource from an RTP
 media locator:
    MediaLocator ml=new MediaLocator("rtp://1.2.3.4:40000/audio/1");
    DataSource rDS=Manager.createDataSource(ml);
      The last “1” in the RTP media locator represents the time to live (TTL) in RTP packets.
 For the sending case, in its simplest form, in order to send a media stream over the network using RTP,
 we just need to create a DataSink object and pass two arguments to it:
      The DataSource object that represents the media stream that we want to send over the network.
      A RTP media locator that identifies the destination of the stream. Let us assume that the address of the destination is
      5.4.3.2:50000:
      MediaLocator ml=new MediaLocator("rtp://5.4.3.2:50000/audio/1");
      DataSink sink=Manager.createDataSink(tDS,ml);
 Once the data sink has been created, we just need to open and start it:
      sink.open();
      sink.start();




                                                          (30)
JMF Operation-Receive and Send Media from/over the
Network
 Another approach to receive and send media from/over the network consists of using a
 SessionManager.
 In order to receive incoming streams, our application would implement the
 ReceiveStreamListener interface. As soon as the incoming session is detected (i.e., the first
 RTP packets are received), the SessionManager will post a NewReceiveStreamEvent.
 From that event, we will get the ReceiveStream, and from the ReceiveStream, it is possible
 to obtain a DataSource (rDS).
 On the other hand, let us assume that we want to transmit via RTP the stream represented by
 a DataSource (tDS). First we need to obtain a reference to a SendStream object from the
 DataSource object. Then we would simply call the start() method on the SendStream object
 in order to start transmitting.




                                           (31)
JMF Operation-Receive and Send Media from/over the
Network
 Another approach to receive and send media from/over the network consists of using a
 SessionManager.
 In order to receive incoming streams, our application would implement the
 ReceiveStreamListener interface. As soon as the incoming session is detected (i.e., the first
 RTP packets are received), the SessionManager will post a NewReceiveStreamEvent.
 From that event, we will get the ReceiveStream, and from the ReceiveStream, it is possible
 to obtain a DataSource (rDS).
 On the other hand, let us assume that we want to transmit via RTP the stream represented by
 a DataSource (tDS). First we need to obtain a reference to a SendStream object from the
 DataSource object. Then we would simply call the start() method on the SendStream object
 in order to start transmitting.




                                           (32)
JMF Operation-Receive and Send Media from/over the
Network
 Let us see step-by-step how this works.First we need to create an object that implements the SessionManager interface.
       RTPSessionMgr sm = new RTPSessionMgr();
 In order to receive the ReceiveStreamEvents, our class needs to implement the ReceiveStreamListener interface. We also need to register our
 interest in receiving events from the SessionManager. That is achieved by invoking the method addReceiveStreamListener() on the
 SessionManager:
       sm.addReceiveStreamListener(this);
 Then we need to initialize and start the session in the SessionManager. We need to pass some configuration parameters as arguments to the
 initSession() and startSession() methods on the SessionManager.
 The necessary code would be:
     InetAddress localIP = InetAddress.getByName("1.2.3.4");
     InetAddress remIP = InetAddress.getByName("5.4.3.2");
     SessionAddress senderAddr = new SessionAddress();
     SessionAddress localAddr = new SessionAddress(localIP,
     40000,localIP,40001);
     SessionAddress remAddr=new SessionAddress(remIP,50000,remIP,
     50001);
     sm.initSession(senderAddr, null, 0.05, 0.25);
     sm.startSession(localAddr,localAddr,remoteAddr, null);
 If that is the case, we obtain the ReceiveStream. From it, we obtain the DataSource object, which was our target:
 public class MyReceiveStreamListener implements
     ReceiveStreamListener {
     public void update(ReceiveStreamEvent event) {
     if (event instanceof NewReceiveStreamEvent){
     rs=event.getReceiveStream();
     DataSource rDS=rs.getDataSource();
     }}}
 In order to send data, once the session manager is started, we just need to create a SendStream from our DataSource and
 invoke the start method on the SendStream objects:
     ss = tManager.createSendStream(tDS, 1);
     ss.start();




                                                                    (33)
Putting It All Together: The VoiceTool
 The VoiceTool Java class contains the necessary methods to start and stop transmission
 and reception of voice. It uses a single session manager, myVoice-SessionManager, for
 both reception and transmission, which is defined as a member of the class. VoiceTool
 implements the ReceiveStreamListener interface.
   public class VoiceTool implements ReceiveStreamListener {
   private RTPSessionMgr myVoiceSessionManager=null;
   private Processor myProcessor=null;
   private SendStream ss=null;
   private ReceiveStream rs=null;
   private Player player=null;
   private AudioFormat af=null;
   private DataSource oDS=null;
 VoiceTool offers three methods:
     int startMedia (String peerIP, int peerPort, int recvPort,int fmt): This method creates
     the RTP unicast session between the local host at recvPort, and the remote host,
     peerIP, at peerPort.Then it starts voice transmission and reception. The last argument,
     fmt,indicates the audio format used for transmission. For simplicity, we will consider
     only two possible video formats (GSM_RTP and G723_RTP). This method will
     return an integer value of 1 if it was executed successfully, or a negative value if an
     error was encountered.
     void stopMedia(): This method is used to stop voice transmission and reception.
     void update(ReceiveStreamEvent event): This is a method from the
     ReceiveStreamListener interface that VoiceTool implements.
                                           (34)
Putting It All Together: The VoiceTool




                       (35)
Putting It All Together: The VideoTool
 The VideoTool Java class contains the necessary methods to start and stop transmission and reception
 of video. It uses a single session manager, myVideo-SessionManager, for both reception and
 transmission, which is defined as a member of the class. VideoTool implements the
 ReceiveStreamListener interface.
 public class VideoTool implements ReceiveStreamListener {
 private RTPSessionMgr myVideoSessionManager=null;
 private Processor myProcessor=null;
 private SendStream ss=null;
 private ReceiveStream rs=null;
 private Player player=null;
 private VideoFormat vf=null;
 private DataSource oDS=null;
 private VideoFrame vframe;\
 VideoTool offers three methods:
 int startMedia (String peerIP,int peerPort,int recvPort, int fmt): This method creates the RTP unicast
 session between the local host at recvPort and the remote host, peerIP, at peerPort. Then it starts video
 transmission and reception. The last argument, fmt, indicates the video format used for transmission.
 For simplicity, we will consider only two possible video formats (JPEG_RTP and H263_RTP). This
 method will return an integer value of 1 if it was executed successfully, or a negative value if an error
 was encountered.
 void stopMedia(): This method is used to stop video transmission and reception.
 void update(ReceiveStreamEvent event): This is a method from the ReceiveStreamListener interface
 that VideoTool implements.



                                                  (36)
Putting It All Together: The VideoTool




                       (37)
Putting It All Together: The ToneTool
 In this section, we will build a simple component that allows playing an
 alerting signal or a ringing tone based on two prestored files to which
 the soft-phone application is supposed to have access:
    alertsignal.wav
    ringtone.wav
 The example is quite straightforward; we will build a class called
 TonesTool that offers three methods:
  void prepareTone (String filename)
  void playTone ()
  void stopTone()




                                   (38)
Using the Components. Example 6




                     (39)

				
DOCUMENT INFO