Techniques for Object-Oriented Network Programming with C++ by yaohongm


 Techniques for Object-Oriented
Network Programming with C++                            Bene ts of distributed computing:
                                                         Collaboration ! connectivity and interworking
                                                         Performance ! multi-processing and locality
     Douglas C. Schmidt                                  Reliability and availability ! replication                                 Scalability and portability ! modularity
Washington University, St. Louis
                                                         Extensibility ! dynamic con guration and recon-
http: schmidt                          Cost e ectiveness   ! open systems and resource

                                              1                                                       2

     Challenges and Solutions
 Developing e cient, robust, and extensible                        Tutorial Outline
 distributed applications is challenging
  e.g., must address complex topics that are less       Outline key challenges for developing dis-
  problematic or not relevant for non-distributed ap-   tributed applications

 Object-oriented OO techniques and lan-               Present a concurrent distributed application
 guage features enhance distributed software            from the domain of enterprise medical imag-
 quality factors                                        ing
  Key OO techniques ! design patterns and frame-
                                                        Compare and contrast an algorithmic and
  Key OO language features ! classes, inheritance,      an Object-Oriented design and implemen-
  dynamic binding, and parameterized types
                                                        tation of the application
  Key software quality factors ! modularity, exten-
  sibility, portability, reusability, and correctness

                                              3                                                       4
                                                                Sources of Complexity
       Software Development                              Distributed application development exhibits
            Environment                                  both inherent and accidental complexity
Note, the topics discussed here are largely              Examples of Inherent complexity
independent of OS, network, and program-
ming language                                              Addressing the impact of latency
   They are currently used successfully on UNIX and
   Windows NT platforms, running on TCP IP and             Detecting and recovering from partial failures of
   IPX SPX networks, using C++                             networks and hosts
                                                           Load balancing and service partitioning
Examples are illustrated using freely avail-
able ADAPTIVE Communication Environ-
ment ACE OO framework components                       Examples of Accidental complexity
   Although ACE is written in C++, the principles          Lack of type-secure, portable, re-entrant, and ex-
   covered in this tutorial apply to other OO lan-         tensible system call interfaces and component li-
   guages                                                  braries
                                                           Wide-spread use of algorithmic decomposition

                                              5                                                       6

                                                            Medical Imaging Topology
    Concurrent Network Server
            Example                                              ATM

The following example illustrates a concur-
rent OO architecture for medical Image Servers
in an enterprise distributed health care de-
livery system                                                                     DIAGNOSTIC
                                                                  ATM              STATIONS
Key system requirements are to support:                 IMAGE
1. Seamless electronic access to radiology expertise
   from any point in the system
2. Immediate on-line access to medical images via ad-                                                 LAN
   vanced diagnostic workstations attached to high-
   speed ATM networks
3. Teleradiology and remote consultation capabilities                          MODALITIES
   over wide-area networks                                                    (CT, MR, CR)

                                              7                                                       8
                                                                    Multi-threaded Image Server
Concurrent Image Server Example                                             Architecture
                                                              CPU1                CPU2             CPU3                   CPU4
                       : IMAGE        : NETWORK   : IMAGE
 : PRINTER             SERVER                     LOCATOR

                                                                 worker             worker              worker               worker
                                                              1: dequeue(msg)    1: dequeue(msg)   1: dequeue(msg)        1: dequeue(msg)
                           SOFTWARE BUS
                                                              2: process()       2: process()      2: process()           2: process()

: NAME               : IMAGE         : IMAGE      : AUTHEN-
SERVER                              PROCESSOR
                     DISPLAY                      TICATOR
                                                                                            : Message

                                                                                                        1: select()
                                                                                MASTER                  2: recv(msg)

  Image Servers have the following responsi-                                    SERVER                  3: enqueue(msg)

  * Store retrieve large medical images
  * Respond to queries from Image Locater Servers
  * Manage short-term and long-term image persistence
                                                                 Worker threads execute within one process
                                                    9                                                                              10

      Pseudo-code for Concurrent                                                Thread Entry Point
            Image Server                                         Each thread executes a function that serves
                                                                 as the entry point" into a separate thread
                                                                 of control
  Pseudo-code for master server
                                                                    Note algorithmic design         :::

  void       master server void
  f                                                                 typedef u_long COUNTER;
         initialize listener endpoint and work queue                   Track the number of requests
         spawn pool of worker threads                               COUNTER request_count;    At file scope.
         foreach pending work request f
               receive and queue request on work queue                 Entry point into the image request service.

         g                                                          void *worker Message_Queue *msg_queue

         exit process
  g                                                                    Message_Block *mb;               Message buffer.

                                                                       while msg_queue- dequeue_head mb                       0
  Pseudo-code for thread pool workers                                       Keep track of number of requests.
  void       worker void
  f                                                                             Identify and perform Image Server
         foreach  work request on queue                                       request processing here...
               dequeue and process request
         exit thread                                                   return 0;

                                                    11                                                                             12
 Master Server Driver Function
 The master driver function in the Image
 Server might be structured as follows:                       Pseudo-code for recv requests
    Thread function prototype.
 typedef void **THR_FUNCvoid *;                           e.g.,
 static const int NUM_THREADS = * ... * ;
                                                              void   recv requests Message Queue &msg queue
 int main int argc, char *argv                              f
   Message_Queue msg_queue;    Queue client requests.
                                                                     initialize socket listener endpoints
     Spawn off   NUM_THREADS to run in parallel.
  for int i =   0; i   NUM_THREADS; i++                            foreach   incoming request
    thr_create   0, 0, THR_FUNC &worker,                          f
      void *   &msg_queue, THR_BOUND | THR_SUSPENDED, 0;               use select to wait for new connections or data
                                                                          if connection
     Initialize network device and recv work requests.                          establish connections using accept
  recv_requests msg_queue;
                                                                          else if data f
                                                                                use sockets calls to read data into msg
     Resume all suspended threads assumes contiguous id's
                                                                                msg queue.enqueue tail msg;
  for i = 0; i   NUM_THREADS; i++
    thr_continue t_id--;
     Wait for all threads to exit.                            g
  while thr_join 0, &t_id, void ** 0 == 0
    continue;    ...

                                              13                                                              14

Limitations with the Image Server
 The algorithmic decomposition tightly cou-                   Eliminating Race Conditions in
 ples application-speci c functionality with
 various con guration-related characteristics,                       the Image Server
   The image request handling service                         The original Image Server uses a Message Queue
                                                              to queue Message Blocks
   The use of sockets and select                                  The worker function running in each thread de-
                                                                  queues and processes these messages concurrently
   The number of services per process
   The time when services are con gured into a pro-           A naive implementation of Message Queue will
   cess                                                       lead to race conditions
                                                                  e.g., when messages in di erent threads are en-
                                                                  queued and dequeued concurrently
 There are race conditions in the code

                                                              The solution described below requires the
 The solution is not portable since it hard-                  thread-safe ACE Message Queue class
 codes a dependency on SunOS 5.x thread-
 ing mechanisms
                                              15                                                              16
                                                                              1 Single-threaded Image Server
  An OO Concurrent Image Server                                                          Architecture
    The following example illustrates an OO so-
    lution to the concurrent Image Server
       The active objects are based on the ACE                        Task      THREAD
       class                                                                                                        : Image                : Image
                                                                                                                    Request                Request
                                                                                                 : Image            Handler                Handler
                                                                                 : Image         Request

    There are several ways to structure concur-                                  Request

    rency in an Image Server                                                                                                          3: handle_input()
                                                                                                                                      4: recv()
                                                                                                                                      5: process()
   1. Single-threaded, with all requests handled in one                                                    : Reactor
      thread                                                                                                              2: dispatch()
                                                                                                  1: select()
   2. Multi-threaded, with all requests handled in sepa-
      rate threads
   3. Multi-threaded, with all requests handled by a thread
                                                                                 Every handler processes one connection

                                                                     17                                                                           18

                                                                               3 Multi-threaded Image Server
  2 Multi-threaded Image Server                                                        Architecture
                                                                             CPU1              CPU2                 CPU3                CPU4

         : Image                  : Image                  : Image
         Request                  Request                  Request
         Handler                  Handler                  Handler               : Image           : Image              : Image               : Image
                                                                                 Request           Request              Request               Request
                                                       1: recv()                 Handler           Handler              Handler               Handler
    1: recv()              1: recv()
    2: process()           2: process()                2: process()          1: dequeue(msg)   1: dequeue(msg)      1: dequeue(msg)       1: dequeue(msg)
                                            : Image                          2: process()      2: process()         2: process()          2: process()
                       : Image              Request       : Image
    : Image            Request              Handler
    Request                                               Request
                       Handler                            Handler
                                       1: recv()
                   1: recv()           2: process()   1: recv()                                            : Message
1: recv()          2: process()                                                                              Queue
2: process()                                          2: process()

                                                                                                  1: select()
                                                                                    : Reactor     2: dispatch()                   MASTER
                                                                                                  3: recv(msg)                    THREAD
                                                                                                  4: enqueue(msg)

    Every handler processes one connection
                                                                                 Every handler processes one request
                                                                     19                                                                           20
              Design Patterns in the Image                                            Using the Reactor for the Image
                         Server                                                                   Server
                                                                                                   REGISTERED                        : Image
                  Active Object                                                                      OBJECTS                         Handler
                                                 Acceptor                                                                                   : Image

                                                                                                                                                   : Image
                                                                                                                : Message                         Handler

              Half-Sync/                                     Service                          : Network
              Half-Async                                   Configurator                         Server
                                                                                                             2: recv_msg(msg)       4: getq(msg)
                                                                                                             3: putq(msg)           5:process(msg)
                                                                                               : Event
  STRATEGIC                            Reactor

                                                                                                   1: handle_input()

                                                                                                                             : Handle
   TACTICAL                 Template        Factory                                                                            Table
   PATTERNS                  Method         Method         Adapter
                                                                                                                                   : Reactor
                                                                                                                OS EVENT DEMULTIPLEXING INTERFACE

         The Image Server is based upon a system
         of design patterns
                                                                       21                                                                            22

       Using the Active Object Pattern                                                 Using the Half-Sync Half-Async
             for the Image Server                                                       Pattern for the Image Server
                                                                                                                       : Image
                                                                                      SYNCH TASK

                                                       : Image                                                                 : Image
                    OBJECTS                            Handler                                                                 Handler : Image

                                                              : Image

                                                                     : Image                          4: getq(msg)
                                : Message                           Handler                           5:process(msg)

                : Network
                              2: recv_msg(msg)        4: getq(msg)

                              3: putq(msg)            5:process(msg)
                                                                                                                         : Message

                 : Event
                 Handler                                                                                                   Queue
                                                                                                        : Network

                  1: handle_input()                                                                       Server

                                                                                                                       2: recv_msg(msg)
                                              : Handle                                                   : Event       3: putq(msg)
                                                                                      ASYNC TASK

                                                   : Reactor

                                OS EVENT DEMULTIPLEXING INTERFACE
                                                                                                           1: handle_input()

                                                                                                                                   : Reactor

                                                                       23                                                                            24
                                                               Network Server Public Interface
   Image Server Public Interface
  The Image Server class implements the ser-                    Network Server implements the asynchronous
  vice that processes image requests synchronously              tasks in the Half-Sync Half-Async pattern
    To enhance reuse, the Image Server is derived
    from a Network Server                                          Reusable base class.
                                                                template class PEER_ACCEPTOR      Passive conn. factory
    template class PEER_ACCEPTOR      Passive conn. factory     class Network_Server : public Task MT_SYNCH
    class Image_Server
      : public Network_Server PEER_ACCEPTOR                     public:
                                                                       Dynamic linking hooks.
    public:                                                       virtual int init int argc, char *argv;
           Pass a message to the active object.                   virtual int fini void;
      virtual put Message_Block *, Time_Value *;
                                                                       Pass a message to the active object.
           Concurrent entry point into server thread.             virtual put Message_Block *, Time_Value *;
      virtual int svc int;
     ;                                                                 Accept connections and process from clients.
                                                                  virtual int handle_input HANDLE;

                                              25                                                               26

      Network Server Protected                                  Network Server Implementation
              Interface                                          Short-hand definitions.
                                                              define PEER_ACCEPTOR PA
       Parse the argc argv arguments.
                                                                Initialize server when dynamically linked.
  int parse_args int argc, char *argv   ;
                                                              template class PA int
      Initialize network devices and connections.
                                                              Network_Server PA ::init int argc, char *argv    
 int init_endpoint void;
                                                               parse_args argc, argv;
      Receive and frame an incoming message.
 int recv_message PEER_ACCEPTOR::PEER_STREAM &,
                                                               thr_mgr_ = new Thread_Manager;
                   Message_Block &*;
                                                                  Create all the threads start them suspended.
      Acceptor factory for sockets.
                                                               thr_mgr_- spawn_n num_threads_,
 PEER_ACCEPTOR acceptor_;
                                                                                        THR_FUNC svc_run,
                                                                                        void * this,
      Track  of requests.
                                                                                        THR_BOUND | THR_SUSPENDED;
 Atomic_Op   request_count_;
                                                                  Initialize communication endpoint.
                                                               init_endpoint ;
       of threads.
 int num_threads_;
                                                                  Resume all suspended threads.
                                                               thr_mgr_- resume_all ;
       Listener port.
                                                               return 0;
  u_short server_port_;

                                              27                                                               28
                                                      Called back by Reactor when events arrive from clients.
                                                      This method implements the asynchronous portion of the
                                                      Half-Sync Half-Async pattern...
template class PA int
Network_Server PA ::init_endpoint void            template class PA int
                                                    Network_Server PA ::handle_input HANDLE h
    Open up the passive-mode server. server_port_;                      PA::PEER_STREAM stream;

    Register this object with the Reactor.              Handle connection events.
 Service_Config::reactor- register_handler         if h == acceptor_.get_handle 
   this, Event_Handler::READ_MASK;                   acceptor_.accept stream;
                                                       Service_Config::reactor- register_handler
                                                         stream.get_handle , this, Event_Handler::READ_MASK;
  Called when service is dynamically unlinked.

template class PA int                                   Handle data events asynchronously
Network_Server PA ::fini void                      else
                                                       Message_Block *mb = 0;
    Unblock threads.
 msg_queue_- deactivate ;                            stream.set_handle h;

    Wait for all threads to exit.                         Receive and frame the message.
 thr_msg_- wait ;                                    recv_message stream, mb;

 delete thr_msg_;                                         Insert message into the Queue this call forms
                                                          the boundary between the Async and Sync layers.
                                                       putq mb;

                                               29                                                     30

                                                    Eliminating Race Conditions Part
   Pass a message to the active object.
template class PA int
                                                                 1 of 2
Image_Server PA ::put Message_Block *msg,
                       Time_Value *tv
                                                      There is a subtle and pernicious problem
 putq msg, tv;                                      with the concurrent server illustrated above:
                                                        The auto-increment of global variable request count
   Concurrent entry point into the service. This        is not serialized properly
   method implements the synchronous part of the
   Half-Sync Half-Async pattern.
template class PA int
Image_Server PA ::svc void                          Lack of serialization will lead to race condi-
  Message_Block *mb = 0;    Message buffer.           tions on many shared memory multi-processor
    Wait for messages to arrive.
 while getq mb != -1
                                                        Note that this problem is indicative of a large class
      Keep track of number of requests.
                                                        of errors in concurrent programs  :: :

      Identify and perform Image Server
      request processing here...
                                                      The following slides compare and contrast a
 return 0;
                                                      series of techniques that address this prob-

                                               31                                                     32
       Basic Synchronization                               C++ Wrappers for
            Mechanisms                                      Synchronization
One approach to solve the serialization prob-
lem is to use OS mutual exclusion mecha-         De ne a C++ wrapper to address portabil-
nisms explicitly, e.g.,                          ity and elegance problems:
   SunOS 5.x, implicitly "unlocked".             class Thread_Mutex
mutex_t lock;
typedef u_long COUNTER;                          public:
COUNTER request_count;                             Thread_Mutex void
                                                     mutex_init &lock_, USYNCH_THREAD, 0;
template class PA int
Image_Server PA ::svc void                      ~Thread_Mutex void   mutex_destroy &lock_;
     in function scope ...                        int acquire void   return mutex_lock &lock_;
    mutex_lock &lock;                           int release void   return mutex_unlock &lock_;
    mutex_unlock &lock;                        private:
     ...                                           mutex_t lock_;     SunOS 5.x serialization mechanism.

However, adding these mutex * calls explic-      Note, this mutual exclusion class interface
itly is inelegant, obtrusive, error-prone, and   is portable to other OS platforms
                                           33                                                 34

    Porting Thread Mutex to                      Using the C++ Thread Mutex
          Windows NT                                        Wrapper
WIN32 version of Thread Mutex:
                                                 Using the C++ wrapper helps improve porta-
class Thread_Mutex                               bility and elegance:
                                                 Thread_Mutex lock;
  Thread_Mutex void
                                                 typedef u_long COUNTER;
    InitializeCriticalSection &this- lock_;
                                                 COUNTER request_count;
 ~Thread_Mutex void
                                                 template class PA int
   DeleteCriticalSection &this- lock_;
                                                 Image_Server PA ::svc void
 int acquire void
                                                     lock.acquire ;
   EnterCriticalSection &this- lock_;
   return 0;
                                                     lock.release ;    Don't forget to call!
 int release void
   LeaveCriticalSection &this- lock_;
   return 0;

       Win32 serialization mechanism.
                                                 However, it does not solve the obtrusiveness
  CRITICAL_SECTION lock_;                        or error-proneness problems     :::

                                           35                                                 36
Automated Mutex Acquisition and
           Release                                               Using the Guard Class
                                                          Using the Guard class helps reduce errors:
 To ensure mutexes are locked and unlocked,
 we'll de ne a template class that acquires               Thread_Mutex lock;
 and releases a mutex automatically                       typedef u_long COUNTER;
                                                          COUNTER request_count;

 template class LOCK                                      template class PA int
 class Guard                                              Image_Server PA ::svc void
   Guard LOCK &m: lock_ m   this- lock_.acquire ;         Guard Thread_Mutex   monitor lock;
   ~Guard void   this- lock_.release ;                      ++request_count;
   LOCK &lock_;

                                                          However, using the Thread Mutex and Guard
                                                          classes is still overly obtrusive and subtle
                                                          e.g., beware of elided braces    :::

 Guard  uses the C++ idiom whereby a con-                   A more elegant solution incorporates C++ fea-
 structor acquires a resource and the destruc-              tures such as parameterized types and overloading
 tor releases the resource
                                             37                                                         38

                                                           Transparently Parameterizing
         OO Design Interlude                                Synchonization Using C++
 Q: Why is Guard parameterized by the type                The following C++ template class uses the
 of LOCK?                                                  Decorator" pattern to de ne a set of atomic
                                                          operations on a type parameter:
 A: since there are many di erent avors of
 locking that bene t from the Guard func-                 template class LOCK = Thread_Mutex, class TYPE = u_long

 tionality, e.g.,                                         class Atomic_Op
                                                            Atomic_Op TYPE c = 0  this- count_ = c;
 * Non-recursive vs recursive mutexes                       TYPE operator++ void
 * Intra-process vs inter-process mutexes                     Guard LOCK m this- lock_; return ++this- count_;

 * Readers writer mutexes                                  void operator= const Atomic_Op &ao
 * Solaris and System V semaphores                           if this != &ao
 * File locks                                                  Guard LOCK m this- lock_; this- count_ = ao.count_;
 * Null mutex
                                                           operator TYPE 
 In ACE, all synchronization wrappers use to                 Guard LOCK m this- lock_;

 Adapter pattern to provide identical inter-                 return this- count_;

 faces whenever possible to facilitate param-                  Other arithmetic operations omitted...

 eterization                                              private:
                                                            LOCK lock_;
                                                            TYPE count_;
                                             39                                                         40
      Thread-safe Version of                                      Using the Service Con gurator
        Concurrent Server                                          Pattern in the Image Server
Using the Atomic Op class, only one change
is made to the code                                            SERVICE                                        : ST
                                                                                                          Image Server
                                                                                   : HS_HA
                                                               RUNTIME                                      : Service
if defined MT_SAFE                                                            Image Server
typedef Atomic_Op   COUNTER;    Note default parameters...                                         SHARED
                                                                  : Service       : Service       OBJECTS
typedef Atomic_Op Null_Mutex   COUNTER;
                                                                 Repository        Object
endif * MT_SAFE *                                                                                         : WP
COUNTER request_count;                                                                                  Image Server
                                                                 : Service       : Reactor
                                                                  Config                                : Service

request count   is now serialized automatically
template class PA int
Image_Server PA ::svc void
       Calls Atomic_Op::operator++void
                                                                 Existing service is based on Half-Sync Half-
    ++request_count;                                             Async pattern, other versions could be single-
                                                                 threaded or use other concurrency strategies           :::

                                            41                                                                  42

  Image Server Con guration                                     Parameterizing IPC Mechanisms
                                                                    with C++ Templates
The concurrent Image Server is con gured
and initialized via a con guration script                        To switch between a socket-based service
 cat . svc.conf
                                                                 and a TLI-based service, simply instantiate
dynamic HS_HA_Image_Server Service_Object *                      with a di erent C++ wrapper
         svcs "-p 2112 -t 4"
                                                                     Determine the communication mechanisms.

Factory function that dynamically allocates                      if defined ACE_USE_SOCKETS
                                                                 typedef SOCK_Stream PEER_STREAM;
a Half-Sync Half-Async Image Server object                       typedef SOCK_Acceptor PEER_ACCEPTOR;
                                                                 elif defined ACE_USE_TLI
                                                                 typedef TLI_Stream PEER_STREAM;
extern "C" Service_Object *alloc_server void;                  typedef TLI_Acceptor PEER_ACCEPTOR;
Service_Object *alloc_server void
                                                                 Service_Object *alloc_server void
 return new Image_Server SOCK_Acceptor ;
    ASX dynamically unlinks and deallocates this object.            return new Image_Server PEER_ACCEPTOR, PEER_STREAM ;

                                            43                                                                  44
                                                        The ADAPTIVE Communication
                                                             Environment ACE
               Main Program                                                               ACCEPTOR    CONNECTOR
                                                                                                                                          CLASS CATEGORIES
                                                                                                                                          AND FRAMEWORKS
                                                                                       ADAPTIVE SERVICE EXECUTIVE (ASX)

Dynamically con gure and execute the net-                 THR                                 LOG                  SERVICE

work service
                                                        MANAGER                               MSG                              MALLOC
                                                                                                      REACTOR      URATOR                             WRAPPERS
                                                      SYNCH         SPIPE   SOCK_SAP/       FIFO                               MEM        SYSV
                                                     WRAPPERS        SAP     TLI_SAP        SAP                                MAP      WRAPPERS
                                                     THREAD       STREAM    SOCKETS/      NAMED      SELECT/      DLOPEN      MMAP      SYSTEM          APIS
                                                     LIBRARY       PIPES      TLI          PIPES      POLL        FAMILY     FAMILY     V IPC
int main int argc, char *argv     
                                                    PROCESS                     STREAMS                           VIRTUAL MEMORY                      GENERAL
                                                   SUBSYSTEM                   SUBSYSTEM                             SUBSYSTEM                     OS   SERVICES
     Initialize the daemon and
     dynamically configure the service.
                                                           A set of C++ wrappers, class categories,
 Service_Config daemon argc, argv;
                                                           and frameworks based on design patterns
     Loop forever, running services and handling
                                                           C++ wrappers
 daemon.run_event_loop ;
                                                                e.g., IPC SAP, Synch, Mem Map
 return 0;

                                                           OO class categories and frameworks
                                                                e.g., Reactor, Service Con gurator, ADAPTIVE
                                                                Service eXecutive ASX
                                           45                                                                                                    46

            Obtaining ACE
The ADAPTIVE Communication Environ-
ment ACE is an OO toolkit designed ac-
cording to key network programming pat-
All source code for ACE is freely available
  Anonymously ftp to
  Transfer the les languages c++ ACE *.gz and
  gnu ACE-documentation *.gz

Mailing list

  http: ~schmidt

To top