ScalAgent

© 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. New Generation Mediation Solutions ScalAgent Distributed Technologies SA Phone : +33 (0)4 76297981 1, rue de Provence – BP208 Fax : +33 (0)4 76338773 F-38334 Echirolles Cedex – France www.scalagent.com A ScalAgent Distributed Technologies White Paper JORAM: The Open Source Enterprise Service Bus Author: Roland Balter Email: contact@scalagent.com March 1st, 2004 JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 2 Executive Summary The Enterprise distributed on Internet is facing the problem of federating semi-autonomous systems within a global coherent information system. This integration challenge has several facets such as application integration (EAI and B2B), data integration and cooperation between independent sub-systems that often rely on different technologies. This coexistence of heterogeneous integration infrastructures within the Enterprise leads to increasing costs and management complexity. In reality, all of these integration systems share common objectives and would be thus able to share also a common communication infrastructure we call the Enterprise Service Bus (ESB). Openness, flexibility and extensibility are the key properties expected from the ESB to meet the sharing objective. This white paper describes an open source solution, based on a messaging system compliant with the JMS specification that meets the requirements for an ESB. Its implementation is based on a cutting-edge technology that provides the flexibility, scalability and extensibility properties required for the construction of adapted integration systems. The proposed solution is a real alternative to proprietary integration platforms. The Distributed Enterprise Integrate Data, Applications and Equipments The use of Internet as a general purpose communication system is growing very fast in all market sectors (bank, distribution, health, industry, aeronautics and defense, building and home automation, etc.). Today enterprise information systems are spread throughout multiple data and computing centers – including smart embedded/mobile devices – geographically dispersed over Internet. Therefore the distributed enterprise is increasingly concerned by the integration of these semi-autonomous entities within a global enterprise information system. The integration objective involves complementary facets. Interoperability between enterprise applications – referred to as EAI (Enterprise Application Integration) or A2A (Application to Application). EAI aims at building new added-value services by combining existing legacy applications. In addition, making enterprise applications communicating with each other is a strong requirement for Business Process Management within the enterprise. Data integration aims at building a business-oriented view of the numerous data flows issues from heterogeneous data sources such as: application status, business process status, usage data from networked equipments and services, etc. This form of integration is often achieved through the use of ETL tools (Extract, Transfer and Load) tot build datawarehouses (or datamarts) which are then exploited by Business intelligence tools to assist decision-makers. Interoperability and cooperation between autonomous systems that are operated independently. This is the case for example in manufacturing to build connections between a process management system and the business information system. In the telecom area, a similar problem arises to establish links between the network management system (i.e. supervision, SLA management) with customer management (billing, CRM, etc.). This form of integration may be considered as hybrid as it combines data integration and application integration. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 3 In most large companies, these various forms of integration coexist and use different technologies. They have in common two strategic objectives: i) decrease the operation cost through the reuse of existing software components and the sharing of distributed resources ; ii) improve the management capabilities through by providing pertinent indicators on all business activities in real time. In addition, companies are facing the key requirement to cooperate with their business partners (customers, providers, sub-contractors, public authorities ...) using Internet facilities. This leads to a new form of loosely-coupled integration, referred to as B2B (Business To Business), which follows different rules due to the fact that communication and synchronization occurs between independent and autonomous organizations. This specific context has led to new application protocols that increase the overall complexity of the integration landscape. Integration technologies: the role of the Enterprise Bus A detailed analysis of integration mechanisms mentioned above shows that they all share a number of principles such as: use of Internet (TCP/IP) and Web protocols (HTTP, XML, etc.), asynchronous communication system based on a MOM (Message-Oriented Middleware), object-oriented programming languages and/or APIs, etc. However these common principles are embedded in incompatible technologies. As a result, we can observe a set of heterogeneous integration facilities that run independently of each other. This leads to a waste of resources, an increase of development and operation costs and a poor quality. Some “big” ISVs are trying to hide this heterogeneity by building a global offering that packages all these integration facilities under a unique commercial banner. These offers are usually very complex to manage because of the heterogeneity issue and also because the packaging imposes constraints that do not allow users to tailor the integration suite to their specific requirements. Finally these packages have in common high costs and a complex pricing strategy. From the pure technical point of view, the demand for a “general purpose” integration system is increasing very fast. From the economical point of view there is also a strong pressure to rationalize and simplify existing infrastructures to reduce operation costs and to increase the ROI. This demand is not satisfied today by existing offerings that are both too expensive and too complex to deploy and to evolve. The solution proposed here is to build the various facets of the integration infrastructure on top of a unique engine that provides the required technical and business properties. This foundation is the Enterprise Bus. From a functional point of view the Enterprise Bus is the integration support for data, applications and equipments in the context of loosely-coupled distributed systems. He provides the foundations for the following key functions: Interoperability between applications both within the enterprise and also with business partners. Support for a large distributed shared information space within the enterprise, Usage data retrieval from any networked resource (equipment, service, application, business process etc.) for remote supervision and control. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 4 Enterprise Bus: functional requirements The following properties are expected from the Enterprise Bus: Asynchronous communications. It is recognized today that asynchronous communication systems are suited for the support of loosely-coupled systems encountered in Internet-based environments (large-scale unreliable networks, disconnected mode, etc.). In addition they provide higher support for reliability, availability, and scalability compared to the communication systems based on the client-server paradigm. Asynchronous communication systems are based on the MOM technology (Message-Oriented Middleware). Openness and interoperability. Several issues are involved. Communication between distributed entities follows widespread standards such as Internet protocols (TCP/IP), and Web protocols (HTTP, XML-SOAP, etc.). The Enterprise Bus is supposed to be “universal” – i.e. available everywhere on a large range of equipments, from the application server to smart internet appliances or mobile devices. This requires the ESB to be as independent as possible from host environments. Unless we agree on a single operating system environment, the only way to achieve this goal is through the use of Java-based environments. Finally the ESB should provide hooks to legacy applications. Flexibility. The ESB is a key building block that should support a wide range of integration services within a heterogeneous distributed environment. In other words the ESB has to be highly flexible and configurable to enable a seamless customization to answer the enterprise requirements and to adapt to the specific constraints of the physical environment. In addition, it should be reminded that the ESB is only a basis to build the integration facilities. Therefore the ESB should be extensible, i.e. provide mechanisms that would allow application services to be developed and deployed seamlessly. To summarize, the ESB is built on top of a MOM, designed to be deployed on large-scale Internet networked environments, that provides an open standard API and that provides configuration and extension capabilities to allow application-oriented developments. The JORAM Solution JORAM (Java Open Reliable Asynchronous Messaging) is a middleware component that provides the required basis for the construction of the targeted ESB. JORAM is an open source software package from the ObjectWeb code base -http://joram.objectweb.org. This component was originally the result of a scientific cooperation between ScalAgent Distributed Technologies, Bull and INRIA. Today the development of JORAM benefits from a large community of developers and users within ObjectWeb. This is a guarantee of durability, quality and evolution. JORAM is a 100% Java implementation of the JMS™ (Java Message Service) specification as defined by Sun Microsystems, Inc. JORAM is compliant with the latest JMS 1.1 specification (that is itself a key part of the J2EE 1.3 specification). Today JORAM is the unique open source JMS implementation available on the market. The latest evolutions of JORAM are concerned with higher availability support (using clustered architectures) and enhanced connectivity. JORAM is now available on lightweight devices. J2ME applications, developed on Java embedded systems, are now able to communicate with business applications using JMS. Through this new functionality JORAM appears today as a universal connector JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 5 between distributed components running on a wide range of equipments from the application server to mobile devices or embedded systems. JORAM also provides a gateway to interoperate with foreign JMS systems. Today JORAM is deployed in numerous operational environments where it is used in two complementary ways: As an independent messaging system between applications running in a Java environment (J2EE to J2ME), As an asynchronous communication component integrated within a J2EE application server. Following this schema JORAM is a key building block of the JOnAS application server also available at ObjectWeb -http://jonas.objectweb.org -. Compliance to JMS and open source are two key properties that make JORAM attractive for building an ESB. We claim that it is not enough. The main asset of JORAM – that provides a key advantage on concurrent implementations, including JMS proprietary products, comes from its internal architecture that benefits from a new generation of message-oriented middleware relying on an agent-based distributed architecture. This cutting-edge technology enables complex applications to be deployed on a large-scale Internet networked environment in an efficient, reliable and flexible way. The goal of this technical white paper is to describe the JORAM design principles that justify its use as an ESB foundation. Section 2 of this paper is a brief overview of the JMS personality provided by JORAM. Section 3 describes its internal architecture and points out the flexibility and extensibility properties. Section 4 briefly describes the technological foundations of JORAM that enable flexibility and extensibility. Finally section 5 explains how JORAM can be used to provide the intended ESB. JORAM: an Overview JORAM is first a messaging component that complies to the JMS specification. This section develops this viewpoint in two steps. In a first stage the principles and limits of the JMS API are briefly reminded. In a further stage the overall structure of JORAM is described. The JMS API in a nutshell JMS (Java Messaging Service) is the specification of a messaging service for Java applications. More precisely JMS describes the API that allows Java programs to communicate through an asynchronous communication system – i.e. sending and receiving messages. A detailed description of the JMS API is beyond the scope of this paper (the reader can refer to the tutorial available online on the JMS Web site). We present here only the JMS elements that are require to the understanding of architecture issues described later on. A JMS application consists of the following elements: A JMS platform (also called JMS provider) that implements the JMS API and that provides a set of control and administrative functions. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 6 The JMS clients are application programs, written in Java, that produce and consume messages following the messaging protocols defined in the JMS API. JMS messages are entities that allow information to be conveyed between JMS clients. Different types of messages are supported in JMS: structured text (e.g. an XML file), binary data, java objects (in a serialized form), etc. Two communication models (called Messaging Domains in JMS) are supported: Point to Point model, based on message Queues. A producer client sends a message to a message Queue where it is stored temporarily. A consumer client may then read the message from the Queue to operate on it. A given message is read only by a single consumer client (this explains the ‘‘point to point’’ designation). The message stays in the Queue until it is read or the message time-to-live expires. The message consumption may be synchronous (explicit call by the consumer client) or asynchronous (call of a pre-defined watch function to be executed at the consumer site). Message consumption is then acknowledged either by the system or by the client application. Multipoint model, based on the Publish/Subscribe paradigm. A producer client publishes a message related to a pre-defined Topic. All clients that have previously subscribed to this topic are notified of the corresponding message. The latest version of the JMS specification (i.e. JMS 1.1.) unifies the handling of the two communication modes at the client level through the introduction of the “Destination” 1 concept to represent both a message Queue or a Topic. This simplifies the API (i.e. the queue and topic functions are syntactically merged) and allows an optimization of communication resources. This unification does not change the semantics of each of the communication models that should be taken into account in programming the JMS client. Finally the JMS specification defines some QoS options, especially for subscriptions (transient or durable) and the guarantee of message delivery through the message persistency property. What JMS does not say and does not do JMS defines a communication protocol between message producers and consumers, but it does not give any hint about the way this protocol should be implemented within the messaging service which stays outside the scope of the specification. Therefore all JMS messaging platforms are proprietary implementations that only conform to the JMS API. A JMS application (i.e. set of JMS clients) that complies to the JMS API is independent of a given JMS messaging service (i.e. portability objective). However interoperability between JMS clients running on two different JMS platforms requires the development of a specific gateway between the two platforms. This function is available on some JMS products. JORAM also provides this feature (see below). The JMS specification is not complete. Some key functions are not described, such as deployment and administration, and are thus subject to proprietary implementations. At the opposite, most of available products (commercial or open source) provide additional functions (not described in the spec) such as hierarchical topics security and high availability features, etc.). On both issues JORAM is not an exception to the general rule. 1 In the rest of the paper the term « destination » will be used to designate indifferently a queue or a topic. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 7 JORAM: an open source JMS platform From the above description JORAM is a JMS platform. As any other JMS platform JORAM is structured in two parts: the JORAM server that manages the JMS abstractions (Queues, Topics, ConnectionFactory, etc.) and the JORAM client that is bound to the JMS client application. As we will see later on in the description of the JORAM architecture, the JORAM server can be implemented as a central service or as a set of cooperating distributed services. It should be noted here that the ability to deploy a JMS platform as a distributed system with variable QoS parameters is a key issue when comparing various JMS platforms. Communication between a JORAM client and a JORAM server is relying on TCP/IP. In addition HTTP/SOAP may also be used for lightweight JMS clients developed in the J2ME environment (this issue is addressed later on). Communication between JORAM servers is achieved through various protocols according to application requirements: TCP/IP, HTTP, SOAP/XML, and secure communication through SSL. Usually, clients and servers run on different machines. However they also can be hosted on the same machine and run in different processes or share the same process. In this case communications are optimized. To summarize, JORAM is an open source JMS platform. The implementation of this platform relies on a MOM that exploits an agent-based distributed technology described in the section 4. JORAM: a distributed configurable architecture This section details the key aspects of the JORAM platform architecture that enable its unrivalled flexibility and scalability properties. The main characteristic of JORAM is its distributed configurable architecture. Basically JORAM has adopted a “snowflake” architecture, i.e. a JORAM platform is composed of a set of JORAM servers interconnected by a message bus that offers various communication protocols. Each server can manage a variable number of JMS clients. The geographical implantation of the servers as well as the distribution of the clients on the various servers is under the responsibility of the architect and is then enforced by the platform administrator. This choice is a first level of configuration. A second level refers to the placement of communication objects (Queues and Topics). We will see later on that these design choices have a strong impact on the application performance and scalability. A third level of configuration consists in defining the QoS parameters (communication protocol, persistency, security, etc.). The choice of a given mechanism is always a tradeoff between the expected QoS level and the cost of the solution. The overall architecture of a JORAM platform is depicted in Figure 1. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 8 Figure 1. JORAM Platform Architecture Logical Architecture The communication path between two JMS clients is illustrated in Figure 2 (for the particular case of a Point to Point communication model). This schema points out the data and control flows involved in this exchange. Connection, Session and Sender (resp. Receiver) are temporary (Java) JMS objects created by a JORAM client when a connection is established with the server1. On the server each JMS client is represented by a Proxy object. This persistent proxy object is created by the server whenever a new user is registered. A proxy object implements two main functions: The management of communications between the server and the client. Message delivery to the destination (here a message Queue). To achieve this goal, the Proxy object implements a Store & Forward function, which plays a key role in communication reliability and message persistency (this issue is detailed below). 1 We do not detail the role and the use of these JMS objects. They are available in the JMS specs. Q1 Ta Q2 Tb Q3 M O M MOM M O M Distributed MOM TCP/IP JORAM Server JORAM Server JORAM Server Q T Queue Topic JMS Client JMS Client JMS Client JORAM Client JMS Client JORAM Client JORAM Client JORAM CLient JMS Client JORAM Client JMS Client JORAM Client JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 9 Figure 2. Point to Point communication Distribution is not depicted in this functional schema (i.e. in a given implementation the Queue and Proxy objects may run on a set of cooperating servers). The communication between a producer client and a consumer client in achieved through the following steps (described in Figure 2 by the red arrows). When a Send operation occurs on the producer client site, a JMS message is sent to the corresponding Proxy object (noted Proxy-P) within the server (plain arrow 1). The Proxy-P object encapsulates the JMS message into a MOM message to be transported by the MOM to the destination site (i.e. the node of the Queue object). The MOM message is saved on a local persistent store. An acknowledgment is returned to the JMS client (dotted arrow 2). From the producer client the send operation is over and it goes on its execution asynchronously with the message delivery. The MOM message is delivered to the Queue object (arrow 3). This may require a communication between two servers if the Proxy-P and the Queue Objects are not handled by the same server. The MOM message is saved on a local persistent store that implements the message queue. When a Receive operation occurs at the consumer client site a control message is sent to the corresponding proxy object (noted Proxy-C) in the server which forwards it to the Queue object. Similarly with step 2, this may require a communication between two servers if Proxy-C and Queue objects are not settled on the same server. The MOM message is retrieved from the Queue object and sent to the Proxy-C object which, in turn, extracts the JMS message from its envelop and returns the JMS message to the consumer client. This set of operations is described by the set of plain arrows 4. At this stage the Receive operation is over. An acknowledgment is returned by the JORAM client to the Queue object in order to free the resources on the server that manages the Queue object (dotted arrows 5). The Ack may be generated by the JMS client or by the JORAM client. In step N° 2 the MOM message is stored before being delivered to the Queue object. This allows the delivery operation to be re-executed in case of failure during the delivery operation (step 3). This feature, referred to as the Store and Forward function, guarantees the message delivery at the JORAM level. It should be noted that we do not describe here exchanges that occurred at the MOM level. Figure 3 depicts the control and data flows for the Publish/Subscribe communication model. Producer Client Consumer Client Joram Server Proxy-P Proxy-C Queue Send JMS Message Receive Receiver Session Connection Sender Session Connection MOM Message MOM Message JMS Message Joram Client Joram Client MOM Message 1 2 4 5 3 4 JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 10 Figure 3. Publish/Subscribe Communication Steps 1 and 2 are similar to those of the point to point communication model. In step 3, the message built by the producer Proxy object (Proxy-P) is delivered directly to the whole set of consumer proxy objects (Proxy-C) where they are stored. This is a key difference with the prior schema as the Topic object is not a final destination but merely a switch towards the set of consumer proxy objects. The consumer operation is implemented by an exchange between the client and the Proxy-C object (plain arrows 4). The acknowledgment is then returned to the Proxy-C object (dotted arrow 5). BY comparison with the preceding schema it can be noted that the consuming dialog is restricted to exchanges between the client and its proxy object. No dialog occurs with the Topic object itself. The next two following sections describe how this logical schema is implemented in a centralized architecture (i.e. a single JORAM server) in a first step, and in a distributed architecture in a further step. Centralized Architecture In this configuration, illustrated in Figure 41, all JMS clients are connected to a single JORAM server that manages all Destination and Proxy objects. The communication protocol is simplified as all communicating objects are located on the same server. Administration and operation are also greatly simplified. This centralized solution has two major drawbacks: lack of availability and lack of scalability. The server is a single-point failure that prevents the whole system to run if a failure occurs on the server side. Computing and storage resources are all concentrated on a single machine. This may result in serious 1 In this figure, the data flow for the message production are described with plain arrows, while the data flow for the message consumption are described by dotted arrows. Acknowledgments are not represented. Producer Client Consumer Client Joram Server Proxy-P Proxy-C Topic Send JMS Message Receive Receiver Session Connection Sender Session Connection MOM Message MOM Message JMS Message Joram Client Client Joram MOM Message 1 2 4 5 3 3JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 11 performance bottlenecks when the load increases (e.g. number of clients, number and size of messages, etc.). A potential solution to these problems is the use of a clustered architecture to support the single server. This solution is expensive and induces an increasing complexity from the administration point of view. The use of clustered architectures is addressed later on in this paper. Figure 4. Centralized Architecture Distributed Architecture In a distributed architecture approach several instances of the JORAM server cooperate, each of them being in charge of a set of JMS clients. This architecture is illustrated in Figure 5 for a scenario composed of three physical servers (to simplify the figure only one JMS client has been represented at each location) 1. The JMS message produced by the Client-1 JMS client to Q3 message queue is sent to the Px1 proxy object on server S1. The JMS message is embedded within a MOM message. The Store and Forward function save the MOM message locally before forwarding it to the S3 server where queue Q3 is located. When stored in Q3 the message can then be retrieved by a JMS client connected to server S3. In the same way, a message produced by the Client-2 JMS client for the Q1 message queue is delivered to S1 through the Px2 proxy object in server S2. The physical communication between the servers is achieved by the underlying distributed MOM that guarantees the delivery of messages at the MOM level despite transient network and server failures. 1 This particular configuration is taken as an example. The placement of Queues and Topics is achieved by the system administrator. Producer Client Send Receive Proxy-P Proxy-C Queue JMS Message MOM Message MOM Message JMS MessageS Joram Client Joram Client JORAM Server Consumer Client JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 12 Figure 5. Distributed Architecture High Availability and Reliability Several mechanisms have been introduced to increase the JORAM platform availability and reliability. Some of these mechanisms have also a positive impact on the performance. A “clustered Topic” is a Topic object that is replicated on several servers. This feature answers two needs: Availability: a server failure has a consequence only for the JMS clients connected to that server. Other clients have still an access to the copies of the Topic. Load balancing: JMS clients can be distributed on various servers so that the load generated by the It should be noted that the use of “clustered Topic” is not restricted to physical clustered architectures (i.e. a set of tightly-coupled servers located in the same room). “Clustered Topics” may also be used in configurations distributed over Internet. The use of clustered Topics does not impact the application programming (i.e there is no change at the JMS client level) but requires careful attention at the platform configuration level to be efficient. The principle of the “clustered queue” is similar. Several instances of the same Queue object may be located on independent servers. The semantic of the Point to Point communication protocol is not modifies (i.e. a given message is retrieved only by one consumer JMS client). This feature provides higher availability and higher efficiency for message queues that are highly used by JMS client (e.g. high message traffic and high level of sharing) without any impact on the application development (i.e code of JMS clients). The development of a HA JORAM Platform (HA for High Availability) targeted at physical clustered architectures is under way. This is achieved through the management of replicated Queue and Topic objects on one hand, and replicated connections between JMS clients and the clustered JORAM server on the other hand. The objective is twofold: better performance through load balancing facilities and high availability for JMS clients. This new version of the JORAM platform is the right answer to critical applications. Client-1 Joram Client Client-3 Joram Client Px1 Px3 Distributed MOM Q1 Q3 Client-2 Joram Client Px2 Q2 Send Receive MOM Message MOM Message MOM Message JMS Message JMS Message Send Receive S1 S2 S3 JOram Server Joram Server Joram Server JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 13 To end up with this issue, it should be reminded that the middleware technology that supports the JORAM platform guarantees end to end message delivery in the context of unreliable (e.g. Internetbassed network connections and disconnected mode. Openness and Interoperability We have already mentioned above the use of Internet and Web standards. The various connectivity elements are briefly summarized in Figure 6. The availability of a lightweight client (named K-JORAM) extends the use of the JORAM platform towards new application domains involving mobile terminals and equipments with limited resources (e.g. PDA, SmartPhone, embedded Java systems)1. J2ME applications, running on these devices, can now use JORAM to cooperate with JMS applications via Internet. In addition, the use of the SOAP protocol between a JORAM client and a JORAM server, allow non-Java clients to benefit from the JORAM messaging services. Thanks to this new functionality, JORAM becomes the universal asynchronous connector between distributed components, Java or not, running on a wide range of machines from the smart Internet device to the application server. Finally the integration of JORAM and Apache/Axis allows JORAM to be used also as a reliable communication platform for XML messaging and asynchronous Web Services. Figure 6. Connectivity elements in JORAM JORAM also provide a generic gateway to interoperate with foreign JMS platforms. This feature allows a JMS client (i.e. without client code modification) to transparently communicate with a JMS client running on a different JMS platform. With this facility a foreign Destination (Queue or Topic) can be managed as 1 J2ME imposes some constraints that do not allow the full JMS API to be implemented. Therefore the lightweight JORAM client has access only to a subset of the JMS API. JORAM Server Queues, topics MOM TCP/IP SSL HTT SOAP Proxy client TCP Proxy client SOAP « lightweight » Client K-JORAM K-Soap, K-Xml Mobile device (PDA, SmartPhone, . .) JMS Client JORAM CLient JMS API Java Objects TCP/IP Soap/XML messages HTTP JMS Client JORAM Client JMS API JORAM Server queues, Topics MOM TCP/IP SSL HTT SOAP Proxy TCP MOM Message Proxy SOAP Apache Axis JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 14 a JORAM Destination. The use of this feature requires particular administration operations to make the shared Destination visible from the two platforms but does not require any modification of the JMS client code. Synthesis: Build the JORAM platform that fits your own requirements Building a JMS platform tailored for a given application context is a difficult task as the resulting architecture is based on complex tradeoffs between numerous evaluation criteria: performance, availability, scalability, flexibility and evolution, security, development and operation costs, etc. Therefore the role of the architect is crucial. Based on a set of application requirements he has to take into consideration the evaluation criteria mentioned above and to give them a specific weight in order to design the architecture that better fits the requirements. This work is achievable if and only if the messaging system provides the configuration capabilities that enable such a design process. JORAM answers this need through a combination of configuration and tuning capabilities available at various levels: Overall organization of JMS servers and clients, and placement of JMS objects, Server sizing (computing power, storage, communication facilities), System extension to support scalability, Communication protocols, Qos parameters (e.g. persistency, security), Level of availability through clustering and replication Today very few platforms provide a level of flexibility comparable to that of JORAM. This is obviously a major advantage compared to concurrent products. JORAM: Technological Foundations JORAM directly benefits from the cutting-edge technology used to implement the platform. This technology and the way it is used in JORAM are briefly presented in this section. Distributed Agents The experience drawn from many years of research in distributed computing has led us to adopt an approach based on distributed intelligence for the design of distributed applications. Bring data handling near data sources allows the computing load to be balanced between the computing resources available on the network, and enables bandwidth reduction as only pertinent pre-handled data is transported on the JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 15 network. This basic design principle is implemented through a programming model, a run-time environment and a set of tools described below: Agent programming model Agents are distributed Java objects that communicate through message passing. An agent body is defined by a Java class that inherits from a pre-defined class “Agent” (i.e. describing the generic behavior of an Agent object). Agents conform to an “Event-React” programming model. An event is the notification of a typed message that turns into the execution of a method of the object class (the “reaction”). This method may in turn generate new events handled by other agents. A “Reaction” is atomic (i.e. “all or nothing” property) and the state of an agent is persistent. Agent Server Agents are lightweight entities that share a hosting environment called an “Agent server”, which is depicted in Figure 7. Agent servers are configurable run-time structures that implement various behavior policies for agents (e.g. atomicity and persistency ). Figure 7. Structure of an Agent Server At the heart of the agent server an engine controls the agent execution flow. This single flow runs as a loop that executes the reaction which corresponds to the next incoming event notification. The control code of the persistency and atomicity functions is also executed if specified. Specific control flows are associated with communication management within the channel sub-system. The cooperation between the server channels implements the distributed message bus. This bus provides the communication path between agents, running in the same server or hosted by different servers (in the same machine or in different machines). Each local bus is composed of two parts: a channel and a set of network components. The channel implements the interface between the agent engine and the network components. A network component implements a given communication protocol between two agent servers (e.g. there are network components for TCP/IP, HTTP, SOAP, SSL, etc.). In addition it is possible to enhance a network component with additional modules to extend its properties if needed (e.g. security, causal ordering, etc.). A set of basic management operations are available to create and configure agent servers in a Internetbaase network environment. Agent React SendT Channel Agent TCP-IP HTTP SSL Agent Engine Agent Server Agent React SendT Channel Agent TCP-IP HTTP SSL Agent Engine Agent Server Distributed MOM JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 16 JORAM: an agent-based JMS Platform The agent-based technology described above has been widely and experimented to build and deploy distributed systems in various application domains. The JORAM platform is an example of such a design. A JORAM server, i.e. the platform component that manages JMS objects is an agent server structured as follows: Queues and Topics are persistent agents. On each server an agent « ConnectionFactory » handles the connections with the JMS clients. A persistent Proxy agent is created for each user registered by the server. This agent manages the communication for the JMS clients (producer and consumer) associated to this user. This agent implements the Store and Forward function. Durable subscriptions and message persistency are achieved directly by making corresponding agents persistent. This facility may be removed if it is not required by the application, thus saving resources and increasing performance. In addition, guarantee of message delivery between two agents is achieved by the MOM. The flexibility of the overall JORAM platform benefits directly from the configuration capability of agent servers, especially for what concern: The choice of communication protocols (TCP/IP, HTTP, SOAP, etc.) and QoS parameters (persistency, security). The way a snowflake architecture is built from a choice of server implantation to meet the application requirements. The placement of Destination and Proxy objects (agents). These configuration parameters may evolve later on to adapt the structure of a JORAM platform to evolving requirements (new server, migration of Queues and Topic objects, change of security parameters, etc.). JORAM: a component of an integration platform JORAM is an element of a broader platform that aims at providing the technical foundations for the development and deployment of distributed applications in the Java/Internet world. The JORAM design allows application programmers to benefit both from the JMS API and the agentbaase programming model. Both APIs have their own interest. JMS provides the conformity to a wellestabblishe international standard for implementing asynchronous communication between Java applications. However the use of the JMS API is complex and error-prone. At the opposite the agent model is much simpler and provides a gain of productivity when developing and deploying applications. Therefore mixing these two ways of programming allows a strong reduction of the Time to market for new services/applications while keeping advantage of a pure JMS system to communicate with legacy and foreign applications. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 17 It should be noted that the global platform “agents+JORAM” is available as a packaged open source component, as illustrated in Figure 8 that describes the overall ScalAgent technical offering. Figure 8. ScalAgent Technology – an Overview On top of the “agent+JMS” platform, ScalAgent has developed a complementary technological offering that aims at facing the challenges of distributed computing for large-scale Internet systems. This offering is relying on two major emergent technologies: components and software architectures. The interest of architects and application developers for “components” is growing very fast (e.g. EJB and .NET). Components can be viewed as coarse-grained application building blocks that encourage software reusability and that ease the construction of complex applications by simply assembling components (i.e. without programming skills). This “programming in the large” vision aims at shortening the application life-cycle but is not actually exploited in EJB or .NET component models. At the opposite, recent approaches based on software architecture and ADL (Architecture Description Languages) provide the foundations for this new way of programming. An ADL is a declarative language (syntactically close to an IDL) that allows the components, their interfaces and their functional dependencies to be specified. The ADL description is compiled to provide a logical view of what the distributed application is. This reference model is the used to automate and control operations such as deployment, monitoring, reconfiguration, etc. This innovative approach has been adopted by ScalAgent to develop an integration framework for enterprise data (called the Mediation Framework). This offering covers the upper layers of Figure 8 and comprises the following elements: A set of pre-defined components for collecting, handling and delivering usage data generated by networked equipments and services A set of user-friendly graphical tools to build, deploy and monitor integration solutions by customizing and assembling these basic components. These tools rely on the ADL approach described above. Java Virtual Machine : J2EE, J2SE, J2ME, JavaCard Scalagent MOM reliable asynchronous communication Agent Management Services creation, persistency, atomicity, . ., . . JORAM Platform Queues, Topics ADL Tools configuration, deployment, control, . . Open-ource Agent API JMS API Monitoring Mediation Solution Legacy applications/back-office Run -T ime Mediation Framework Mediation components reusable /configurable Component Management Services deployment, monitoring, . . . JMS applications JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 18 A deployment service to automate the deployment process of a global distributed integration solution made of numerous components to be installed at the right place on the network. Monitoring and reconfiguration facilities to allow a seamless evolution of the integration solution. This environment shares with JORAM the technological foundations for the management of distributed data and control. JORAM is also used as a standard communication channel between the integration infrastructure and legacy/back-office applications and J2EE application servers that implement the business logic for handling usage data collected and delivered by the mediation infrastructure. Conclusion The decentralized Enterprise leads to rapidly growing information flows to answer various and evolving business requirements. So far this variety of usages has been handled by different communication infrastructures that were simply coexisting but not interoperating. This situation has led to complex and expensive integration solutions. Mastering the development and operation costs requires a rationalization of these integration infrastructures. A first step consists in structuring the multiple interaction models around a common communication channel that we call the Enterprise Bus. The required solution must be efficient to face the challenge of increasing data flows and the growing need for delivering pertinent information in real-time to speed up the decision process. The required solution must be also open and flexible to rapidly adapt to evolving customer requirements and changing run time conditions (machines, networks, mobile users, etc.). Over the past decade, several initiatives have been launched to define (and also to standardize) an Enterprise Bus. Three main action lines may be distinguished. The first one is based on CORBA and further object services such as the notification service to support asynchronous communications. The second way is concerned by asynchronous messaging systems and, more recently, the support of the JMS API (the first de facto standard in asynchronous computing). The third initiative is related to the success of XML as a general purpose and universal data representation model. Recent activities on Web Services belong to this category. In fact this evolution is concerned more by the semantic of data exchanges rather than the communication and synchronization modes of cooperating entities. CORBA-oriented systems, based on the client-server paradigm, are not well adapted to large-scale distributed systems encountered on Internet. At the opposite, asynchronous messaging systems bring better answers to failure management, disconnected mode and scalability issues. This partly explains the decreasing interest for CORBA-based systems and the growing use of MOM systems. For this class of middleware Java is a key advantage as it provides a certain level of independence between the middleware layer and hosting systems. Finally, the JMS API, although it is still suffering from limitations, is a complementary asset as it introduces a standard way of programming on top of a MOM. However one can regret the lack of interoperability standard at the MOM level as defined for example in the CORBA approach through the IIOP protocol. Finally it should be noted that the choice of a JMS messaging systems is complementary from the choice of XML as a data representation model. From this viewpoint, recent initiatives such as JAXM to define a complete XML messaging system does not appear to be relevant, except for what concerns the definition of messaging profiles adapted to specific application domains. The choice of a JMS messaging system as a basis for an ESB is a first step. The further step deals with the choice of a particular implementation to meet the objectives mentioned above. “Big” ISVs are offering complete packaged integration platforms that embed a messaging system, an application server, JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 19 business process management tools, and a variety of complementary integration functions. Deploying these platforms is both complex and expensive (licenses, development and operation costs, etc.). Their use is usually constrained as the architecture of the packaged product does not allow the level of flexibility required (e.g. for the management of mobile users and devices, for the rapid deployment of new services or for the collection and delivery of new business indicators). An alternative approach consists in adopting a flexible and extensible messaging system that provides the hooks to build customized integration solutions that can then evolve accordingly. This is the approach followed by ScalAgent Distributed Technologies, based on the open source JORAM component. This choice is motivated by the following key advantages: Openness and connectivity through the conformity to main Web standards: JMS, J2EE, HTTP, SOAP-XML, etc. The JORAM connectivity spreads up to mobile equipments and embedded devices that can be now considered as full elements of the distributed information system. An unrivalled configurability that enables fine-grained sizing of the ESB to answer precisely the enterprise requirements. The bus may later on be reconfigured easily to adapt to the evolving enterprise. In addition the configuration of QoS parameters allows a significant reduction of operation costs as it is possible to pay only for the properties actually needed. High efficiency, reliability and availability. The truly distributed architecture enables an optimization of computing and networking resources. The HA approach, based on replicated resources, provides support for critical applications that require a high level of availability. Cost-effective solution due to the open source approach (no license fees) and the seamless deployment and operation procedures (e.g. a two-day training session allows users to build and deploy their own JORAM system). JORAM users benefit from a free level of support provided by the open source community. In addition ScalAgent Distributed Technologies provides professional support with QoS guaranty at low cost. Product durability and quality guaranteed by the growing open source community (both developers and users). Numerous industrial and academic research teams are today contributing to the development and experimentation of advanced facilities that will be integrated within the product when they are available. Based on these properties, JORAM appears to be a real alternative to proprietary messaging JMS products and integration platforms. Services available besides the JMS API (i.e. the agent API and the mediation components framework) provide an adequate technological basis for the construction of data integration solutions (e.g. real time ETL), workflows systems, Business Process Management and Business Activity Management systems. The cost of developing and deploying these solutions would be much lower than using a packaged product with a better adequacy to the targeted usage. JORAM: The Open Source Enterprise Service Bus © 2004 SCALAGENT DISTRIBUTED TECHNOLOGIES S.A. 20 About ScalAgent Distributed Technologies ScalAgent Distributed Technologies is an emerging start-up company, spin-off from Bull and INRIA, which aims at developing advanced mediation solutions for large-scale Internet-based distributed applications in the Internet/Java world. For many years, our technical and business professionals have gained considerable experience in the design and development of advanced prototypes in distributed object-oriented systems for the telecom and industry domains. This expertise and the technology building blocks developed so far are exploited to help our customers build the mediation solutions that meet their own requirements. For his customers, ScalAgent Distributed Technologies develops mediation solutions tailored for their business requirements and for the constraints of large-scale networked infrastructures. These mediation services are designed to be easily integrated within global management solutions built by integrators for device manufacturers and for service-application-content providers. To learn more, please visit us at www.scalagent.com .