The Edge of Origin Adam Chase Kalyana Prattipati Dave Quinn Martin Sweitzer May 06, 1998 Version 1.0 Introduction In today’s fast pace environment any down time is expensive in both time and money. In addition a service that is slow or lagged out is also something that costs resources. Problems of low availability and the slow or lagged service in modern networked games are solved by the Extreme Distributed Gaming Environment (EDGE). The EDGE is a Java-based distributed entertainment architecture which allows multi-player games to be partitioned over multiple servers across the Internet. In addition, it is fault tolerant, highly available, provides a redundancy mechanism for backup purposes, and maintains state via checkpointing. The EDGE allows administrators to keep their game up and running long past the competition! Motivation The motivation for creating the EDGE is that there are many persistent worlds in existence on the Internet, but they are all running on single servers. When the server crashes the people connected to it will be unable to play for some time. In addition to crashes, virtual worlds often have issues with performance. Since most worlds are quite large and host upwards of 200 players at once, they can easily saturate a T1 line with all the data being passed around. Not only is there a bandwidth problem but the server is usually lagged and slow due to all of the computations it must do. EDGE is a solution for making these mega-worlds more playable. Usage The EDGE is a framework which allows games to be partitioned across multiple computers such that downtime is greatly minimized and performance is high. To develop the EDGE system a developer should be knowledgeable in CORBA and Java in order to fully understand the mechanics behind the games. To prove that the EDGE actually works, a trivial MUD was developed using it. Below, the steps needed to implement the MUD using the EDGE system are briefly explained. World Creation A world was created from scratch. Rooms were designed with detailed descriptions and then linked to other rooms in an orderly fashion. Linking CORBA to the World Code The administrator needs to know how to look at a CORBA IDL file in order to create the CORBA-ized APIs that allow the distributed magic to occur. After merging the CORBA code into the code base the administrator can basically just forget that it is there! World Partitioning Administrators must now partition of the world among the available servers. In the MUDS that exist today there are usually certain areas that have a greater than average use. These areas should be placed on a separate server to give them the computational power they need to provide a lag free environment. Once areas have been partitioned the connections between areas need to be modified from being on a single machine to that of “pointing” to the remote host. This simply involves changing the exit of the room to point to the remote server. For example, the demo world was transformed from the single server model to the distributed in about 10 minutes of work. Database The administrators need to set up the EDGE databases. This may require some work converting the existing format to a JDBC compliant database. Many of the MUDs today use a flat file format which should be easily converted into tables in a database. Client Side The client must have the CORBA code linked in. We made a trivial TTY and a GUI client for accessing the EDGE world. One of the designers was able to develop the GUI client totally 1 independent from the main EDGE development and was easily able to link in his work . An example GUI is in Appendix C. Summary The EDGE system is meant to be used by the administrators of a game world. The administrators will place their gaming environment “on top” of the EDGE. EDGE will seamlessly allow the administrator’s game to be run on multiple servers such that if one of the servers crashes, the game will not be interrupted. In addition, as the game has been partitioned among multiple computers the response time will be faster than if the entire game was on a single server. Architecture The major components of the EDGE system are the Client, the Game, the Server, the Databases, and the player. The relationships between them are shown in the following figure: 1 The actual linking of the CORBA code into the GUI was trivial. What was problematic was getting JFC to work with JDKs previous to 1.2beta3. The Client component is used to get input from the end user. Users first log in and register with a server. If that is successful the user can now interact with a world by manipulating the user interface. RPCs are called by the UI, executed by the Server, and the results are then displayed to the user. In addition, the Client also figures out what country the player was on when he last played, and then returns them to that area of the game. A Game is a collection of rooms which represent some area or set of areas of the MUD world. They are hosted primarily by one server, but is also backed up on another. In addition, there are “virtual” rooms that, when accessed by players, allow travel between different areas of the MUD. These “virtual” rooms are really just pointers to games on other servers to which the Client connection gets passed. The Server of the EDGE is the representation of the actual machines which are running the mud. Servers are currently only capable of hosting two games, one primary and one backup. Future plans involve a dynamic solution which will allow any number of games to run on any of the servers in the system. For an example of a server topology, see the diagram in Appendix B. The Databases are the persistent storage system of the EDGE. The typical game database holds information about a specific world, and the player information for the players in that world. Each primary database has one backup, which takes over if the primary goes down. The databases get updated by regular checkpointing. The process involves a thread goes through all the objects on the server and checking the timestamps. If there have been changes, the appropriate data will be written to the database which minimizes the data loss if a database crashes. The player objects exist on every server. The server creates one when a player logs in at the beginning of a session by loading all of the data from the database into the player object. A reference to that object then gets passed to the client. Whenever a player switches to a new world, that player’s object gets written into the databases for that object. Design and Implementation The EDGE system was developed using Java, CORBA for Java, mSQL, the Java Development Kit, and Visigenic’s Visibroker. Java was used as the development language because of its ease of portability between different platforms and the desire of team members to experiment with a new language. The choice of Java matched several of the goals set in the design stage of the EDGE. First, it was necessary that the system be portable to many platforms. Second, it was desired that the code base be clean and subject to the principles of Object Oriented Design. Third, the EDGE required a system with a readily available and easily understood interface with CORBA. Since Java matched or exceeded expectations on the given goals, it was the clear choice. CORBA is used to provide an interface between the servers, and the clients. The motivating factor behind the use of CORBA is that it adds levels of abstraction between the developer and the network sockets. This reduced the amount and the complexity of the code that needed to be written. CORBA was chosen over RMI because it was shown in Client/Server Programming with JAVA and CORBA, that CORBA is actually faster than RMI for passing references between different objects, a frequent occurrence in the EDGE. CORBA is also better than RMI in that in CORBA the programmer does not have to worry about the packing and unpacking of parameters to method calls. Another advantage to using CORBA is that it allows multiple versions of clients to be used at the same time without having to change any of the server code via well defined APIs. This is very useful for rapid development and testing of clients because multiple developers can be working and testing a client simultaneously on the same servers. These APIs are defined in the Interface Definition Language, or IDL. The interfaces for the EDGE system can be found in the Edge.idl file. Once the interface was written, the CORBA tool, idl2java was used to generate the stubs and skeletons for the interfaces to each of our objects, these can be found in the directory TheGame. TheGame directory is a package of all of the CORBA-generated Java files, none of these files were ever edited by developers on the team. The main implementation issues were how to have clients communicate between each other, how to make sure that when a server crashed the clients all go to the correct backup, and how to assign names to the different players. There were two ideas for communication between the clients. The first was that the clients talk directly to each other. This idea was rejected because of the worst case scenario where clients would crash because they had so many open connections to other clients. The second idea was that a client send the message to the server, along with the recipient, and then have the server pass along the message. This plan was chosen because it allowed for a more lightweight client and allows the server to control how much traffic is sent to a client at any given time. Making sure that the clients all got to the backup server when a primary went down was a tricky issue. It was decided that the servers would notify the clients of their backup when the client first registered with the server. If a server is down the client will get an error passed to it, and as a result it will connect to the backup. A resulting issue of this was how to ensure that the clients went back to the primary when it came back online. There were two ideas on how to do this. The first one was to just force all of the clients to switch back to the primary server once it came back up. This was rejected on the basis that it would not be a good idea to transfer too many players en masse. The method that was used is that the clients switch back to the primary server, once it is back up, as soon they try to execute a command. So the client sees that the primary is backup by a flag set in the backup, it then switches back to the primary, where it then executes the command. The issue with player names is that no two players can have the same name. So every time a new player tries to join a game, that name has to be checked to make sure that it is not already in use. One way of doing this is to have a Distributed Database, but it was decided that this implementation would push the project over the deadline. Another idea was to have each server try to gain a lock on all of the other servers every time a new player joined the game. The idea that was used is a name server with a backup because it was quick and reasonably effective. So every time a new player joins, the server locks the name server, checks to see if the name is valid, if it is, it adds the name to the list and then unlocks the name server. The inherent flaw with this design is that there are only two points of failure for adding new players. This solution was deemed acceptable, because the name servers receive little usage. Also, even if both the primary and the backup name servers both go down, the system will continue as before, except that no new players can join while the name servers are down. In a newer release the problem of naming would be better solved by incorporating a simple name transaction or creating a true Distributed Database for the names. The EDGE system uses databases to store the player information, and information about the rooms within the worlds that players are in. The DBMS that is used is mSQL. This was chosen because both it and it’s documentation were readily available and because it works well with Unix and Windows. The databases are kept up to date via checkpoints which occur at predefined periods. This ensures that in case the game crashes the player loses at most n seconds worth of game data, where n is the amount of time specified for checkpointing. Another implementation issue was how to have a client move between games on different servers. In the database, there are “virtual rooms” which are rooms with negative identifier numbers. Their description is set to be the name of the remote server and another field to correspond to which room to go to on that remote server. The server then detects this case, clones the Player object (the object that represents a player), and then passes the object to the Client, which then binds to the new server. Discussion Overall, the EDGE System is a major success. The designers were able to implement an extreme distributed gaming environment with all of the features that were planned in the requirements phase. The system is a primary-backup system with checkpointing, and it allows for the MUD to be partitioned transparently across multiple servers. The graphical user interface was completed but there was trouble in integrating it into the rest of the system because of compatibility problems between the new beta version of Java and the version of Visibroker (i.e. it was using Java 1.1.3 and 1.1.5) that was being used. Another problem that could arise if the system got a large amount of usage is that the servers keep CORBA references to each of the other servers, this could lead to a large number of socket connections being maintained, which will eventually be a scalability problem. Also, it is not certain how CORBA will perform on a WAN: there could be some lag issues. The architecture that was decided on is nonetheless very sound and a number of features that could be added would make the EDGE System the premier gaming environment of today. The design decisions to use Java, Visibroker, mSQL, and Swing (for UI purposes) are definitely debatable. Java, it seems, was a good decision, just based on the portability freedom it adds. Visibroker and Swing don’t seem to get along (or at least the versions that were used). mSQL was a good database to use, but it is commercial so a “free” product which does the same thing needs to be found. An alternative would be mySQL which we definitely be used in the next version of EDGE. Visibroker is also a commercial product, an alternative might be Sun’s Joe, which is a free ORB. CORBA was useful in that it gave some ease of programming (sockets are hard), but had a price of needing high bandwidth and having questionable performance on WANs. Swing looked cool, but compatibility problems made it painful to use. In addition to the compatibility problems there is a lot of the same code in swing that is written time and times again (i.e. each component needs a listener). After gaining the experience of creating the EDGE, the programmers definitely would do things differently. Regardless of that, the solidity of the architecture is indisputable. The project itself was interesting and enjoyable to work on, and the development of it required learning in Software Engineering and Distributed Systems. Summary During the course of the semester the EDGE system was created which allows multiplayer games to be partitioned over multiple servers across the Internet. In addition, it is fault tolerant, provides a redundancy mechanism for backup purposes, is highly available, and shares state via checkpointing. A persistent world administrator who uses the EDGE will strengthen their world by allowing the players to continuously exist in the world and on servers that are responsive. The designers enjoyed the experience of creating a working Distributed System and learned a lot about Java, JDBC, CORBA, and distributed bug hunting. The entire EDGE team is hoping to continue work on the EDGE system and has started designing an even sharper EDGE. Bibliography Client/Server Programming with JAVA and CORBA 1 Edition, by Robert Orfali and Dan Harkey nd Client/Server Programming with JAVA and CORBA 2 Edition, by Robert Orfali and Dan Harkey Java in a Nutshell, by David Flanagan st Appendix A: Code Summary Checkpoint.java Objectdesc.java TheServerImpl.java ClientControlImpl.java Player.java UserInputThread.java Designer.java Room.java gamedb.java Game.java ServerInputThread.java namedb.java TheClient.java Obj.java TheServer.java TheGame/ Has the code which checkpoints the database every 10 minutes Allows for a description of objects. Implements the interface to the server object Implements the interface to the client object Implement the interface to the player object Gets the input from the player A standalone program which can be used to setup databases Implements the interface to the room object The class which accesses the database The class which represents the actual game Starts up the client interface The class which accesses the name databses Starts up the client A class which represents a generic game object Starts up the server CORBA generated stubs and skeletons Appendix B: Server Topology This diagram shows how each server of the EDGE supports a primary (light shade) and backup (dark shade) game. The arrows are representative of data flow to the backup database, not connections between servers. Appendix C: Example User Interface This example EDGE user interface shows off some of the features of the EDGE client API. Specifically, the multimedia capabilities, as shown by the image displayer, and an example of API calls retrieving useful world infor, as shown by the character window. This UI also supports customizable buttons which will execute user-defined macros.