PKTown: A Peer-to-Peer Middleware to Support IPTV and Multiplayer Online Games Hai Jin1, Hong Yao1, 2, Xiaofei Liao1, Sirui Yang1, Wei Liu1, Yong Jia1 Cluster and Grid Computing Lab.1 School of Computer Science2 Huazhong University of Science and Technology China University of Geosciences Wuhan, 430074, China Wuhan, 430074, China firstname.lastname@example.org email@example.com ABSTRACT which suits for the efforts to lower down the service cost for Mul- Peer-to-Peer (P2P) can support service for lots of end users with tiplayer Online Games (MOG) and Massively Multiplayer Online little hardware investment, which suits for the efforts to lower Games (MMOG). Some interesting works are reported recently, down the service cost for Multiplayer Online Games (MOG). In which focus on the problems when replacing the C/S architecture this paper, we introduce PKTown, a P2P middleware inserted into with P2P in MMOG design, including game consistency , Star Craft and the network layer. PKTown captures all packets cheat prevention , etc. These issues are new and further stud- generated by Star Craft. Application Layer Multicast (ALM) is a ies are still needed. However, to advocate these issues, new MOG popular technique adopted by IPTV. We apply ALM to transmit or MMOG prototypes design are introduced correspondingly. the broadcast packets through overlay network. PKTown extends Age of Empires , Counter-Strike , and Star Craft , are the LAN game experiences to players belong to different LAN. typical MOGs. To play these MOGs, there are three ways. First, We design a k-regular random overlay network based on game players who have the game copy can launch a game and enjoy specific requirements analysis. No change is made on binary code together if they are in the same Local Area Network (LAN). How- of Star Craft. Preliminary test results show that PKTown works ever, if the assumption is not satisfied, players can not enter the well and supports other MOG in the same way. Our contribution same game session even if they are near, or have small latency in is to provide a new game experience method with little hardware network communication. Alternatives require hardware invest- investment. ment. Second, players can play the game over Internet using the game service provided by the game provider , which means game servers deployed by the game provider transmit the game Categories and Subject Descriptors packets. Third, they can use game services provided by third party, C.2.4 [Distributed Systems] Distributed applications; H.4.m commercial game service providers  who deploy the transmis- [INFORMATION SYSTEMS APPLICATIONS]: Miscellane- sion servers. The difference between the second and the third way ous is not only who will deploy the hardware, but also some skills in detail. Third party game service providers can be supported by the General Terms game provider which means they can act as the second way, or Management, Design they have to capture the game packets in network layer and trans- mit the packets in a method they design, out of the control by the original game software. Generally speaking, the third way pro- Keywords vides a virtual LAN over Internet. Multiplayer Online Games, Peer-to-Peer, Overlay, Middleware, Application Layer Multicast This paper introduces PKTown, a Peer-to-Peer middleware work- ing under the MOG software. PKTown captures all packets gen- erated by games, and transmits them through a P2P overlay net- 1. INTRODUCTION work, instead of through game servers deployed as transmitters. Based on Peer-to-Peer (P2P) overlay networks, there are many Application Layer Multicast (ALM)  is a popular tech- researches to provide interesting products, including file sharing nique adopted by streaming live media. Media data is delivered  , streaming media , etc. Base on streaming media, by ALM commonly. We show that ALM can also be applied to IPTV  has gained much attention recently. P2P can sup- achieve the broadcast function in LAN game application. Star port service for lots of end users with little hardware investment, Craft is chosen as a representative real MOG to validate the mid- dleware. No change is made on the binary code of the game. The purpose of the middleware is to offer the ability to interconnect players who have small latency with each other, enabling them to play together with little additional hardware investment. The rest of paper is organized as follows. Section 2 introduces related works. Section 3 summarizes missions for PKTown to achieve with analysis of Star Craft LAN game option and services IPTV Workshop, International World Wide Web Conference, provided by third party. Section 4 describes the design of May 23, 2006, Edinburgh, Scotland, United Kingdom PKTown. Section 5 shows the preliminary test results, and section 3.1 Analysis of Game Packets in LAN Game 6 concludes this paper. To make the description simple, two PCs (named A and B) run- ning Windows XP act as end players, each with a Star Craft copy. 2. RELATED WORKS Sniffer is chosen as the analysis tool to capture packets generated by Star Craft in the network layer. We start Sniffer first, and 2.1 Replace C/S Architecture with P2P in launch a LAN game in UDP manner when selecting the game MOG or MMOG Design option on both PCs. Ghost  studies how to manage game consistency across a set of players with heterogeneous network resources and a protocol There are two kinds of packets from our observation. provides cheat-free for P2P games. DHT and a zoned federation Broadcast Packets: the IP packets that contain native address (A model  are used to adapt MOG to P2P networks. SimMud  or B) as source address and treat 255.255.255.255 as destination uses DHT and ALM for message exchange. A low-latency event address. ordering protocol, called NEO , is designed for MMOG to prevent protocol level cheats. Time-Prisoners , a fully distrib- Unicast Packets: the IP packets that contain native address A as uted MMOG, is designed in P2P architecture by JXTA. the source address, and B as the destination address, and vice versa. Similarity in all researches mentioned above is to replace the C/S architecture with P2P in MOG or MMOG design. When we ana- The usage of broadcast packets is to enquire who is here, and lyze game software, game servers always take care of data consis- announce itself at the same time. The usage of unicast packets is tency, state updating, etc. The challenge is to shift to a fully dis- to exchange the detail information needed by the game session tributed architecture. next. Based on our analysis, both functions and the usage are very common compared with the method used by ARP (Address Reso- lution Protocol)  in network layer of TCP/IP. 2.2 P2P Middleware Support Game Reuse PeerBooster  is the most similar way to PKTown to the best The observation explains why players can not enjoy together in of our knowledge. The problem of reusing MOG is that services LAN game manner if they are not in a same LAN even if they cannot continue to be provided by game venders as time goes by, have small latency. The transmission range of broadcast packets is and there is still need for consumers to play the game whether the limited to LAN, and no router will transmit this kind of packets service exist or not. A middleware called PeerBooster works be- over Internet. tween network layer and games. One module of PeerBooster called Message Handling captures packets and transmits them It is obvious that broadcasting from A and B will stop once the through P2P overlay, which approximates PKTown. unicast packets are received. In other words, only a newcomer will generate broadcast packets. This broadcasting will stop as However, there are some obvious differences between Peer- soon as other peers reply unicast packets. Booster and PKTown. First, the preliminary assumption of Peer- Booster is the server binaries are released by its game provider because one of the modules needs to act as server application. 3.2 Analysis of Methods Used in Game Ser- This assumption can not be satisfied by every MOG vender. Sec- vice Provided by Third Party ond, a module called Synchronization plays important role. This We choose Holdfast , the largest online game platform, to con- module is embedded in each peer to keep a global clock using tinue our analysis. Holdfast offers various hottest games service, NTP or GPS algorithm. Third, another Zone Management module including Star Craft, Counter-Strike, War Craft, Age of Empires, filters the packets. However, when we refer the problem of reus- etc. The similarity of these games is the ability to allow players to ing MOG, there is only one zone actually. The zone management play in same LAN, based on UDP communication. issue always comes with MMOG. At last, Topology Construction When a player login the nearest game server, enter a game room module is reported using algorithms for tree-like ALM. However, related to one game, a unique temporary reserved address, as tree-like ALM contains only one data source, topologies opti- 192.168.X.X, is assigned. All players in the same room are allo- mized like tree may not be appropriate in designing MOG mid- cated in a virtual LAN. This indicates why we find at most 250 dleware. players in one room. For PKTown, there is no need for server source code to be re- The packets generated by games are captured and reconstructed. leased as an assumption. The design of PKTown keeps simple, The real IP addresses are used to communicate between the player dispenses with considering synchronization, zone management, and the room server. etc. Such issues are left to the original games, all of which have been solved by each MOG in LAN game manner. This is the main A basic assumption is that any player can find a near room server difference between the works mentioned above and ours. We start with small latency with him. So we can understand the deploy- our design based on the observations introduced in next section. ment of many game servers working as transmitters in different ISPs. 3. ANALYSIS OF GAME ENVIRONMENT We analyze the first way and the third way mentioned in section 1, 3.3 Missions of PKTown then, we bring forward the missions of PKTown. PKTown needs to provide the ability to capture packets generated by Star Craft. Considering the method used in  caused delay on every packets for about 60-80 milliseconds without network transmission, the Hook programming skill can be used. We re- ity of 1−1/n using only 9 neighbors, and, k-regular graphs exhibit write Socket APIs to send and receive data. When Star Craft is the highest resilience. The conditions mentioned above satisfy launched, process injection can be used to replace the socket func- design of PKTown quite well. Based on this study, we adopt a k- tions used by Star Craft. Other software using network in the regular random overlay construction, with each node is given a background is not affected. fixed degree 9, the number of neighbors. We need to build an overlay with latency constraints, which means players are grouped if they are near. This requirement will Star Craft cause partition of the overlay network. Nevertheless, we prefer to divide the whole overlay. If two players have large latency, it is meaningless to bring them together. There will be several over- PKTown lays to accommodate players to different groups. However, when Game Packets Capturer we refer single overlay, it must be well connected in a global view, accompanied with random departure of peers. Outgoing The broadcast packets should be transmitted to peers as many as Packets Type possible, and as soon as possible. Players would not stay too long Checker to examine all the available servers. Member Table Broadcast Packets The unicast packets should be transmitted between game players Rebuilder in a same session as quickly as possible. Incoming Packets Overlay Type Checker 4. SYSTEM DESIGN OF PKTOWN Constructor 4.1 Assumptions of PKTown The first assumption is that MOG is designed to support LAN game based on UDP communication. We design PKTown to ex- Packets Packets Sender Receiver tend the game experience in the first way mentioned in section 1. Some old games using IPX are not considered. Almost all MOGs satisfy this assumption. Network Second, reserved network addresses are not assigned to peers. Broadcast Packets There are many peers behind NAT . However, we leave this issue in the near future. Unicast Packets Member Table Maintenance Third, game ports and other unoccupied ports are available. Two Broadcast Packets With Tag more ports are used to reconstruct UDP packets from broadcast packets to unicast packets. We assume that there are no firewalls Overlay Packets or the communications through the firewalls are allowed. Figure 1. Architecture and Data Flow of PKTown 4.2 Overlay Design 4.2.1 Peers Join The architecture and data flow of PKTown is shown in Figure 1. It is still an open question to predict latency among peers without When broadcast packets are captured, how to deliver them to direct measure. Unlike GNP , Vivaldi  or Meridian , other players over the same overlay is equivalent to ALM adopted we use GID based service scheme. A table containing full correla- by IPTV. In media data delivery, there is always a single source tion between IP address and real address all over China is col- to distribute data, so, spanning tree optimized overlay is ideal. lected. The correlation table is given to each peer and a unique However, any peer in our overlay will send out broadcast packets GID in the global network can be constructed. The GID value of when they enter the game selection interface, in other words, an end host is a 128-bits integer encoded by geometrical informa- looking for other players in same overlay. Additionally, without tion corresponding to ISPs, cities, campuses, and buildings, re- resources to share, players have no reason to stay in overlay if spectively. This method is lack of theoretical proofs, but to our they leave the game. This causes trouble in overlay connectivity experience in ALM work , this skill is sufficient for initializa- maintenance. tion. These requirements exclude tree like topologies applied in ALM. A Bootstrapping server keeps information about each overlay The appropriate choice is mesh. However, mesh like Gnutella 0.4 partition. All peers alive in one overlay are sorted by ascending  follows the power-law . Gnutella 0.6  and KaZaA  order. When a player launches PKTown first, the GID is sent to even strengthen the power-law distribution. Super peers are sup- Bootstrap server. One peer is selected as a representative of the posed to be stable. However, it is hard to find one in our overlay overlay, which is alive and adjacent to the new comer. One ping- with extremely high churn rate. pong delay is measured. If the latency is small, the new comer is considered as near to the rest of others. We set the latency restric- It is reported  recently that each user in a system with n=100 tion to 30ms in prototype. billion peers, 30-minute average lifetime, and 1-minute node- replacement delay can stay connected to the graph with probabil- After the latency is affirmed, the new peer sends Join requirement range in overlay. If the destination address is not broadcast, the packet to bootstrapping server for a candidate list. A candidate list packets are sent out without any change. is constructed with 5 to 8 peers selected randomly within the To extend the transmission range of broadcast packets from LAN overlay. Join insistently packets are sent to each candidate. to overlay network, an understandable solution is flooding like Gnutella. However, flooding generates too many redundant pack- 4.2.2 Overlay Maintenance ets and wastes bandwidth greatly. We take K-random walk  Each peer holds a 9 by 8 two-dimensional array. The content of  as our choice. each line is the neighbor at neighbor list. Heartbeat packets are sent to each neighbor 6 times per minute. If degree is changed, the When packets are received at the destination peer, packets tagged change detail will be carried in heartbeat packets. with CHANGED will be reconstructed to broadcast packets and sent to game. At the same time, these packets are also recon- When Join insistently packets are received, any peer gets a full structed with a random neighbor’s address as the destination ad- degree will send Cut packet to the neighbor with the highest de- dress. The original source address, of which a peer sends the gree, if all degrees are full, select one randomly. broadcast packets out, is carried within the new packets. Once a When broadcast packets are sent through overlay, if any peer former receives the transmitted broadcast packet and sends back receives the packets, they will check both of the original sender unicast packets, original source address will be used as destination and itself whether degree is full or not. If so, they will send An- address. nounce packet to declare their existence. This mechanism pro- vides an opportunity for a new peer and former make connection. 5. EXPERIMENT AND DISCUSSION The broadcast packets are generated by game, and we reuse them 5.1 Experiment Environment for overlay maintenance at the same time. We evaluate the prototype of PKTown in environment illustrated When Cut packets are received, no operation happens. in Figure 2. Heartbeat packets are sent to Bootstrapping server also, with a lower frequency. 4.2.3 Peers Leave If a peer leaves normally, Bye packets are sent to each of the peers in its member table. Abnormal leave is detected by heartbeat checking. When one neighbor leaves, if the degree decreases from full to 8, no operation happens; otherwise, a peer will be randomly selected from the corresponding neighbor list, and a Join insistently pack- ets will be sent. The primary packets used in PKTown are summa- rized in Table 1. Table 1. Primary Packets Used in PKTown Packets Type Function Figure 2. Separated LANs as Experiment Environment Join insistently Used for Initialization, or Peer Leaves There are three LANs separated in Huazhong University of Sci- ence and Technology. All LANs are 100Mbps Ethernet with la- Announce Hope To Add New Connection tency less than 5 ms. Join Randomly Select From Peers Announced Computers are heterogeneous from AMD 2500+ CPU with 1GB RAM to Intel Celeron3 1.1GHz with 256MB RAM. We gather Heartbeat Overlay Maintenance together 26 peers in three LANs assigned with IP address in Cut Join insistently Packets Received 202.114.14.X, and 202.114.5.X, and 202.114.8.X. Bye Peers Leave Normally 5.2 Experiment Design All other 24 peers have PKTown and Star Craft: Brood War in- 4.3 Management of Packets stalled. Two PCs illustrated in Figure 2, namely A and B, have All packets generated by Star Craft are captured by Hook. If the special edition of PKTown to collect trace. Both PCs and boot- destination address is broadcast, the packets are reconstructed to strapping server are Intel P4 2.66GHz with 512MB RAM. overlay multicast by replacing the broadcast address with some of neighbor addresses. Ports used by game are also replaced by ports The two special editions of PKTown add native time stamp on used by PKTown. The original information about source address every packet when they are captured. A little dissimilarity with and ports are carried on. Another CHANGED tag is attached other copies in source code is that A and B are always settled as which means this packet need to be rebuilt at the other end. A neighbors. All the packets transmitted between A and B is sent Time to Live (TTL) value is also added to control the transmission back, additionally with the original capture time stamp carried on. Other PKTown edition used by peers can read these special pack- ets, but treat them as normal packets. When these packets are sent mitted through overlay. We apply ALM in mesh simulating the back, a new native time stamp is added on. Without bothering broadcast process in LAN. Preliminary test results show that our clock synchronization, half of the values are estimated as transac- design is correct and game experience in LAN is extended suc- tion latency and communication latency. When we measure the cessfully. bandwidth consumption, this option is turned off. We leave issues such as packets synchronization to game software, based on the assumption that we group peers with small latency. 5.3 Preliminary Test Results and Analysis We focus on how to make the use of P2P networks to provide The latency trace result is shown in Figure 3. When players are game service without change the MOG binary code. This differs well connected, PKTown provides the game experience just like with many other works. in the same LAN. From Figure 3, we can draw the conclusion that latency consists of transmission delay and thread schedule by Future research includes several issues. First, NAT problem will operating system. The time cost by hook and packets rebuilding be solved in the near future; otherwise PKTown can not be attrac- are ignorable. tive in reality. Second, the scalability of PKTown will be studied more carefully. Third, the Bootstrapping server may cause single 10 point of failure, and fully distributed solution like Meridian will be studied in the near future. We will also study how to support 8 Counter-Strike in the same way. Delay (ms) 6 7. ACKNOWLEDGEMENTS This work is supported by National Natural Science Foundation 4 of China under grant No.60433040, and China CNGI projects under grants No.CNGI-04-12-2A and CNGI-04-12-1D. 2 8. REFERENCES  Age of Empires. http://www.microsoft.com/games/empires 0 0 100 200 300 400 500 600  Counter-Strike. http:// www.counter-strike.net Time (s)  Holdfast. http://www.cgaasia.com/index.htm Figure 3. Latency Measured in 600 Seconds  KaZaA. http://www.kazaa.com The bandwidth consumption is shown in Figure 4. From Figure 4, we find the bandwidth is very low, and most bandwidth is con-  Gnutella protocol specification 0.6. http://rfc- sumed by Star Craft. The low bandwidth consumption of Star gnutella.sourceforge.net Craft gives us great confidence in the scalability of PKTown.  Gnutella protocol specification 0.4. http://rfc- • Bandwidth Overall gnutella.sourceforge.net/developer/stable/index.html 3.0 ♦ Bandwidth of Star Craft 2.8  Star Craft. http://www.blizzard.com/starcraft 2.6  Star Craft service. http://www.battle.net/ 2.4  B. Alfonsi, "I Want My IPTV: Internet Protocol Television- Bandwidth (KB/s) 2.2 Predicted a Winner", IEEE Distributed Systems Online, Vol. 6, 2.0 No. 2, 2005. 1.8  S. Cherry, “The Battle for Broadband [Internet protocol tele- 1.6 vision]”, IEEE Spectrum, vol. 42, issue 1, Jan. 2005. 1.4  F. Dabek, R. Cox, F. Kaashoek, and R. Morris, “Vivaldi: A 1.2 Decentralized Network Coordinate System”, Proceedings of SIG- 1.0 COMM, Portland, OR, August 2004. 0.8  C. G. Dickey, D. Zappala, V. Lo, and J. Marr, “Low Latency 0 100 200 300 400 500 600 and Cheat-proof Event Ordering for Peer-to-Peer Games”, Pro- Time (s) ceedings of NOSSDAV, Cork, Ireland, June 2004 Figure 4. Bandwidth Measured in 600 Seconds  C. Gkantsidis, M. Mihail, and A. Saberi, “Random Walks in Peer-to-Peer Networks”, Proceedings of INFOCOM, Hong Kong, 6. CONCLUSIONS AND FUTURE WORK China, March 2004. In this paper, we introduce a novel method to combine peer-to-  A. S. John and B. N. Levine, “Supporting P2P Gaming When peer overlay network with a representative MOG, Star Craft. We Players Have Heterogeneous Resources”, Proceedings of NOSS- build a k-regular overlay to maintain the connectivity with high DAV, Stevenson, Washington, USA, June 2005. churn rate. Broadcast packets act key function to find other play- ers in different LANs. When captured, they are rebuilt and trans-  Y. Kaneda, H. Takahashi, M. Saito, H. Aida, and H. Tokuda,  M. Merabti and A. El Rhalibi, “Peer-to-Peer Architecture “A Challenge for Reusing Multiplayer Online Games without and Protocol for a Massively Multiplayer Online Game”, Pro- Modifying Binaries”, Proceedings of NetGames, Hawthorne, New ceedings of GLOBECOM Workshops, Dallas, TX, November York, USA, October 2005. 2004.  B. Knutsson, H. Lu, W. Xu, and B. Hopkins, “Peer-to-peer  T. Ng and H. Zhang, “Predicting Internet Network Distance support for massively multiplayer games”, Proceedings of INFO- with Coordinates-Based Approaches”, Proceedings of INFOCOM, COM, Hong Kong, China, March 2004. New York, NY, June 2002.  D. Leonard, V. Rai, and D. Loguinov, “On Lifetime-Based  D. C. Plummer, “An Ethernet Address Resolution Protocol -- Node Failure and Stochastic Resilience of Decentralized Peer-to- or -- Converting Network Protocol Addresses to 48.bit Ethernet Peer Networks”, Proceedings of SIGMETRICS, Banff, Alberta, Address for Transmission on Ethernet Hardware”, STD 37, RFC Canada, June 2005. 826, MIT, November 1982.  X. Liao, H. Jin, Y. Liu, L. M. Ni, and D. Deng, “AnySee:  P. Srisuresh and M. Holdrege, “IP Network Address Transla- Inter-Overlay Optimization based P2P Live Streaming System”, tor (NAT) Terminology and Considerations”, RFC 2663, 1999. Proceedings of INFOCOM, Barcelona, Spain. April 2006.  B. Wong, A. Slivkins, and E. G. Sirer, “Meridian: A Light-  T. Limura, H. Hazeyama, and Y. Kadobayashi, “Zoned Fed- weight Network Location Service without Virtual Coordinates”, eration of Game Servers: a Peer-to-peer Approach to Scalable Proceedings of SIGCOMM, Philadelphia, Pennsylvania, USA Multi-player Online Games”, Proceedings of SIGCOMM Work- August 2005. shops, Portland, Oregon, USA, August 2004.  X. Zhang, J. Liu, B. Li, and T.-S. P. Yum, “CoolStream-  Q. Lv, P. Cao, E. Cohen, K. Li, and S. Shenker, “Search and ing/DONet: a data-driven overlay network for live media stream- replication in unstructured peer-to-peer networks”, Proceedings of ing”, Proceedings of INFOCOM, Miami, FL, USA, March 2005. International Conference on Supercomputing (ICS), New York City, NY, USA, June 2002.