Get documents about "Microsoft .NET and XNA Framework for the Robot Operating System
Document Sample
Page |1
Microsoft .NET and XNA Framework for the Robot
Operating System
Michael Edoror
B.S Computer Science „12
University of Maryland, Baltimore County
Mentor: Chad Jenkins, Ph.D
Robotics, Learning, and Autonomy Lab
Department of Computer Science
Brown University
Page |2
Table of Contents:
1. Introduction
2. ROS & Rosbridge
3. JSON
4. .NET & XNA Integration
5. Conclusions & Future Work
6. References
Page |3
1. INTRODUCTION
The Robot Operating System (ROS) is an open source system which provides libraries and tools
to help software developers create robot applications. The most important feature with ROS is
its abstraction of hardware and device specific engineering. Conceptually, ROS sits on top of
hardware and programmers write software to control the hardware.1
My research dealt with creating an interface which can communicate with the ROS runtime
environment using .NET and Microsoft XNA. This framework can publish and subscribe to
information from ROS and its connected devices over a TCP/IP socket connection.
2. ROS & ROSBRIDGE
ROS executes its programs as nodes which can run simultaneously. In order for ROS entities to
communicate with each other, ROS nodes publish data to listeners called topics. Conversely, in
order for entities to receive information they subscribe to topics.
The Robotics Lab at Brown University have been developing software for ROS, creating many
programs including special software that will allow for connection to ROS and ROS objects via
the web. The interface is a ROS program called Rosbridge and that functionality allows people
to theoretically control ROS object from anywhere and on any type of computer. ROS has been
developed and maintained primarily on Ubuntu Linux but with Rosbridge and my research, I
have created a framework for .NET and XNA.
The overall goal of Rosbridge is to be able to send and receive messages made by ROS objects
on the ROS server. Messages in the correct format can be sent over to the ROS object through
the topic. In order to receive messages, Rosbridge will subscribe to a certain topic and basically
listen to what the ROS object publishes to that topic.
Page |4
For this project I used two ROS enabled robots. The first was an iRobot_Create. The robot is
connected through a serial connection with a netbook that has ROS installed. I would optionally
connect a Playstation Eye to the netbook in order to get a video stream from the robot. The
second robot I used was an AR_Drone. I made no modifications to the drone and it connects
wirelessly to a netbook which has ROS installed.
2. JSON
Rosbridge requires a certain format in order to send and receive data across the socket
connection. This format is called JavaScript Object Notation (JSON). It is an open standard
many programmers use to interchange data, usually for JavaScript. The exact form varies but for
Rosbridge, message data is arranged in the following manner; JSON messages are just sequences
of “name : value” pairs wrapped in either quotes for strings, square brackets for arrays, or curly
brackets for objects. For an example this is the JSON formatted string I used to subscribe to a
topic in ROS.
service =
"{\"receiver\":\"/rosjs/subscribe\",\"msg\":[\"/sensorPacket\",0]}";
Figure 1. JSON Formatted String
3. MICROSOFT .NET & XNA
The .Net Framework is just like any other programming environment with the standard tools to
help developers create software. The Framework supports programming in the typical
programming language paradigms: functional (F#), procedural (Visual Basic), and object
oriented (C#). The most popular Integrated Development Environment (IDE) is Microsoft
Page |5
Visual Studio. In order to create my framework for the ROS connection I specifically used
Microsoft Visual Studio 2008 with Microsoft Framework version 3.5 and C#.
My first task was to create the client interface that communicates between ROS and the host
Windows Machine. The way Rosbridge was created makes this process pretty straight forward.
In fact, a JavaScript client interface was written in 73 lines of code. The process of creating the
client is as follows: create a socket connection, connect the socket to the correct ROS server, and
send the appropriate handshake. Afterwards, I would be able to send and receive messages from
the ROS server. The big issue that I ran into was that after establishing the connection and
sending the handshake, the messages I would summarily send over would not be seen by the
ROS server. In order to alleviate this issue I had to send multiple messages at the same time in
order for ROS to register it as seen. It turns out that Rosbridge arbitrarily drops messages. When
it was all said in done, I had a fully functional client interface now called WinRos which can
send and receive messages from a ROS server all in around 85 lines of code. With all of that in
place, I wrote a small teleoperator application that would be able to control an iRobot_Create and
an AR_Drone. This application runs through the keyboard and can be found here.
Once the foundation of communication was reached, I had hopes on expanding on Windows
development and making something interesting. I decided to integrate my existing code with
Microsoft XNA to create a video game.
The Microsoft XNA Framework is a collection of classes and libraries used to enhance
development of games within the .NET framework. In addition to using XNA, I also used a
game engine, FlatRedBall (FRB). FRB is an engine written over top of XNA which encapsulates
all of the coding intensive tasks of XNA like loading art assets or creating screens. FRB also has
Page |6
a tool called Glue which provides a visual-interactive interface for game development on XNA.
For the game I used FRB June Release and Microsoft XNA Game studio version 3.1. This game
is made for XNA PC.
The process for creating the game was as follows: Use the existing WinRos interface to get a
connection to Rosbridge through XNA, subscribe to topics that ROS objects on the server
transmit like movement and video data, read/process the data from the socket, and display the
data on the XNA side by drawing it to the screen. It seems straightforward but I still ran into a
couple of issues.
Getting a video stream was probably the biggest issue. The XNA framework supports video but
only static videos. Streaming video is not supported for the most part. Another issue is that
getting video from the Rosbridge connection wasn‟t traditional in that the data I was reading was
an encoded binary string. The normal forms to transmit video are of a compressed video format.
To deal with all of this, I had to do a little conversion gymnastics to decode the binary string in
base64 format. Now in order to display the video I had to attach it to a Sprite instance. A sprite
is the container of the game object or character in XNA. Every sprite has a texture, which is the
actual image. In this case the texture is the video data, so I had to attach the video information to
the texture. However texture information has to be in a certain form (in this case an array of
bytes) so I further converted the binary string into an array of bytes and read it into the texture
using a memory stream. The following code snippet shows the conversion and the next snippet
is the process for attaching the video to the texture.
Page |7
public MemoryStream decodeFrom64(string encodedData)
{
//decodes the binary string
byte[] encodedDataAsBytes=
System.Convert.FromBase64String(encodedData);
//creates an instance of memory which is just a buffer for the image
data
MemoryStream ms = new MemoryStream(encodedDataAsBytes, 0,
encodedDataAsBytes.Length);
//writes the image data to the buffer
ms.Write(encodedDataAsBytes, 0, encodedDataAsBytes.Length);
return ms;
}
Figure 2. Binary String Conversion
//converts raw data to string
image
= Convert.ToString(((Hashtable)serviceReturn["msg"])["uri"]);
//removes the json part of the string
image = image.Remove(0, 23);
//creates the file stream
MemoryStream stream = decodeFrom64(image);
//these next lines return the buffer to the original position
stream.Flush();
//zero base64
stream.Position = 0L;
//attaches the stream to the texture
video.Texture =
Texture2D.FromFile(FlatRedBallServices.GraphicsDevice, stream);
video.ScaleX = 10.5f;
video.ScaleY = 10.5f;
video.Position = new Vector3(5, 0, 0);
Figure 3. Code Snippet of the complete process
After all of this, I was getting a very choppy image display. The problem was that the frame rate
was moving faster than the display of the image. Slowing the frame rate from 60 frames per
second to 5 really showed streaming in the video. There was still a 2 second delay between
actual movement and the display, but display is a lot more functional than before.
Page |8
4 .CONCLUSIONS & FUTURE WORK
I feel very happy about what I accomplished this summer. I learned that I have the capability to
develop software at a level that I thought wasn‟t reachable yet. Prior to this internship, I had no
experience with the Robot Operating System. I had very little exposure to programming on the
Windows environment but now I feel very comfortable. I also did not have experience with
developing applications, but now I do thanks to this internship and the constant pushing from my
mentor.
I hope to continue the work I have done this summer. There is a lot I can do to improve and
expand upon. For example, I would like to add complete drone functionality to the
WinRos_XNA interface. Being able to control a drone through XNA would be very exciting. I
would also like to use a robot called the Neato vacuum robot which uses its laser range-finder to
create mappings of its surroundings. With that I could possibly display this data on the XNA
side to navigate robots by video and map. I would also like to make the application into an
actual Xbox 360 console game. As of now it can only run on an XNA-enabled PC. In order to
do this, I would need to upgrade my XNA game studio version and probably rewrite some code.
I would also like to add better art assets to the XNA interface. A large problem I had was the
time it took to create the art for the game being that I was very indecisive and not capable of
producing good art. Finally, I would also need to upgrade the code itself to make sure every part
is as efficient as possible.
Page |9
6. REFERENCES
1 Conley, Ken. "Ros/Introduction." ROS.org. Creative Commons Attribution 3.0, 18Jul2011.
Web. 15 Aug 2011. <http://www.ros.org/wiki/ROS/Introduction>.
