Programming User
Interfaces using The
Nintendo Wii Remote
Charles Pheatt
Scott Goering
Emporia State University
The Motivational Video
Wiimote + 15 Ton Robot .
WHY?
It’s fun.
Useful as an alternate input device to a PC.
Provides valuable lessons in multithreaded
applications.
Provides valuable lessons in UI implementation.
Hopefully provides some ideas for student
projects.
The Real Original Motivation
Reaction time project
Joint project with physical education faculty.
Extend classic methods of measuring reaction time.
Uses force plates to capture reaction time.
Decidedly “low cost” approach with real-time data
acquisition.
Phase I – proof of concept (really low $)
Phase II – refine the measures
Phase III – new horizons
Parallax Force Plate
Reaction Time – Phase I
The $100 Data Acquisition Solution
Parallax Basic Stamp and some A2Ds
Acquisition rate 1k/s
Reaction Time – Phase II
The $1000 Data Acquisition Solution
National Instruments device and software (C#)
Acquisition rate 250k/s
Reaction Time – Phase III
Can we do the same thing for a really low cost?
Acquisition rate 0.1k/s
Wii Factoids
67.4 million units (December 2009)
Johnny Chung Lee
Carnegie Mellon Ph.D. student (now @ Microsoft)
Human-Computer Interaction
Known for interactive whiteboards
WiiBrew
Wiki dedicated to homebrew on the Nintendo Wii
Coding4Fun (MSDN)
Managed Library for Nintendo's Wiimote
Our Project
Select one of the available libraries for Wiimote
communication.
Many of the available libraries are NOT professional
development efforts.
Investigate the libraries applicability for easily creating
Wii projects.
Selected the WiimoteLib (Managed Library for
Nintendo's Wiimote)
Develop a group of tutorials (handouts) to reduce the
steepness of the learning curve.
Wiimote Attributes
Uses the Bluetooth protocol for communication.
Nominal data rate is 100 samples/second
Peripheral choices:
Wiimote (buttons, accelerometer, IR)
Nunchuk (buttons, accelerometer, joystick)
Classic Controller (buttons, joysticks)
Balance Board (load cells)
Guitar (buttons, touch-bar, strum, joystick)
Drums (buttons, pads, cymbals, joystick)
WiimoteLib Attributes
Requires the use of the WiimoteLib.dll.
Supplied with multiform C# and VB examples.
Provides infrastructure to capture and pass
Wiimote state data.
Wiimote data is available through a class
hierarchy called WiimoteState.
State change is noted in the UpdateState
method.
Finally the Tutorial
Connection (Handout 0)
General Programming Setup (Handout 1)
Hello World Programs (Handout 2)
Wiimote Accelerometer (Handout 4)
Wiimote IR Detector (Handout 5)
Tiny Tennis Application
Connection
WiimoteLib Interface Framework
CollectWiimotes
Provides access to the Wiimote dictionary which
stores the Wiimote IDs and class instances as well as
the WiimoteCollection class.
We must create instances of each and provide get
and set methods.
public partial class CollectWiimotes
{
// Dictionary holds Guids and class instances
// Allows us to easily access an individual wiimote
private static Dictionary mWiimoteMap =
new Dictionary();
// WiiMoteCollection is a class from the DLL that stores all
// connected wiimotes
private static WiimoteCollection mWC = new WiimoteCollection();
// Standard Get/Set methods for our Dictionary/Collection
public static Dictionary wmMap
{
get { return mWiimoteMap; }
set { mWiimoteMap = value; }
}
public static WiimoteCollection WiiCollect
{
get { return mWC; }
set { mWC = value; }
}
}
WiimoteLib Interface Framework
WiimoteInfo
Provides access to the Wiimote state information.
The UpdateState method is invoked by the
WiimoteInfo Class (via the wm_WiimoteChanged
method) when the Wiimotes state changes.
The UpdateState method updates the user interface.
UpdateExtension is invoked by the WiimoteInfo
Class (via the wm_WiimoteExtensionChanged
method) when the Wiimote has extensions
(numchuck, etc.) attached.
public partial class WiimoteInfo
{
private Wiimote mWiimote;
public int WiimoteLED;
// semaphore used to prevent race condition in console display
static private Semaphore updateSemaphore = new Semaphore(1, 1);
//Default Constructor
public WiimoteInfo()
{
}
// Assignment constructor
// Is passed a Wiimote object and assigns it to the instance in the
// WiimoteInfo class
public WiimoteInfo(Wiimote wm)
: this()
{
mWiimote = wm;
}
// Used for updating the wiimote itself
// This is where you put output statements to see changes in the wiimote
// state
public void UpdateState(WiimoteChangedEventArgs args)
{
WiimoteState ws = args.WiimoteState;
updateSemaphore.WaitOne();
Console.SetCursorPosition(0, this.WiimoteLED);
Console.WriteLine(this.WiimoteLED.ToString() +
": Current Battery Value = " + ws.Battery.ToString("F1"));
updateSemaphore.Release();
}
// Used for updating the extensions (if any are plugged in)
public void UpdateExtension(WiimoteExtensionChangedEventArgs args)
{
updateSemaphore.WaitOne();
Console.SetCursorPosition(40, this.WiimoteLED);
Console.WriteLine(" ");
Console.SetCursorPosition(40, this.WiimoteLED);
Console.WriteLine(args.ExtensionType.ToString());
updateSemaphore.Release();
}
// get/set methods for the class
public Wiimote Wiimote
{
get { return mWiimote; }
set { mWiimote = value; }
}
}
C# WiimoteLib Setup
1. Download the Managed Library for Nintendo's Wiimote. Unzip
the archive. Locate the file WiimoteLib.dll in the primary folder
WiimotLib_1.7.
2. Create a C# Windows Forms Application or Console
Application in the .NET IDE.
3. Put the file WiimoteLib.dll in your project folder. We generally
place the dll in the folder that contains the source code files.
4. For the project you created in the C# .NET IDE, open the
Solution Explorer tab. Right click on References and navigate to
Add Reference | Browse Tab. Use the dropdown file list and
locate the WiimoteLib.dll in your project folder. Highlight the
dll and click OK.
5. Add a using WiimoteLib; statement to files which will reference
the WiimoteLib methods.
Hello World
Console Application
Publisher/Subscriber Pattern
Background Worker
Delegate
Console Application
UI Issues
Questions
What is the best model for integrating an external
data source (external device, external data source,
client/server) with our UI?
How can I update my user interface from a thread
that did not create it?
Where is the thread of control for my program
anyway?
Publisher/Subscriber Pattern
Is a variation of the message queue pattern.
Senders (publishers) of messages are not
programmed to send their messages to specific
receivers (subscribers).
Used in Service Oriented Architecture (SOA).
Publisher/Subscriber in C#
Publisher/Subscriber
WiimoteInfo - UpdateState
WiimoteInfo - wm_Publisher
WiimoteInfo - public delegate void PublishToForm
WiimoteInfo - PostMessage
WiimoteInfo - public event PublishToForm
Form1 - wm_Subscriber
Form1 - delegate void updatetextBoxDelegate
Form1 - updateTextBox
Background Worker
BackgroundWorker – is a toolbox control in the
C# IDE
Intensive tasks need to be done on another
thread so that the UI doesn't freeze.
The BW posts messages and updates the user
interface as the task progresses.
Data to BW
BW is created on the same thread as the form.
May update form at will.
Blocking Queue structure used to transfer
updates to the BW.
In a client/server model - Clients add requests
to a queue. A server retrieves requests from the
queue.
We want the BW to stall (nicely) if there are no
messages.
private BackgroundWorker BW = new BackgroundWorker();
// initialize the background worker
BW.WorkerReportsProgress = true;
BW.DoWork += new DoWorkEventHandler(bw_DoWork);
BW.ProgressChanged += new ProgressChangedEventHandler(bw_ProgressChanged);
private void bw_DoWork(object sender, DoWorkEventArgs e)
{
BackgroundWorker worker = sender as BackgroundWorker;
while (true)
{
// Dequeue the state information and pass it on to bw_ProgressChanged
// via ReportProgress
// BQ remains blocked until there is new state information
object WiiState = GlobalData.BQ.Dequeue(Timeout.Infinite);
worker.ReportProgress(0, WiiState);
}
}
// Update UI as necessary
private void bw_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
GlobalData.WiiData WiiState = (GlobalData.WiiData)e.UserState; ;
TextBoxes[WiiState.WiiIndex].Text = WiiState.WiiInformation;
}
The Delegate in C#
A delegate is a type that references a method.
Once a delegate is assigned a method, it behaves
exactly like that method.
Delegates are similar to C++ function
pointers, but are type safe.
Delegates allow methods to be passed as
parameters.
Delegates can be used to define callback
(functor) methods.
// delegate definition
public delegate void UpdateWiimoteStateInformationCallback(object
sender, WiimoteChangedEventArgs e);
// delegate to update the UI
private void UpdateWiimoteStateInformation(object sender,
WiimoteChangedEventArgs e)
{
WiimoteInfo wi = CollectWiimotes.wmMap[((Wiimote)sender).ID];
WiimoteState ws = e.WiimoteState;
TextBoxes[wi.WiimoteLED - 1].Text = ws.Battery.ToString("F2") +
"\r\n" + ws.ExtensionType.ToString();
}
// Called when the wiimote changes state
// Then invokes the delegate to the UI
private void wm_WiimoteChanged(object sender, WiimoteChangedEventArgs e)
{
WiimoteInfo wi = CollectWiimotes.wmMap[((Wiimote)sender).ID];
try // try block needed when main form is disposed
{
BeginInvoke(new
UpdateWiimoteStateInformationCallback(UpdateWiimoteStateInformation),
new object[] { sender, e});
}
catch { }
}
Accelerometer
The Wiimote and
Nunchuck have the
ability to sense
acceleration along three
axes through the use of
an accelerometer.
The accelerometer
measures acceleration
with a minimum full-
scale range of ±3 g.
G Calculations
The Wiimote provides raw accelerometer values as byte
values (via the AccelState structure).
The raw values are transformed to normalized
accelerometer values (in gravity units) using values from
the AccelCalibrationInfo structure.
Gravity (g) values are calculated as:
AccelState Structure
AccelCalibrationInfo Structure
Smoothing may be your friend.
// used for smoothing accelerometer data
public Queue Xvalues;
public Queue Yvalues;
public Queue Zvalues;
public float Xsum;
public float Ysum;
public float Zsum;
// The WiimoteLib will return accelerometer values of infinity, which is problematic
if (!float.IsInfinity(WiiState.AccelState.Values.X))
{
Xsum = Xsum + WiiState.AccelState.Values.X - Xvalues.Dequeue();
Xvalues.Enqueue(WiiState.AccelState.Values.X);
}
if (!float.IsInfinity(WiiState.AccelState.Values.Y))
{
Ysum = Ysum + WiiState.AccelState.Values.Y - Yvalues.Dequeue();
Yvalues.Enqueue(WiiState.AccelState.Values.Y);
}
if (!float.IsInfinity(WiiState.AccelState.Values.Z))
{
Zsum = Zsum + WiiState.AccelState.Values.Z - Zvalues.Dequeue();
Zvalues.Enqueue(WiiState.AccelState.Values.Z);
}
Accelerometer Sample Application
Uses SharpGL – a C# control that can be added
to the C# IDE to provide OpenGL drawing
and rendering.
Accelerometer_2_Exerciser_Forms_Application
Demo.
Wiimote IR Detector
Wiimote contains an IR image sensor that will
detect multiple IR sources.
The sensor includes a 128x96 monochrome
camera with built-in image processing.
The built-in processor uses 8x sub-pixel analysis
to provide 1024x768 resolution for the tracked
points.
The camera's built-in image processing is
capable of tracking up to 4 moving objects.
Sensor Bar
The sensor bar is powered by the Wii base unit,
and contains IR LEDs.
Homemade sensor bars have been effective with
fewer LEDs, so long as the intensity is
sufficient.
No information is passed either to or from the
sensor bar, and the LED intensity is not
modulated in any way.
IRState and IRSensor Data
Structures
IRState and IRSensor Data
Structures
IRState and IRSensor Data
Structures
Sample Application 1
Uses background worker approach.
Creates a bitmap object to render IR sources.
Uses normalized source locations.
private void bw_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
GlobalData.WiiData WiiState = (GlobalData.WiiData)e.UserState; // state
int rectangleSize = 10; // for plotting sensor locations
// calculate the screen positions for each sensor reading
int x1 = (int)(WiiState.IR.IRSensors[0].Position.X * pictureBox1.Size.Width);
int y1 = (int)(WiiState.IR.IRSensors[0].Position.Y * pictureBox1.Size.Height);
int x2 = (int)(WiiState.IR.IRSensors[1].Position.X * pictureBox1.Size.Width);
int y2 = (int)(WiiState.IR.IRSensors[1].Position.Y * pictureBox1.Size.Height);
int x3 = (int)(WiiState.IR.IRSensors[2].Position.X * pictureBox1.Size.Width);
int y3 = (int)(WiiState.IR.IRSensors[2].Position.Y * pictureBox1.Size.Height);
int x4 = (int)(WiiState.IR.IRSensors[3].Position.X * pictureBox1.Size.Width);
int y4 = (int)(WiiState.IR.IRSensors[3].Position.Y * pictureBox1.Size.Height);
int x_mid = (int)(WiiState.IR.Midpoint.X * (pictureBox1.Size.Width - 0));
int y_mid = (int)(WiiState.IR.Midpoint.Y * (pictureBox1.Size.Height - 0));
G.Clear(Color.White); // picture box background is white
Brush myBrush = Brushes.Black;
Font myFont = new Font("Courier", 10);
StringFormat strFormat = new StringFormat();
strFormat.Alignment = StringAlignment.Center;
strFormat.LineAlignment = StringAlignment.Center;
if (WiiState.IR.IRSensors[0].Found)
G.DrawString("1", myFont, myBrush,
new RectangleF(x1, y1, rectangleSize, rectangleSize), strFormat);
if (WiiState.IR.IRSensors[1].Found)
G.DrawString("2", myFont, myBrush,
new RectangleF(x2, y2, rectangleSize, rectangleSize), strFormat);
if (WiiState.IR.IRSensors[2].Found)
G.DrawString("3", myFont, myBrush,
new RectangleF(x3, y3, rectangleSize, rectangleSize), strFormat);
if (WiiState.IR.IRSensors[3].Found)
G.DrawString("4", myFont, myBrush,
new RectangleF(x4, y4, rectangleSize, rectangleSize), strFormat);
if (WiiState.IR.IRSensors[0].Found && WiiState.IR.IRSensors[1].Found)
G.DrawString("M", myFont, myBrush,
new RectangleF(x_mid, y_mid, rectangleSize, rectangleSize), strFormat);
pictureBox1.Refresh();
Sample Application 2
Render a triangle representing the orientation of
LEDs 1 and 2.
Uses 2D transformation to orient the triangle.
Same basic setup as Application 1.
Requires a little “finesse” to keep the triangle
movement consistent.
public partial class Form1 : Form
{
private BackgroundWorker BW = new BackgroundWorker();
// used for displaying IR data
public Bitmap my_bitmap;
public Graphics G;
// maintains cursor location when cursor moves off screen
public int lastGoodCursor_x;
public int lastGoodCursor_y;
// update last on-screen cursor location
if (WiiState.IR.IRSensors[0].Found && WiiState.IR.IRSensors[1].Found)
{
lastGoodCursor_x = x_mid;
lastGoodCursor_y = y_mid;
}
// calculations for the triangle (rotation transformation)
double angle = Math.Atan2(x2 - x1, y2 - y1);
double side = 30;
double p_x1 = -side / 2 * Math.Cos(angle) - -0.5 * Math.Sqrt(3) * side / 2 *
Math.Sin(angle);
double p_y1 = -0.5 * Math.Sqrt(3) * side / 2 * Math.Cos(angle) + -side / 2 *
Math.Sin(angle);
double p_x2 = side / 2 * Math.Cos(angle) - -0.5 * Math.Sqrt(3) * side / 2 *
Math.Sin(angle);
double p_y2 = -0.5 * Math.Sqrt(3) * side / 2 * Math.Cos(angle) + side / 2 *
Math.Sin(angle);
double p_x3 = 0 * Math.Cos(angle) - 0.5 * Math.Sqrt(3) * side / 2 *
Math.Sin(angle);
double p_y3 = 0.5 * Math.Sqrt(3) * side / 2 * Math.Cos(angle) + 0 *
Math.Sin(angle);
Tiny Tennis
Based on a Coding4Fun article available on the
MSDN blogs.
The implementation uses sprites for the bats and
ball and is easily extendable.
The Wiimote A and B buttons act in place of
the original keyboard buttons.
Component Sources
Bluetooth Dongle & IR components
http://www.wiiteachers.com/
List of Working Bluetooth Devices
http://wiibrew.org/wiki/List_of_Working_Bluetooth_Devices
Vishay TSAL 6400 IR LED
http://www.mouser.com
http://search.digikey.com
References
Johnny Chung Lee
http://johnnylee.net/projects/wii/
WiiBrew
http://wiibrew.org
Reaction time project
http://pheattarchive.emporia.edu/projects/reactiontime
Publishers/Subscribers Pattern
http://www.akadia.com/services/dotnet_delegates_and_events.html
Background Worker
http://dotnetperls.com/backgroundworker
References
C# implementation of a Bound Blocking Queue
http://www.eggheadcafe.com/articles/20060414.asp
Java 5's BlockingQueue
http://www.developer.com/java/ent/article.php/3645111/Java-5s-BlockingQueue.htm
SharpGL: a C# OpenGL class library
http://www.codeproject.com/KB/openGL/sharpgl.aspx
Tiny Tennis
http://blogs.msdn.com/coding4fun/archive/2006/10/31/916355.aspx