Smart Projector 4th Year Engineering Project
Professor T. Pearce September 24, 2004 Edward Chow <100288605> Chris Thomas <100286486>
Table of Contents
1.0 INTRODUCTION .................................................................................................................................. 3 2.0 OBJECTIVE ........................................................................................................................................... 3 3.0 BACKGROUND ..................................................................................................................................... 3 4.0 TECHNICAL OVERVIEW .................................................................................................................. 4 5.0 TIMELINE.............................................................................................................................................. 6 6.0 REQUIREMENTS ................................................................................................................................. 7 7.0 DELIVERABLES ................................................................................................................................... 7
2
1.0 Introduction
This document is a proposal for a 4th year engineering project entitled „Smart Projector‟ which is to be undertaken by Edward Chow and Chris Thomas, under the supervision of Professor Trevor Pearce. The timeline of the project spans September 2004 to March 2005.
2.0 Objective
To implement an intelligent projector system (herein referred as Smart Projector) which uses a standard projector, software algorithms and a camera input to calculate the required geometric remapping/re-colouring of image pixels before output to adapt to various surfaces and angles producing proper image aspect and colour output while improving on some current shortcomings of the technology.
3.0 Background
Projector technology has evolved to a level where it has become a widely used tool for both business and personal applications. Many business conference rooms are now equipped with projectors and because of new projector technology constantly being developed, they have become affordable enough to make their way into the home theatre leapfrogging over televisions in screen size which dominated the large screen market for years. Surprisingly, while there have been great advances in improving projector image output such as Digital Light Processing (DLP) technology developed by Texas Instruments which produces high quality images by using a few hundred thousand tiny mirrors attached to hinges controlled by tiny motors all on a wafer not much larger than a square centimetre – there have not been commercial implementations of projectors taking advantage of the processing power readily available in modern microprocessors which, when coupled with relatively low-cost, high quality imaging sensors such as Charged Coupled Device (CCD) sensors, could effectively solve many of the current issues slowing projection technology‟s rapid growth. One major issue is the requirement of a large flat screen to achieve a bright, distortion free image. The screen must be also be mounted in a position that provides an obstruction-free view for everyone while still maintaining a proper angle for the projector. This is not always possible in all room layouts with limited wall space and usually results in limited seating arrangements. Projector placement also makes it difficult for presenters to avoid casting shadows on the screen. Because of these stringent requirements of correct placement of both the projector and the screen, the setup cannot be changed easily either. These limiting factors are unique to front projection screens
3
and make them less attractive compared to other upcoming large screen technologies such as plasma televisions. Clearly, removing the requirement of a screen and being able to place the projector anywhere would make projectors a much more viable option to a wider market. The problem lies in that projectors are designed to be used with a high quality white, matte screen which provides a much more ideal surface than any other wall found in a normal room. Walls made of materials such as concrete, brick and wood all have their own colour and texture with many imperfections on their surface (scratches, gaps, holes). They could also have objects such as paintings, curtains or posters hanging on them as well. The obvious result of projecting an image on these surfaces is an image that is warped and off colour throughout; placing the projector at a non-perpendicular angle to the surface further skews the image. These conditions would of course make for a very unpleasant experience for the viewer. A Smart Projector system could alleviate these problems. By gathering data from the projected image with a simple digital camera and calculating the amount of warp, artifacts and discolouration the surface and angle causes, an algorithm could be developed to compensate by remapping and re-colouring each individual pixel of the image before output. The end result would ideally be a high quality image with its aspect ratio preserved and have colours matching the original input with out visible artifacts caused by imperfections on the wall.
4.0 Technical Overview
Project work will follow an iterative approach by reaching basic milestones to test functionality before working towards final more complex goals. This ensures that any problems discovered can be troubleshot from a more narrow scope reducing debugging time. The software package will be of an Object Oriented design with separate modules for each major component of the system. Initial software design planning will be in the form of a set of Unified Modeling Language (UML) diagrams to layout class structure and functionality. The main layers will be the Input, Camera, Calibration, Transform and Output. The Input layer will be responsible for converting all input data from a source into a readable format for the Transform component. The goal of this design requirement is to allow for easier implementation of different input sources in future developments such as DVD or another camera. The Camera layer will be responsible for controlling the digital camera used for calibration. It will interface with the underlying operating system to access hardware drivers and serve as a bridge for the Calibration layer.
4
System initialization begins in the Calibration layer. It will receive data from the Camera layer and use it to generate the transform filter used to alter pixel colour values and coordinates before being output. Most of the processing will be done in the Transform layer. Its main responsibility will be to take the images from the Input classes and apply the transform filter to them. This effectively pre-warps the image so that upon output, it is warped again to appear as it would if it were on a normal surface. The Output layer will take the transformed image and display it on the computer screen. This may not be required if the Windows API or DirectX allows a simpler approach such as direct warping of the desktop or a direct hardware pass-through to the projector port. The camera to be used will be a digital camera using a CCD sensor. A CCD sensor was chosen because CCD sensors produce truer colour output in a wider range of lighting conditions compared to older CMOS sensors. Quality input is very important for the calibration process because of the pixel level detail required to generate an accurate transform filter especially in the low light conditions that projectors are usually used in. The calibration sequence will utilize a series of images projected on the surface to find in order, the boundaries, pixel shift and base colour values. Projection boundaries will be found by projecting an all white image on the surface and comparing it to the surface with nothing projected on it. An algorithm that subtracts the two images and finds the difference will give the base image to find the boundaries of the projection area with. The largest rectangle within those boundaries will then be the usable area for the image projection. Current approaches use a series of horizontal and vertical line patterns to detect variations in positioning of pixels along the lines to determine how to map pixel shifts along the surface. The process takes approximately 30 seconds with current methods. Research will be done to combine stages or find more efficient methods to reduce the calibration duration. There have been a few papers published within the last few years with material related to different parts of the technical aspects outlined. One paper attempted to extend its algorithm to include continuous feedback in order to continually adjust the image during runtime [1]. One flaw it had was that it sometimes took several frames to adjust the transform filter properly. Exploration into improving the algorithm to require less or even 1 frame to effectively compensate or developing a method of pre-mapping a large area spanning beyond the initial focus for panning on may be attempted time permitting. The ability to do so would create new commercial uses such as moving illusionary
[1] Nayer S, Peri H, Grossberg M, Belhumeur P “A Projection System with Radiometric Compensation for Screen Imperfections.” Available Online http://www.procams.org/papers/nayar_peri_grossberg_belhumeur.pdf
5
lighting effects at places such as theme parks or night clubs. A near real-time compensation could provide for projecting on a moving background such as a swinging door. Even further development into the realm of object tracking could be used to superimpose imagery on an object such as a model and change its appearance or “skin” on the fly which could be used to demonstrate things such as how a car might look in a different colour or change actors appearances while still on stage. With the modular design to be taken, individual components could be used or added to the “toolset”. For example, future implementation of a scripting engine could allow adjusting and integrating functions of the system to support multiple different inputs for completely different uses in the future. Imagine a smart home for example; a rotating smart projector could be programmed to do things such as take a snapshot of the contents of a refrigerator when open and display an image of its contents on the door when closed. It could later display an image of who‟s at the door on nearby surface to whoever is home at the time. When not interacting with residents, it could return to an aesthetic role such as cycling through images on an empty frame hanging on the wall while displaying the time. When complete, the results of this project could form the framework for future developments like the ones just described. Implementation of the foundation software algorithms will follow the timeline described in section 5.0.
5.0 Timeline
Date Oct 1st Oct 18th Oct 25th Nov 1st Nov 15th Dec 1st Jan 24th Mar 7th Milestone Design and basic layout of software system complete with UML diagrams Basic rendering engine framework with input/output functionality - Retrieve an image from the camera, distort it and display it using the projector. Border corner detection - Detect the borders of the visible image and draw a frame to display them Basic keystone correction on flat surfaces. - Display an image on at flat wall in its proper shape. Luminance/Colour level detection. - Use the camera to gather colour data of an arbitrary wall and generate a lookup table for use in the color correction process. Luminance/Colour compensation. - Display an image on an arbitrary flat wall in its intended form. Advanced keystone correction on wall intersections. - Project images into a corner and have it appear correctly. Advanced keystone correction on curving surfaces. - Project an image on a curved surface with minimal distortions.
6
6.0 Requirements
For the development of the project to proceed, the following equipment is required: a computer with an attached projector, and a camera. The projector needs to be freely adjustable and not mounted on anything since it will need to be tested against various walls and surfaces. A suitable camera has already been acquired; however a projector and a computer for testing are still needed. These resources will not be needed all the time, and a lot of the development can be done without them using software to simulate their functionality.
7.0 Deliverables
When the project is complete, the following deliverables will accompany the final report: Fully commented source code in a CVS tree User manual detailing setup instructions as well as system usage Design documentation including UML diagrams and details on how the project was implemented A demonstration package showing the advantages of the Smart Projector technology including comparison images and performance data. These will allow future developers to quickly acquaint themselves with our architecture and implement any future requirements that arise.
7