PRISM: Planning a Center for
Perceptual Robotics, Intelligent Sensors & Machines at the City College of New York
Zhigang Zhu Visual Computing Laboratory Department of Computer Science City College of the City University New York http://www-cs. ccny.cuny.edu/~zhu/ John (Jizhong) Xiao Robotics and Intelligent Systems Lab Department of Electrical Engineering City College of the City University New York Website: http://www-ee.ccny.cuny.edu/www/web/jxiao/jxiao.html
C
CVC L
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
2
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
3
C
City College of New York
CVC L
CUNY System w/ Vision and Robotics
CUNY
City College Graduate Center
Brooklyn
Hunter
Queens
…
Flagship campus HSI / MI
SOE
(PhD granting institution * )
* Site of PhD programs in Engineering
ME
CS
EE
PRISM Center
4/28/2009
Planned Center for Perceptual Robotics, Intelligent Sensors and Machines under NSF MII planning project Grant No. CNS-0424593
NSF Infrastructure 2005 4
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
5
C
City College of New York
CVC L
Research Theme and Threads
One Research Theme: RISE-NET
Robotized Intelligent SEnsor NETworks in a 3D space w/ air, ground, wall-climbing robots Perceptual Robots – navigate, network & plan
Four Research Threads
For surveillance & inspection For surveillance, inspection & education For human interfaces in surveillance & advanced e-learning
Intelligent Sensors – multimodal, multi-level
Smart Machines – HCI - M.I and H.I.
Reliable Communication
Ad hoc networks, Multimedia networks
More Detailed Technical Experiences
Pervasive Computing Workshop
NSF Infrastructure 2005 6
4/28/2009
C
City College of New York
CVC L
Perceptual Robotics: Mobile Sensor Network in 3D for surveillance and security
Joint Work with George Wolberg, John Xiao @ CCNY Bob Haralick @ CUNY Graduate Center Ed Riseman, Al Hanson, Rod Grupen @ UMass Tom Huang @ UIUC Supported by NSF, DARPA,ARO, NYSIA, ACTI, etc
PRISM Center Thread 1
4/28/2009
NSF Infrastructure 2005
7
C
City College of New York
CVC L
Goal
Dream:
transform the present 2-D world of mobile rovers into a new 3-D universe – air, ground, under-object, and wall-climbing move on ground, climb walls, walk on ceilings, transit between surfaces.
Applications:
Urban warfare applications: surveillance and reconnaissance,
weapon delivery, guiding perimeter around a building, etc Security and counter-terrorist applications: intelligence gathering about a hostile situation within a building, etc. Inspection and maintenance applications: routine inspection of buildings, nuclear containment domes, and other hard-to-reach places, inspection of aircraft, sand blasting of ship hulls, etc. Other Civilian applications: assistance in firefighting, search and rescue operations, etc.
4/28/2009
NSF Infrastructure 2005
8
C
City College of New York
CVC L
Some Real Examples
Robot Navigation (DARPA, ARO)ground video: medium range (~100ft)
City/Urban modeling (NSF/AFRL/NYSIA) - airborne video: far-view (1000 ft)
Under-vehicle inspection (ACTI) - car drives over camera: near-view (< 8 in)
Facade inspection (ARO) – climbing robots: near-view (inches?)
4/28/2009
NSF Infrastructure 2005
9
C
City College of New York
CVC L
Objectives
Rapidly create large FOV image mosaics that are
geo-referenced, with 3D (stereo) viewing and with all dynamic targets identified
As a light UAV flights over an area
4/28/2009 NSF Infrastructure 2005 10
C
City College of New York
CVC L
Region-Based Stereo Matching
Left mosaic – Full Depth Map
4/28/2009 NSF Infrastructure 2005 11
C
City College of New York
CVC L
Major Robots Acquired/Fabricated
Ground Robots at CCNY
Wall-Climbing Robots at CCNY
air out Vacuum motor Exhaust (inner) air out
Exhaust (outer) Vacuum impeller
air in
4/28/2009
NSF Infrastructure 2005
12
C
City College of New York
CVC L
Robot Localization and View Planning
Mutual awareness Heterogeneous sensors Active Sensing View Planning
for tracking and recognition
4/28/2009
NSF Infrastructure 2005
13
C
City College of New York
CVC L
PRISM Center Thread 2
Intelligent Sensors: Multimodal Human Signatures
for Surveillance & Inspection
Joint Work with Weihong Li, George Wolberg (CCNY) Ning Xiang (RPI), Tom Huang (UIUC) Lt. Jonathan Lee, Mr. Robert Lee (AFRL) Supported by AFRL, CUNY RF Publications : OTCBVS05
4/28/2009
NSF Infrastructure 2005
14
C
City College of New York
CVC L
Multimodal Human Signatures
Prof. Tom Huang at UIUC Prof. George Wolberg
Collaboration with
Goal
Remote hearing by human and machines Noisy LDV signals Low-res, non-front faces Target detection??
Figure 1. Multimodal remote hearing by human and machine: system diagram
Challenges
4/28/2009
NSF Infrastructure 2005
15
C
City College of New York
CVC L
Major Sensor Acquisition
OFV-505 Sensor Head, OFV-5000 Controller Tip-Tilt Stage VIB-A-P05 with Telescope ThermoVision® A40M Infrared Camera Canon VC-C50i Low Light IR PTZ Camera Omnidirectional cameras, stereo head Camcorders, webcam, CMOS sensors
Polytec Laser Doppler Vibrometer (LDV)
FLIR ThermoVision Camera
Canon Color/IR PTZ Camera
Other video cameras
4/28/2009
NSF Infrastructure 2005
16
C
City College of New York
CVC L
PRISM Center Thread 3
Smart Machines: Multimedia Integration
in a Virtual Classroom
Collaborators
Rick Adrion, Ed Riseman, Al Hanson, Jim Kurose, (UMass) Parviz Kermani, IBM / PINY Student: Weihong Li, Hao Tang, Chad McKittrick CUNY CISDD
4/28/2009 NSF Infrastructure 2005 17
Supported by
C
City College of New York
CVC L
Multimedia Virtualized Classroom
Goal: automating e-learning in MVC
Automatic capture (A, V, WB, BB) Intelligent media integration User-customized presentation
Problem and Promise
Large proposal with UMass declined by NSF CRI Keep Trying… NSDL Collaborative Research
with UMass who has a successful VIP on campus
MII -> CRI, more competitive
4/28/2009
NSF Infrastructure 2005
18
C
City College of New York
conversion
CVC L
VC: Automatic MM capture
index
PPT Capture Add-on
QuickT ime™ and a T IFF (Uncompressed) decompressor are needed to see this pi cture.
registration
Mimio Virtual Ink
QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.
video transformation
Qu ick Time ™ a nd a TIF F (U nc omp res sed ) de com pre sso r ar e ne ede d to see this pic ture .
QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.
QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.
Remote Reality Omnidirectional Camera
4/28/2009
NSF Infrastructure 2005
19
C
City College of New York
CVC L
VC: Media Integration
Toward an immersive VC interface
Active WWW link Synthetic Hi-Res digital display Slide combined with video
Panoramic mosaic from video
Annotated and active instructor
Question Window, ToC, etc
4/28/2009 NSF Infrastructure 2005 20
C
City College of New York
CVC L
VC: Media Integration
Instructor Extraction…and integration
video
image
shadow
4/28/2009
contour
NSF Infrastructure 2005 21
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
22
C
City College of New York
CVC L
PRISM Team
(15+ people in CS, EE &ME)
Prof. Myung Lee (EE)
Wireless sensor networks, ad hoc networks, Multimedia Networks Spoken language HCI, machine learning MEMS, micro device, biomechanics body/machine interaction Vision, graphics Robotics, control, sensor network Vision, multimodal sensors, HCI
4/28/2009 NSF Infrastructure 2005 23
Prof. Esther Levin (CS)
Prof. Ali Sadegh (ME)
Prof. George Wolberg (CS)
Prof. Jizhong (John) Xiao (EE)*
Prof. Zhigang Zhu (CS)*
* Co-PI of the MII project
C
City College of New York
CVC L
Faculty Involvement
Problem
MII/CRI limited funding for faculty support Heavy teaching load at CCNY Lack interest in student mentoring work
Call for proposals from mentoring faculty Obtain course release time (commitment from CCNY) Seek External Funding (NSF, DoD, etc)
4/28/2009 NSF Infrastructure 2005 24
Center allocates funding for small focused projects
Funding for faculty and student research*
C
City College of New York
CVC L
Current Funded Projects
Zhu, Wolberg, Haralick, Dynamic Pushbroom Stereo Mosaics for 3D and Moving Target Extraction. 02/15/05- 02/14/09, $267,437 (AFRL) Xiao, Zhu, Cooperative Wall-climbing Robots in 3-D Environment for Surveillance and Target Tracking, $375,237, 11/01/04-10/31/07 (ARO) Uyar, Xiao, Smart Reconfigurable Miniature Robot System Based on System-On-Programmable-Chip Technology”, $202,844, 09/04 –09/06 (NSF MRI)
Zhu, Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Displays, $94,076, 08/24/03 – 10/24/04 (AFRL)
4/28/2009 NSF Infrastructure 2005 25
C
City College of New York
CVC L
Current Funded Projects
Wolberg, Log-Polar Transforms for Optical Image Processing and Target Recognition, $600,000, 3/1/03 -10/1/06 (ONR) Lee (Co-PI), CTA ( Collaborative Technology Alliance), Wireless ad hoc networks, routing, MAC, multicasting, reliable transport, $2.5 M, 07/01-05/06 (ARL) Lee (Director), Samung-CCNY Joint Laboratory on wireless sensor networks , 2003-07 $1.2M (Industry) Sadegh, Northrop Gruman, $10K, Crown Corp., $52K, ALCOA Inc., $14K, Con Edison $8.5K. (Industry)
4/28/2009
NSF Infrastructure 2005
26
C
City College of New York
CVC L
Current Funded Projects
Zhu, Tracking Moving Objects via a PTZ or Moving Camera, $23,333, 08/01/04- 07/31/05 (NYSIA) Zhu, Wolberg, Integration of Laser Vibrometry, Infrared and Video for Multimodal Human Detection, $50,000, 02/19/04-02/18/05 (CUNY Research Equipment Grant) Xiao, Lee, Acquisition of equipment for multi-robot and mobile sensor networks research, $22,250, 7/05~7/06 (CUNY Research Equipment Grant)
4/28/2009
NSF Infrastructure 2005
27
C
City College of New York
CVC L
Pending Proposals to NSF…
Zhu, Wolberg, Huang (UIUC), Multimodal Remote Hearing for Humans and Machines, , $731, 776.00, 01/0/1/06-12/31/08 (NSF IIS) Zhu, Levin, Adrion & Riseman (UMass), Transforming Digital Video Libraries for Intelligent Access and Navigation, $240,499.00 (NSF NSDL) Levin, 2 NSF IIS proposals Grossberg, NSF CAREER proposal
…… NSF CRI proposal in progress
MII planning award ->CRI project
NSF Infrastructure 2005 28
4/28/2009
C
City College of New York
CVC L
Summary: Funding for Faulty Research
Overall > $5 M in the last 5 years Vision + Robotics > $1.6 M in the last 3 years New faculty involvement Active small-scale collaborations SOE at CCNY investment
Physical infrastructure, new science center, release time, lecture series, very attractive startups
=> A Center for multidisciplinary, large-scale collaborations in research and education
4/28/2009 NSF Infrastructure 2005 29
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
30
C
City College of New York
CVC L
Infrastructure
Existing Equipment
As described with the research threads Airborne and Outdoor Mobile Platforms
Wish List for CRI 2006
UAVs, outdoor robots Miniature and high-end cameras, acoustic GPS, INS , ladars, onboard processors Compact, high-end, low-cost, low-power Practical for imaging sensors and robots
Heterogeneous Sensor Packages
Various Levels of Vision Processors
Wireless Sensor Networks
4/28/2009
NSF Infrastructure 2005
31
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
32
C
City College of New York
CVC L
Education and Outreach
CCNY is both MI and HSI
Student retention
Undergrad, MS and PhD Complete the pipeline
New courses and academic programs
Outreach to K-12 students
4/28/2009
NSF Infrastructure 2005
33
C
City College of New York
CVC L
Education and Outreach (I)
CCNY is MI and HSI
1/3 African Americans, 1/3 Hispanic-Americans
Long distance commuters Majority of students hold part/full-time jobs Students retention attractive research and education programs
NSF Infrastructure 2005 34
Urban institution without dorms
Student faces financial hurdles
MAJOR challenge at CCNY
Solution ?
4/28/2009
C
City College of New York
CVC L
Education and Outreach (II)
Introduce new courses
video computing, mobile sensor network
Combine robotics, vision and mechatronics attract high school students to CISE disciplines Retain and timely graduate under-rep undergrads Support talented minority students for graduate ed.
Reshape capstone design courses
Create “Pipeline” to
All team members are on the newly established CpE program
4/28/2009 NSF Infrastructure 2005 35
C
City College of New York
CVC L
Education and Outreach (III)
NYC-LSAMP
New York Academy of Sciences (NYAS)
Louis Stokes Alliance for Minority Participation Summer Research Training Program (SRTP) For Inspiration and Recognition of Science and Technology Mentor high-school kids participating FIRST robotics competitions Organize workshops
New York/New Jersey FIRST
CCNY Robotics Club (organize outreach activities) Climbing Robot Competition 2005 in UK Affinity Mentoring
4/28/2009 NSF Infrastructure 2005 36
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
37
C
City College of New York
CVC L
Academic Outreach / Partnership
Advisory Committee
of experts in robotics, computer vision and sensor net
on computer vision, robotics and HCI Institutions with successful PRISM programs MI/HSI institutions
Lecture Series
Visits
4/28/2009
NSF Infrastructure 2005
38
C
City College of New York
CVC L
Advisory Committee
Roderic A. Grupen
Director, Laboratory for Perceptual Robotics, UMass-Amherst Chair of HCII, Head of IFP, Beckman Institute, UIUC
Thomas. S. Huang
Allen R. Hanson and Edward Riseman
Co-Directors, Computer Vision Laboratory, UMass-Amherst
Director, Collaborative Systems Laboratory, UMinn
Richard Volyes
Ning Xi
Director, The Laboratory of Robotics and Automation, MSU
USC, UCSD, UCF, CU…
4/28/2009 NSF Infrastructure 2005 39
C
City College of New York
CVC L
Lecture Series
Web page
http://www-cs.ccny.cuny.edu/ ~zhu/CcvcL/html/ccny_lecture_series.html excite interests, foster collaboration & increase visibility Huang at UIUC, Hanson/Riseman & Grupen at UMass, Shah at UCF Wittenburg at MERL, Kumar at Sarnoff ( CUNY GC events) Ning at MSU, Trivedi at UCSD , Medioni/Nevatia at USC and more
NSF Infrastructure 2005 40
Objectives
Lectures given
Lectures in plan
4/28/2009
C
City College of New York
CVC L
Visits to other institutions
Visits Made
USC IRIS and IMSC (vision, robotics and graphics labs)
Presentation by Z. Zhu
UCSD Robotics and Computer Vision Lab UMass Computer Vision Lab and Lab for Perceptual Robotics
Presentations by Z. Zhu and J. Xiao
In Plan
MERL (August for given a talk) CMU Robotics Institute (Fall) UPenn GRPASP Lab (Fall) U. Texas at El Paso (Fall) MSU, Uminn…
NSF Infrastructure 2005 41
4/28/2009
C
City College of New York
CVC L
Outcomes of Academic Outreach
All the three objectives – pleasant surprise
excite interests, foster collaboration & increase visibility
Faculty and students involvement Recognition by outside world Multiple Joint Proposals
UIUC, UMass, RPI USC, UT, etc in plan
NSF Infrastructure 2005 42
4/28/2009
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement Infrastructure Education and Outreach Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
43
C
City College of New York
CVC L
Evaluation Plan
Research achievement metrics and Educational effectiveness metrics
based on the corresponding NSF guidelines
Possible metrics
immediately tangible research results seeded by the funding (e.g., student participation, publications) improvement of minority students’ retention rates number of students entering graduate schools, etc
Under development…
4/28/2009 NSF Infrastructure 2005 44
C
City College of New York
CVC L
Outline
Who Are We & Project Goal Problems and Promises
Research Theme/Threads Team and Faculty Involvement (*) Infrastructure Education and Outreach (*) Academic Outreach / Partnership Evaluation Plan
Summary
4/28/2009
NSF Infrastructure 2005
45
C
City College of New York
CVC L
Summary
internal examination and feedback from external committee members confident in excelling our research and innovating the minority educational programs under the theme of the PRISM center: RISE-NET is critical for both research and ed., and this is well recognized to find mechanisms (funding!) to motivate talented researchers and educators to devote into PRISM center
Conclusions Bases
Recognition in Research and Education Programs
Hardware infrastructure
The most challenging issue
Goal - to make CCNY
a center of excellence in cutting-edge research and a national urban model for minority education
in the fields of robotics, computer vision and sensor network
NSF Infrastructure 2005 46
4/28/2009
C
City College of New York
CVC L
Thank you!
http://www-cs. ccny.cuny.edu/~zhu/ http://www-ee.ccny.cuny.edu/www/web/jxiao/jxiao.html
4/28/2009
NSF Infrastructure 2005
47