"Remote Operation of The Two Keck 10-meter Telescopes"
Remote Operation of The Two Keck 10-meter Telescopes Robert Kibrick, Director of Scientific Computing, University of California Observatories / Lick Observatory Remote Instrumentation Discussion Spring 2002 Internet2 Member Meeting Overview of Presentation • Background – The Keck Telescopes – Motivation for remote operation • Remote observing with the Keck Telescopes – From remote control room at summit (30 meters) – From Keck Headquarters in Waimea, Hawaii (32 km) – From Santa Cruz, California via Internet2 (3200 km) • Operational models and issues 9/8/2010 2 The Keck Telescopes 9/8/2010 3 The Keck Telescopes 9/8/2010 4 Keck Telescope Facts • Twin, 10-meter optical/infrared telescopes • Largest telescopes of this type in the world • Construction funded by W. M. Keck Foundation • Observing time shared between 4 institutions – California Institute of Technology (Caltech) – University of California (UC) – National Aeronautics and Space Adminstration (NASA) – University of Hawaii (UH) • Located atop 4,200 meter summit of Mauna Kea 9/8/2010 5 Mauna Kea summit on the island of Hawaii 9/8/2010 6 Mauna Kea Summit • Dormant volcano • Premier astronomy site in N. Hemisphere –Above 90% of water vapor in atmosphere –Non-turbulent airflow over the summit –Sub-arcsecond atmospheric seeing • Home to many international observatories 9/8/2010 7 Mauna Kea Optical Observatories • UH 0.6 meter • UKIRT • UH 2.2 meter • Gemini North • CFHT • NASA IRTF • Keck-2 • Keck-1 • Subaru 9/8/2010 8 Common Facilities Operated by University of Hawaii • Dormitories at Hale Pohaku • Altitude is 2,800m • All water must be trucked in • Acclimatization required before ascent to the 4,200m summit 9/8/2010 9 Mauna Kea Observatories Serve An International Community • Argentina • Australia • Brazil • Canada • Chile • France • Japan • Netherlands • Taiwan • United Kingdom • United States 9/8/2010 10 Keck Telescopes use Classical Scheduling • Kecks not designed for queue scheduling • Schedules cover a semester (6 months) • Approved proposals get 1 or more runs –Each run is between 0.5 to 4 nights long –Gaps between runs vary from days to months –Half of all runs are either 0.5 or 1 night long • Separate schedules for the two Kecks 9/8/2010 11 Keck Telescope Schedule APRIL 2001 Keck I Semester 2001A Su 1 31 Wolfe, Prochaska HQ HIRESr6 UCSD CW RG U9H Wolfe Mo 2 22 Vogt " HIRESr7 UCSC " " U1H Vogt Tu 3 13 " " " " CW/MW/BS RGoc " " We 4 5 Segment Exchange HQ* PCS KECK " KECK Segment Exchange Th 5 0 " " PCS* " " " " Engineering/Butler, BS/MW* Fr 6 0 Marcy, Vogt* HQ* PCS/HIRESr8* KECK/NASA* RC KECK/N6H* Engineering/Butler* Sa 7 0 Butler, Marcy, Vogt* " HIRESr8* " MW/GS* RCoc " Butler* I-Eng/Butler, Marcy, GS* Su 8 0 Vogt* S/HQ* DSM/HIRESr8* KECK/NASA* RCoc* KECK/N6H* I-Eng/Butler* Mo 9 9 Stephens, Marley* " NIRC2 NASA " RC N43N Stephens Tu 10 19 " " " " " GWoc* " " Ellis, Brinchmann, GS/CS* We 11 28 Santos* " NIRC3 CIT RC C23N Ellis Th 12 38 " " " " CS* RCoc " " Fr 13 47 Harrison, Mao* " LRIS17 " " GW C35L Harrison Sa 14 55 " " " " " GWoc " " Su 15 63 Filippenko " LRISp18 UCB CS/MW* GW U25L Filippenko 9/8/2010 12 From 1993 to 1995, all Keck observing was done at the summit Observers at the summit work from control rooms located adjacent to the telescope domes 9/8/2010 13 Conducting observations involves coordinated effort by 3 groups • Telescope operator (observing assistant) – Responsible for telescope safety & operation – Keck employee; normally works at summit • Instrument scientist – Expert in operation of specific instruments – Keck employee; works at summit or Waimea • Observers – Select objects and conduct observations – Employed by Caltech, UC, NASA, UH, or other 9/8/2010 14 Keck 2 Control Room at the Mauna Kea Summit Telescope operator, instrument scientist, and observers work side by side, each at their own computer. 9/8/2010 15 Observing at the Mauna Kea summit is both difficult and risky • Oxygen is only 60% of that at sea level • Lack of oxygen reduces alertness • Observing efficiency significantly impaired • Altitude sickness afflicts some observers • Some are not even permitted on summit: –Pregnant women –Those with heart or lung problems 9/8/2010 16 Initiative to support remote observing from Keck Headquarters • 1995: Remote control rooms built at Keck HQ • Initial tests via 1.5 Mbps (T1) link to the summit • 1996: Videoconferencing connects both sites • Remote observing with Keck 1 begins • 1997: >50% of Keck 1 observing done remotely • Link to the summit upgraded to 45 Mbps (DS3) • 1999: remote observing >90% for Keck 1 and 2 • 2000: remote observing is now the default mode 9/8/2010 17 Keck 2 Remote Control Room at the Keck Headquarters in Waimea Observer and instrument scientist in Waimea use video conferencing system to interact with telescope operator at the summit 9/8/2010 18 Videoconferencing has proved vital for remote observing from Waimea • Visual cues (body language) important! • Improved audio quality extremely valuable • A picture is often worth a thousand words • Troubleshooting: live oscilloscope images • “Cheap” desktop sharing (LCD screens) • Chose dedicated versus PC-based units: –Original (1996) system was PictureTel 2000 –Upgrading to Polycom Viewstations 9/8/2010 19 Keck 2 Remote Observing Room as seen from the Keck 2 summit Telescope operators at the summit converse with astronomer at Keck HQ in Waimea via the videoconferencing system. 9/8/2010 20 The Remote Observing Facility at Keck Headquarters in Waimea • Elevation of Waimea is 800 meters • Adequate oxygen for alertness • Waimea is 32 km NW of Mauna Kea • 45 Mbps fiber optic link connects 2 sites • A remote control room for each telescope • Videoconferencing for each telescope • On-site dormitories for daytime sleeping 9/8/2010 21 The Keck Headquarters in Waimea Most Keck technical staff live and work in Waimea. Allows direct contact between observers and staff. Visiting Scientist’s Quarters (VSQ) located in same complex. 9/8/2010 22 Limitations of Remote Observing from Keck HQ in Waimea Most Keck observers live on the mainland. Mainland observers fly > 3,200 km to get to Waimea Collective direct travel costs exceed $400,000 U.S. / year 9/8/2010 23 Remote Observing from Waimea is not cost effective for short runs • Round trip travel time is 2 days • Travel costs > $1,000 U.S. per observer • About 50% of runs are for 1 night or less • Cost / run is very high for such short runs • Such costs limit student participation 9/8/2010 24 Motivations for Remote Observing from the U.S. Mainland • Travel time and costs greatly reduced • Travel restrictions accommodated –Sinus infections and ruptured ear drums –Late stages of pregnancy • Increased options for: –Student participation in observing runs –Large observing teams with small budgets • Capability for remote engineering support 9/8/2010 25 Fast and reliable network needed for mainland remote observing • 1997: 1.5 Mbps Hawaii -> Oahu -> mainland • 1998: 10 Mbps from Oahu to mainland • 1999: First phase of Internet-2 upgrades: –45 Mbps commodity link Oahu -> mainland –45 Mbps Internet-2 link Oahu -> mainland • 2000: Second phase upgrade: –35 Mbps Internet-2 link from Hawaii -> Oahu –Now 35 Mbps peak from Mauna Kea to mainland 9/8/2010 26 Mainland remote observing goals and implementation strategy • Goals: –Target mainland facility to short duration runs –Avoid duplicating expensive Waimea resources –Avoid overloading Waimea support staff • Strategy: –No mainland dormitories; observers sleep at home –Access existing Waimea support staff remotely –Restrict mainland facility to experienced –Restrict to mature, fully-debugged instruments 9/8/2010 27 Mainland remote observing facility is an extension of Keck HQ facility • Only modest hardware investment needed: –Workstations for mainland remote observers –Network-based videoconferencing system –Routers and firewalls –Backup power (UPS) – especially in California!!! –Backup network path to Mauna Kea and Waimea • Avoids expensive duplication of resources • Share existing resources wherever possible –Internet-2 link to the mainland –Keck support staff and operational software 9/8/2010 28 Keck software is accessed the same regardless of observer’s location • The control computers at the summit: –Each telescope and instrument has its own computer –All operational software runs only on these computers –All observing data written to directly-attached disks –Users access data disks remotely via NFS or ssh/scp • The display workstations –Telescopes and instruments controlled via X GUIs –All users access these X GUIs via remote displays –X Client software runs on summit control computers –Displays to X server on remote display workstation 9/8/2010 29 Why did we choose this approach? • Operational Simplicity • Operational control software runs only at the summit • All users run identical software on same computer • Simplifies management between independent sites • Allowed us to focus on commonality • Different sites / teams developed instrument software • Large variety of languages and protocols were used • BUT: all instruments used X-based GUIs 9/8/2010 30 Overall Topology 9/8/2010 31 Focus effort on X standardization and optimization over long links • Maintain consistent X environment between sites • Optimize X performance between sites • Eliminates need to maintain: • Diverse instrument software at multiple sites • Diverse telescope software at multiple sites • Coordinate users accounts at multiple sites • Fewer protocols for firewalls to manage 9/8/2010 32 Accessing Keck software and data from Keck HQ in Waimea Telescope operator uses display workstation at summit. Instrument scientist and observers use display workstations in Waimea. 9/8/2010 33 Accessing Keck software and data from the mainland Telescope operator uses display workstation at summit. Instrument scientist uses display in Waimea Observers use display on mainland 9/8/2010 34 Remote observing differences: Waimea versus the mainland • System Management: –Keck summit and HQ share a common domain –Mainland sites are autonomous • Remote File Access: –Observers at Keck HQ access summit data via NFS –Observers on mainland access data via ssh/scp • Propagation Delays: –Summit to Waimea round trip time is about 1 ms. –Summit to mainland round trip time is about 100 ms. 9/8/2010 35 Increased propagation delay to mainland presents challenges • Initial painting of windows is much slower • But once created, window updates fast enough • All Keck applications display to Waimea OK • A few applications display too slowly to mainland • System and application tuning very important –TCP window-size parameter (Web100 Initiative) –X server memory and backing store –Minimize operations requiring round trip transactions 9/8/2010 36 Simulating long propagations delays in the lab • Instruments are designed and built on mainland • Software is debugged on local area network • Testing on LAN does not reveal delay problems • Must measure delay effects before deployment • Simulate WAN delays using NIST simulator –Requires Linux PC with dual Ethernet interfaces –Can select specific packets delays, losses, jitters –http://www.antd.nist.gov/itg/nistnet 9/8/2010 37 Shared access and control of instruments • Most software for Keck optical instruments provides native multi-user/multi-site control • All users have consistent view of status and data • Instrument control can be shared between sites • Multipoint video conferencing key to coordination • Some single-user applications can be shared via X-based application sharing environments: – XMX http://www.cs.brown.edu/software/xmx – VNC http://www.uk.research.att.com/vnc 9/8/2010 38 Tradeoffs from this approach to remote observing • Disadvantages: –X protocol does not make optimal use of bandwidth –Long propagation delays require considerable tuning • Advantages: –Minimizes staffing requirements at mainland sites –Only “vanilla” hardware and software needed there –Simplifies sparing and swapping of equipment –Simplifies system maintenance at mainland sites –Simplifies authentication/access control 9/8/2010 39 End-to-end reliability is critical to successful remote operation • Keck Telescope time is valued at $1 per second • Each observer gets only a few nights each year • Observers won’t use facility if not reliable • What happens if network link to mainland fails? –Path from Mauna Kea to mainland is long & complex –At least 14 hops crossing 7 different network domains –While outages are rare, consequences are severe –Even brief outages cause session collapse & panic –Observing time loss can extend beyond outage 9/8/2010 40 Mitigation plan: install end-to-end ISDN-based fallback path • Install ISDN lines and routers at: –Each mainland remote observing site –Keck 1 and Keck 2 control rooms • Fail-over and fallback are rapid and automatic • Toll charges incurred only during network outage • Lower ISDN bandwidth reduces efficiency, but: –Observer retains control of observations –Sessions remain connected and restarts avoided –Prevents observer panic 9/8/2010 41 Summary of ISDN-based fallback path • Install 3 ISDN (6 B channels) between sites • Install Cisco 2600-series routers at each end –Dual auto-sensing Ethernet interfaces –Quad BRI interfaces • Inverse multiplexing • Dial-on-demand (bandwidth-on-demand) • Caller ID (reject connections from unrecognized callers) • Multilink PPP with CHAP authentication • Uses GRE tunnels and OSPF routing • No manual intervention needed at either end 9/8/2010 42 Current status • Prototype mainland facility operational at UCSC in temporary quarters • Enabled continuity of operation during week of September 11, 2001 (no flights to Hawaii) • Moving to larger, permanent facility in June • Similar facilities being assembled at other sites in California: –California Institute of Technology –UC San Diego –UC Berkeley 9/8/2010 43 Similar efforts at other observatories • IRAM 30-meter radiotelescope in Spain –Remote operation from Granada, Spain and Grenoble, France since 1998 • NASA Infrared Telescope Facility on Mauna Kea –Remote observing to observers’ home institutions –IRTF instruments successfully operated from Observatoire de Paris-Meudon during January 2002 • SOAR Telescope on Cerro Pachon in Chile –Remote Observing Center at Michigan State –High speed link via Internet-2 and REUNA 9/8/2010 44 Recent and upcoming conferences • Advanced Global Communications Technologies for Astronomy I – Munich, March 2000, SPIE Proceeding Vol 4011 • Advanced Global Communications Technologies for Astronomy II – Waikoloa, Hawaii, August 2002 • http://spie.org/info/as 9/8/2010 45 Summary • Internet-2 makes mainland operation feasible • Proposed model should be affordable: –Mainland sites operate as satellites of Keck HQ –Leverage investment in existing facilities and staff –Leverage investment in existing software –Share existing resources wherever feasible –Maintain Keck HQ as focal point for remote operation –Avoid wasteful duplication of resources –Avoid expensive and inefficient travel for short runs 9/8/2010 46 Acknowledgments • U.S. National Science Foundation • U.S. Department of Defense • University of Hawaii • Gemini Telescope Consortium • University Corp. for Advanced Internet Development (UCAID) • Corporation for Education Network Initiatives in California (CENIC) 9/8/2010 47 Author Information Robert Kibrick, UCO/Lick Observatory University of California, Santa Cruz California 95064, U.S.A. E-mail: firstname.lastname@example.org WWW: http://www.ucolick.org/~kibrick Phone: +1-831-459-2262 FAX: +1-831-459-2298 9/8/2010 48 www.internet2.edu