The Martian Principles for Successful Enterprise Systems Best Practices by kyliemc

VIEWS: 11 PAGES: 78

									The Martian Principles for Successful Enterprise Systems

Best Practices from NASA’s
Mars Exploration Rover Mission

        Best Practices SIG
        March 7, 2007




Ronald Mak
CTO and Cofounder, Willard & Lowe
Presentation Agenda
       Mission Overview
       The Collaborative Information Portal
       Middleware Architecture

       Best Practices for Developing Reliable
       Enterprise Software

       Conclusion


March 7, 2007
                          Best Practices SIG
                                                2
NASA Background
       Senior scientist, Research Institute for
       Advanced Computer Science
       (RIACS), contracted to NASA
       Architect and lead developer of the
       middleware for the
       Collaborative Information Portal,
       2002-2003
       Supported the Mars Exploration Rover
       (MER) mission, 2004




      Academic appointment, University of California at
      Santa Cruz, contracted to NASA
      Architect, lead developer, and integrator of the
      Systems Health Information Portal, 2005

March 7, 2007
                                           Best Practices SIG
                                                                3
Mission Overview
Mars Exploration Rover Mission
       Twin robot geologists search for past liquid water.

       Launched: June 10 and July 7, 2003
       Landed: January 3 and 24, 2004
       Duration: 90 days
       (But after over three
       Earth years, both are
       still operating!)

       Mission Center:
       Jet Propulsion Laboratory
       Pasadena, CA

March 7, 2007
                              Best Practices SIG
                                                             5
Surface Operations




       Solar-powered rovers operate in daylight.
                On opposite sides of Mars, so there is always one in daylight
March 7, 2007
                                          Best Practices SIG
                                                                                6
Daily Process for Each Rover Team
                Receive downlink of data from a rover
                Process and analyze results
                Plan tomorrow’s activities
                Construct rover command sequence
                Send uplink of command sequence to rover




March 7, 2007
                                  Best Practices SIG
                                                           7
The mission has been a great success …




March 7, 2007
                 Best Practices SIG
                                      8
Key scientific discoveries

       Scientists now
       believe that
       liquid water
       did indeed cover
       parts of the surface
       of Mars in the past.

      More discoveries are
      being written up in
      prestigious scientific
      journals …

March 7, 2007
                               Best Practices SIG
                                                    9
The Collaborative Information Portal
Mission Needs
       Time
       management

       Data
       management

       Personnel
       management




March 7, 2007
                    Best Practices SIG
                                         13
Mission Needs: Time Management
       What time is it?
                Mission runs on Mars time
                Martian and earth time zones
                A Martian “sol” is 40 minutes
                longer than an earth day


       What’s happening?
                Two mission teams
                Work on several
                floors of a high rise




March 7, 2007
                                        Best Practices SIG
                                                             14
Mission Needs: Data Management
       What was planned?
                Hand-over process between science and engineering

       What actually happened?
                Correlate between planned and actual activities

       Where are the data?
                Large data repository
                    Flexible structuring
                    Security restrictions
                Get data products as soon as they are available
                Stove-piped, specialized analysis tools for data



March 7, 2007
                                        Best Practices SIG
                                                                    15
Mission Needs: Personnel Management
                What’s most important to me?
                  Different roles have different information needs

                What do I need to know?
                  Management needs to communicate with personnel

                When am I working? In what role? With whom?
                  Staffing is complex given the unusual work schedule
                  and large numbers of tasks and roles
                  Color coding: Spirit and Opportunity


March 7, 2007
                                   Best Practices SIG
                                                                     16
The Collaborative Information Portal (CIP)

Broadcast
messages

 Clocks
                                       Event
Tool tabs                              horizon




                                       Schedule
                                       viewer




March 7, 2007
                  Best Practices SIG
                                          17
Clocks
           Tick-over at top-of-minute




Mars time
displayed
in Sols
                                                               Local
                                                               time

UTC
shown
in Day                                                        Time at
of Year                                                       DSN
                                                              location

                                                             Time at
                                                             collaborator’s
                                                             location

March 7, 2007
                                        Best Practices SIG
                                                                       18
Schedule Viewer

                                                  Schedule
                                                  update
                                                  indicator


                                                 Multiple
                                                 time
                                                 scales




                                                 Unified
                                                 schedules




                “Now” bar
March 7, 2007
                            Best Practices SIG
                                                      19
Data Product Navigator




March 7, 2007
                Best Practices SIG
                                     20
  New Files



Duration


  Icons




   Files




  March 7, 2007
                  Best Practices SIG
                                       21
CIP is a Mission–Critical Application




March 7, 2007
                  Best Practices SIG
                                        22
CIP is used everywhere …
                                                         … in homes and offices




 … inside mission control




                  … in the science labs


March 7, 2007
                                    Best Practices SIG
                                                                                  23
Three–Tiered Enterprise Architecture
       Client
                Java application (Swing)

       Middleware Server
                Web services, Enterprise JavaBeans,
                Java Message Service
                Service-oriented architecture (SOA)

       Data Repository
                Database (Oracle)
                File monitor (Java application)
                Data loader (Java application)

March 7, 2007
                                    Best Practices SIG
                                                         24
Architecture Overview




March 7, 2007
                Best Practices SIG
                                     25
    CIP Middleware




Sunset on Mars
Middleware “ilities”
                Accessibility
                Scalability
                Extensibility
                Reliability
                Flexibility
                Adaptability
                Maintainability
                Security–ility



March 7, 2007
                                  Best Practices SIG
                                                       27
But the bottom line for middleware is …

       Make the client applications look good.

       Remain invisible to the users.

       “What server?”




March 7, 2007
                           Best Practices SIG
                                                 28
CIP Middleware Services
                User management                        Mars and Earth time
                Metadata                               File and directory
                Schedules                              Message




March 7, 2007
                                  Best Practices SIG
                                                                             29
Middleware Technologies
       Enterprise JavaBeans (EJBs) to achieve reliability, scalabilty,
       security, platform independence, and standards.
                Stateless session beans
                Stateful session beans
                Message–driven beans

       Web services to expose the remote methods of the Service
       Provider EJBs to the client applications.

       Java Message Service (JMS) for synchronous and
       asynchronous messaging.

       BEA WebLogic application server.


March 7, 2007
                                          Best Practices SIG
                                                                         30
Middleware Architecture
                                     WebLogic Application Server




                                          SOAP
                                                       Service Provider
                                        Processor       Remote Stateless
                                                          Session EJB




                                          SOAP
                                                       Service Provider
                                        Processor       Remote Stateless
                                                          Session EJB




                                                    • Stateless session
                                                      beans are the
                                                      Service Providers.


March 7, 2007
                Best Practices SIG
                                                                           31
Middleware Architecture
                                                                 WebLogic Application Server
                                 Web
                Application
                               Service
                  Java
                              Client Stub
                                                                      SOAP
                                                                                   Service Provider
                                                                    Processor       Remote Stateless
                                                                                      Session EJB
                                 Web
                Application
                               Service
                  Java
                              Client Stub




                Application
                                 Web
                                                                      SOAP
                                                                                   Service Provider
                               Service                                              Remote Stateless
                   C++                                              Processor
                              Client Stub                                             Session EJB




  • Web Services                                                                • Stateless session
    expose the remote                                                             beans are the
    methods of the                                                                Service Providers.
    Service Provider
    beans.
March 7, 2007
                                            Best Practices SIG
                                                                                                       32
Middleware Architecture
                                                    F
                                                    I                  WebLogic Application Server
                                                    R
                                 Web                E
                Application
                               Service             HW PS
                                                    TT
                  Java
                              Client Stub           A
                                                    L                       SOAP
                                                                                         Service Provider
                                                    L                     Processor       Remote Stateless
                                                                                            Session EJB
                                 Web               HTTPS
                Application
                               Service
                  Java
                              Client Stub
                                                   HTTP
                                                       S
                                                    F
                Application
                                 Web                I                       SOAP
                                                                                         Service Provider
                               Service              R                                     Remote Stateless
                   C++                             HTTPS                  Processor
                              Client Stub           E                                       Session EJB
                                                    W
                                                    A
                                                    L
                                                    L


  • Web Services                            • HTTPS encrypts                          • Stateless session
    expose the remote                         the transmissions                         beans are the
    methods of the                            and gets them                             Service Providers.
    Service Provider                          through the
    beans.                                    firewall.
March 7, 2007
                                                  Best Practices SIG
                                                                                                             33
Caching Data
            F     WebLogic Application Server
            I
            R
            E
            W
            A
            L
            L
                       SOAP
                                  Service Provider
          HTTPS                    Remote Stateless
                     Processor
                                     Session EJB



            F
            I
            R
            E
            W
            A
            L
            L




March 7, 2007
                                                Best Practices SIG
                                                                     34
Caching Data
            F     WebLogic Application Server
            I
            R                                                            Data
            E                                                        Local Stateful
            W                                                         Session EJB
            A
            L
            L
                       SOAP
                                  Service Provider                       Data
          HTTPS                    Remote Stateless                  Local Stateful
                     Processor
                                     Session EJB                      Session EJB




            F                                                            Data
            I                                                        Local Stateful
            R                                                         Session EJB
            E
            W
            A
            L
            L            Cache Registry




         • Stateful session beans cache data in memory.

March 7, 2007
                                                Best Practices SIG
                                                                                      35
Caching Data
            F     WebLogic Application Server
            I
            R                                                            Data
            E                                                        Local Stateful
            W                                                         Session EJB
            A                                                                         JDB
            L                                                                             C
            L
                       SOAP
                                  Service Provider                       Data
          HTTPS                    Remote Stateless                  Local Stateful   JDBC       Database
                     Processor
                                     Session EJB                      Session EJB

                                                                                             C
                                                                                       JDB
            F                                                            Data
            I                                                        Local Stateful
            R                                                         Session EJB
            E
            W
            A
            L
            L            Cache Registry




         • Stateful session beans cache data in memory.

March 7, 2007
                                                Best Practices SIG
                                                                                                        36
Event Notification
            F
            I   WebLogic Application Server
            R
            E
            W
            A                                                       “A new pancam
            L                                                       image file
            L                                                       has just been      File Monitor
                                                                    downloaded         Application
                                                                    from Mars!”




                                                                   JMS Server

                                                                       Schedules
                                                                          Monitor
                                                                         Topic
                                                                           Topic
                                                                           Resources
                                                                             Topic




        • Publish-subscribe model with topics


March 7, 2007
                                              Best Practices SIG
                                                                                                 37
Event Notification
            F
            I   WebLogic Application Server
            R
            E
            W
            A                                                       “A new pancam
            L                                                       image file
            L                                                       has just been      File Monitor
                                                                    downloaded         Application
                                                                    from Mars!”



            F
            I                                                      JMS Server
            R
            E                                                          Schedules
            W                       JMS                                   Monitor
                                                                         Topic
            A                     Consumer                                 Topic
                                                                           Resources
            L                                                                Topic
            L




        • Publish-subscribe model with topics


March 7, 2007
                                              Best Practices SIG
                                                                                                 38
Event Notification
                WebLogic Application Server


                                                                       “A new pancam
                                                                       image file
                                                                       has just been      File Monitor
                                                                       downloaded         Application
                                                                       from Mars!”
                                  User Proxy
                                Local Stateful
                                 Session EJB

                                                                      JMS Server
                  Message
                  Converter                                               Schedules
                                    JMS                                      Monitor
                                                                            Topic
                                  Consumer                                    Topic
                                                                              Resources
                                                                                Topic




        • Publish-subscribe model with topics


March 7, 2007
                                                 Best Practices SIG
                                                                                                    39
Event Notification
            F
            I     WebLogic Application Server
            R
            E
            W                     Service Provider
            A           SOAP                                             “A new pancam
          HTTPS       Processor   Remote Stateless
            L                       Session EJB                          image file
            L                                                            has just been      File Monitor
                                                                         downloaded         Application
                                                                         from Mars!”
                                    User Proxy
                                  Local Stateful
                                   Session EJB
            F
            I                                                           JMS Server
            R       Message
            E       Converter                                               Schedules
            W                         JMS                                      Monitor
                                                                              Topic
            A                       Consumer                                    Topic
                                                                                Resources
            L                                                                     Topic
            L




        • Publish-subscribe model with topics


March 7, 2007
                                                   Best Practices SIG
                                                                                                      40
Broadcast Message Service
            F     WebLogic Application Server
            I
            R
            E
            W           SOAP
                                  Service Provider
          HTTPS                    Remote Stateless
            A         Processor
            L                        Session EJB
            L



                                    User Proxy
                                  Local Stateful
                                   Session EJB
            F
            I
            R
            E        Message                                      JMS Server
            W        Converter       JMS
                                                                   Broadcast Messages
            A                      Consumer                               Topic
            L
            L




March 7, 2007
                                                   Best Practices SIG
                                                                                        41
Broadcast Message Service
            F     WebLogic Application Server
            I
            R
            E
            W           SOAP
                                  Service Provider
          HTTPS                    Remote Stateless
            A         Processor
            L                        Session EJB
            L

                                                                          Publisher
                                    User Proxy                          Local Stateless
                                  Local Stateful                         Session EJB
                                   Session EJB
            F
            I
            R
            E        Message                                      JMS Server
            W        Converter       JMS
                                                                   Broadcast Messages
            A                      Consumer                               Topic
            L
            L




March 7, 2007
                                                   Best Practices SIG
                                                                                          42
Broadcast Message Service
            F     WebLogic Application Server
            I
            R
            E
            W           SOAP
                                  Service Provider
          HTTPS                    Remote Stateless
            A         Processor
            L                        Session EJB
            L

                                                                          Publisher
                                    User Proxy                          Local Stateless
                                  Local Stateful                         Session EJB
                                                                                          Archivist          Message
                                   Session EJB                                              MDB       JDBC
                                                                                                             Archive
            F
            I
            R
            E        Message                                      JMS Server
            W        Converter       JMS
                                                                   Broadcast Messages
            A                      Consumer                               Topic
            L
            L




       • Message–driven bean (MDB) archives broadcast
         messages to persistent storage for later retrieval.

March 7, 2007
                                                   Best Practices SIG
                                                                                                               43
Broadcast Message Service
            F     WebLogic Application Server
            I
            R
            E
            W           SOAP
                                  Service Provider
          HTTPS                    Remote Stateless
            A         Processor
            L                        Session EJB                                             Delegate
            L                                                                             Local Stateless
                                                                                           Session EJB      JD
                                                                                                              BC
                                                                          Publisher
                                    User Proxy                          Local Stateless
                                  Local Stateful                         Session EJB
                                                                                            Archivist              Message
                                   Session EJB                                                MDB           JDBC
                                                                                                                   Archive
            F
            I
            R
            E        Message                                      JMS Server
            W        Converter       JMS
                                                                   Broadcast Messages
            A                      Consumer                               Topic
            L
            L




       • Message–driven bean (MDB) archives broadcast
         messages to persistent storage for later retrieval.

March 7, 2007
                                                   Best Practices SIG
                                                                                                                     44
CIP Data Repository Tier
Metadata Generation




      File changes are logged by nfslogd and the File Monitor, or found by the
      File Detector.
                A message is sent to the monitor topic for each file change
      The Data Loader generates metadata about the file
                Retrieves messages from the monitor topic
March 7, 2007
                                              Best Practices SIG
                                                                                 46
Best Practices for Developing
Reliable Enterprise Software
How Reliable Has CIP Been?
       The middleware code has not changed since the code
       freeze in early November 2003.
       CIP operational since mid–December 2003,
       two weeks before the first rover landed.

       CIP has run continuously for as long as 77 Earth days
       without interruption.
       As of the end of the day on December 31, 2005, the
       mission has lasted 17,904 Earth hours.

       During that entire period, CIP has been down less than
       10 hours, which is > 99.9% uptime.

March 7, 2007
                             Best Practices SIG
                                                               48
Software Architecture

       A software architecture is only as good
       as its implementation.

                If there are multiple implementations,
                then the architecture is only as good as
                its worst implementation.




March 7, 2007
                                 Best Practices SIG
                                                           49
Software Engineering
Software engineering is all about the 9 D’s:
                Discovery                          Debugging
                Diplomacy                          Documentation
                Definition                         Deployment
                Design                             Dmaintenance
                Development




March 7, 2007
                              Best Practices SIG
                                                                   50
Best Practice

       Use sound software engineering practices.

                Component–based architecture
                Design patterns


       Get buy–in from all development team
       members.


March 7, 2007
                               Best Practices SIG
                                                    51
Best Practice

       Do lots of user testing.

                Users are the best testers.
                Developers are the worst testers.
                Customers don’t know that they want until
                they see it.




March 7, 2007
                                Best Practices SIG
                                                            52
Best Practice
       Adhere to industry standards and use
       commercial off–the–shelf (COTS) software.

                Standards provide the most options.
                Software vetted and validated by industry.
                Large user and developer communities.
                Most documentation (“Dummies” books).
                Stable underlying “plumbing”.
                Don’t re–invent the wheel!

March 7, 2007
                                 Best Practices SIG
                                                             53
Best Practice

       The real challenge of enterprise development
       is not in the coding. It’s in the integration.


                Integrate early.
                Integrate often.
                Get something working as soon as possible.



March 7, 2007
                                Best Practices SIG
                                                         54
Best Practice

       Keep your services independent of each other.

                Plug and Play: Be able to swap services in
                and out as needed.
                Hot redeployment: Swap a service while it is
                running.




March 7, 2007
                                 Best Practices SIG
                                                           55
Best Practice

       Don’t hard–code parameters.

                Use external property files that are editable.
                No need to recompile and rebuild.




March 7, 2007
                                  Best Practices SIG
                                                                 56
Best Practice

       Make each service dynamically reconfigurable.

                Use external property files.
                During development, it’s impossible to
                anticipate all operational contingencies.




March 7, 2007
                                 Best Practices SIG
                                                            57
Best Practice

       Log everything!

                Each service has its own log.
                Watch what users are doing in real time.
                Determine what was going on before an
                anomaly occurred.
                Log mining: Analyze the logs to determine
                usage patterns and fine–tune parameters.

March 7, 2007
                                Best Practices SIG
                                                            58
Example Log Entries: Metadata Service
2003-11-06 05:03:57,788 INFO : jdoe: Metadata.query()
2003-11-06 05:03:57,813 DEBUG: SELECT file_view.* FROM MER_T.file_view WHERE
((file_view.seqnum = 1) AND (file_view.category = 'dataProduct') AND (file_view.owner =
'opgs') AND (file_view.type LIKE 'edr/%/PORT-6B' ESCAPE '\'))
2003-11-06 05:03:58,401 DEBUG: Records fetched: 11, skipped: 0
2003-11-06 05:03:58,657 INFO : jqpublic: Metadata.query()
2003-11-06 05:03:58,662 DEBUG: SELECT file_view.* FROM MER_T.file_view WHERE
((file_view.seqnum = 1) AND (file_view.category = 'dataProduct') AND (file_view.owner =
'soas') AND (file_view.type LIKE '%/rcam/PORT-6B' ESCAPE '\'))
2003-11-06 05:03:59,282 DEBUG: Records fetched: 3, skipped: 0
2003-11-06 05:04:02,485 INFO : mjane: Metadata.getObjectsByParent()
2003-11-06 05:04:02,490 DEBUG: SELECT * FROM MER_A.file_view WHERE (parent_pk = 1617105)
2003-11-06 05:04:03,143 DEBUG: Records fetched: 11, skipped: 0




       Timestamp, user name, method name
       SQL statement
       Information about the results

March 7, 2007
                                       Best Practices SIG
                                                                                     59
Example Log Entries: Streamer Service
2004-12-20 20:23:33,435 INFO : mjane: Streamer.putDataFile(/opt/bea/user_projects/cip/
conf/preferences/m.preferences)
2004-12-20 20:23:33,439 DEBUG: Begin upload of file '/opt/bea/user_projects/cip/conf/
preferences/sthompso.preferences'
2004-12-20 20:23:39,140 DEBUG: Completed upload of file '/opt/bea/user_projects/cip/conf/
preferences/sthompso.preferences': 35659 bytes
'/opt/bea/user_projects/cip/conf/global.properties': 13453 bytes
2004-12-20 20:28:57,516 INFO : jdoe: Streamer.getPreferences(user)
2004-12-20 20:29:29,721 INFO : jdoe: Streamer.getFileInfo()
2004-12-20 20:29:30,784 INFO : jdoe: Streamer.getFileInfo(/oss/merb/ops/ops/surface/
tactical/sol)
2004-12-21 19:17:43,320 INFO : jqpublic: Streamer.getDataFile(/global/nfs2/merb/ops/ops/
surface/tactical/sol/120/opgs/edr/jpeg/1P138831013ETH2809P2845L2M1.JPG)
2004-12-21 19:17:43,324 DEBUG: Begin download of file '/global/nfs2/merb/ops/ops/surface/
tactical/sol/120/opgs/edr/jpeg/1P138831013ETH2809P2845L2M1JPG'
2004-12-21 19:17:44,584 DEBUG: Completed download of file '/global/nfs2/merb/ops/ops/
surface/tactical/sol/120/opgs/edr/jpeg/1P138831013ETH2809P2845L2M1.JPG': 1876 bytes



     Timestamp, user name, method name
     Information about the uploaded or downloaded file

March 7, 2007
                                       Best Practices SIG
                                                                                     60
Example Graph from Log Mining




March 7, 2007
                 Best Practices SIG
                                      61
Best Practice

       Put in hooks for run–time, real–time monitoring.

                Middleware monitor utility is a
                normal client application.
                Quick overview of the server status.
                Expose the invisible.



March 7, 2007
                                 Best Practices SIG
                                                       62
Middleware
Monitor Utility

       Server statistics




March 7, 2007
                           Best Practices SIG
                                                63
Middleware
Monitor Utility

       Users




March 7, 2007
                  Best Practices SIG
                                       64
Middleware
Monitor Utility

     Files uploaded
     or downloaded




March 7, 2007
                      Best Practices SIG
                                           65
Middleware
Monitor Utility
       Cache contents




March 7, 2007
                        Best Practices SIG
                                             66
Best Practice

       Do lots of stress testing.

                Middleware stress tester utility is a
                normal client application.
                If you don’t find the limits of your middleware,
                your clients will …
                … at the worst possible moments.



March 7, 2007
                                  Best Practices SIG
                                                              67
Middleware Stress Tester

       Simulate multiple
       users performing
       client operations
       simultaneously.

       Can the server
       handle a heavy
       load?


March 7, 2007
                           Best Practices SIG
                                                68
Best Practice

       Isolate the build environment from the
       development environment.

                No dependencies on development tools.
                Don’t let a developer’s personal version of a
                library slip into the build.




March 7, 2007
                                 Best Practices SIG
                                                            69
Best Practice

       Understand the data usage patterns.

                Build an appropriate logical data model.
                Map the logical data model to a good
                physical data model (e.g., database
                schema).




March 7, 2007
                                 Best Practices SIG
                                                           70
Best Practice

       Keep it simple!

                Enterprise development is hard enough
                as it is …
                … whether on Earth or on Mars.




March 7, 2007
                                Best Practices SIG
                                                        71
Conclusion
Middleware Performance
       Enterprise JavaBeans and WebLogic
                Reliable: runs nonstop > 77 days at a time, > 99.9% uptime
                Scalable: handle heavy loads well
                Good performance: average response time ~0.1 second
                Flexible: plug–in, hot–redeployable services

       Web services
                Good interface between clients and EJBs
                Easy for clients to connect and request services
                Good performance: ~100 MB per hour throughput
                Language independent: Java and C++ client applications
March 7, 2007
                                      Best Practices SIG
                                                                             73
Summary

       Reliable software is good architecture plus
       good software engineering.




March 7, 2007
                           Best Practices SIG
                                                     74
CIP Development Team
                Roy Britten                           Carson Little
                Louise Chan                           Vish Magapu
                Sanjay Desai                          Ronald Mak
                Matt D’Ortenzio                       Quit Nguyen
                Glen Elliott (JPL)
                                                      Tarang Patel
                Bob Filman
                                                      John Schreiner
                Dennis Heher
                                                      Elias Sinderson
                Kim Hubbard
                Sandra Johan                          Joan Walton
                Leslie Keely                          Bob Wing (JPL)


March 7, 2007
                                 Best Practices SIG
                                                                        75
Official Mission Website



    http://marsrovers.jpl.nasa.gov/




March 7, 2007
                    Best Practices SIG
                                         76
Shameless Plug

                           Also:
                           Beautiful Code:
                              Leading Programmers
                              Explain How They Think
                           By Greg Wilson and Andy Oram
                           To be published by O’Reilly (June 2007)




March 7, 2007
                 Best Practices SIG
                                                                     77
Contact information

       www.apropos-logic.com
                Personal website


       www.willardlowe.com
                Consulting company website


       ron.mak@willardlowe.com
                E–mail address



March 7, 2007
                                     Best Practices SIG
                                                          78

								
To top