Optimizing Test System Development
Wyatt Meek
VISTA Product Manager
National Instruments Confidential
Agenda
• Boeing Test Engineering Priorities
• V I Engineering Overview
• Addressing Common Software Engineering Issues
– Configuration Management
– Project Management
– Reuse
• Discussion
Boeing Priorities
• What are the top initiatives within the test engineering
group?
• How do you currently measure success/failure?
• What issues does your group currently face?
• Are your suppliers required to follow
process/development guidelines?
History of V I Engineering
• Founded in 1992
• Select NI Alliance Member
• Four Locations
– Farmington Hills, MI (HQ)
– St. Paul, MN
– Indianapolis, IN
– Los Angeles, CA
• Business Units
– Engineering Services
– Engineering Information Management Systems (EIM)
– VISTA
Multi-Industry Experience
Simplified Waterfall Model for Software Design
requirements
research/ ship/
design coding testing
specification
definition maintain
All Too Often…
LabVIEW makes it very easy to quickly put something together
requirements
research/ ship/
design coding testing
specification
definition maintain
Developers do not follow a process
Case Study
Source: CMU/SEI-94-TR-013
Case Study
Source: CMU/SEI-94-TR-013
Common Software Engineering Issues
• Process Issues
– Configuration Management
– Project Management
This leads to poor code quality and
– Requirements Management
– Quality use of resources
inefficient Assurance (V&V)
• Development Issues
– Scalability
– Reusability
Addressing the Issues
• SEI (Software Engineering Institute www.sei.cmu.edu)
– SW-CMM
– CMMI
• IEEE Software Engineering Standards
(www.standards.ieee.org/software)
– 828, 830, 1028, 1220,1058.1
• ISO/IEC
– 9001:2000, 12207, 15288
• Industry Guidelines
– FDA
– DoE
– FAA
Process Implementation
• Process Analysis and Definition
– Prioritize improvement initiatives
– Define development process
– Develop templates and guides
– Select automation tools
• Implement Automation Tools
• Training
– Software development process
– Development and process improvement tools
– Common programming techniques and architecture
Challenges with Process Implementation for
LabVIEW and TestStand
• LabVIEW File Handling
– Embedded Source and Object
– Static Linkage
• Metrics
– Complexity
– Size
• Design Methodology
– Style
– Patterns
Configuration Management
Configuration Management
• Why use Configuration Management?
– To know what you’ve built (configuration)
– To identify and track needed changes
– To track status of configuration items
– To deliver known and verified configurations
• Four main Configuration Management Functions
– Configuration identification
– Configuration control
– Configuration status accounting
– Configuration authentication
CM Terminology
• SCC – Source Code Control
• Configuration Item – files placed in SCC
• Baseline – specific versions of CI’s at a specific time in the
product’s lifecycle (ie Beta Release, Rev 2.0.1)
• Status Accounting – knowing which version belongs in a
baseline and why
• Label – assigned as an attribute to a specific version of a file
or application, may be used to show status accounting (Alpha,
Beta, Reviewed, Verified, etc.)
Example CM Plan and Process
• Configuration Items (specifications, code, daq settings, support
files, exe’s)
• Configuration Control
– User roles (developer, lead developer, project manager, operations)
– Permissions (read/write access, labels)
– Change procedures
• Status Accounting
– Labeling procedures
• Authentication
– Audits
– Integrity checks
CM Recommendations
• Treat read only VIs as locked
• Establish common development environment
• Do not allow multiple checkouts (merging impossible)
• Avoid branching SCC projects (cross linking and merging)
• Do not use LLBs during development (can not locate
modifications within)
• Use labels during development (assign status)
• Deploy exe or use integrity check
CM Integration Solutions - demos
• Basic
– LV 6.0.2 PDS built in SCC
– SCC repository independent of LV
• Intermediate
– LV 6.0.2 PDS and newer built in SCC with VSS or Perforce
• Advanced
– VISTA CM Tools for LV 6.1 (Base, Full, PDS) and newer
with VSS or ClearCase (others in development)
Project Management
Importance of Metrics
• Confirm Estimates
• Project Monitoring
“You can’t manage what
• Cost Justification you can’t measure”
• Risk Analysis
• Impact of Change
• Track Improvement Initiatives
– Reuse
– Productivity
– Diagnose root cause of project failure
Standard Metrics (text based languages)
• Hours
• SLOC
• Function Points
• Cyclomatic Complexity
• EVA (Earned Value Analysis)
LabVIEW Metrics
• Hours
• SLOC Equivalent Metric
– Nodes (VI Metrics)
– GOBs (Graphical OBjects calculator)
• Cyclomatic Complexity (possible, but not yet available)
• EVA
TestStand Metrics
• Hours
• SLOC Equivalent Metric
– Lines in seq file
• EVA
Metrics Trials
• LabVIEW Developer Exams
– GOBs more reliable than nodes
• V I Engineering
– ~10 GOBs/hr for product development and regulated industries
– ~25 GOBs/hr for automotive solutions
• Ball Aerospace
– 8-10 GOBs/hr for 2 week to 2 man year projects
• General Dynamics (CMMI Level 3)
– 8.7 GOBs/hr avg for 8000 man hours of work
Variables
• Style
• Design Patterns / Architecture used (GOOP vs event
driven vs traditional)
• Quality Level Requirement
– Industry (regulated vs non-regulated)
– Application Type (product, manufacturing test system, DVT,
etc.)
Establishing a Project Plan with Metrics
• Review Requirements
• Decompose Requirements into Tasks/Components
– Complete preliminary design
– MS Project example WBS
• Estimate Development Tasks based on Historical Metrics and
Risk (spreadsheet example)
• Select Lifecycle
• Identify Quality Goals
• Assign Estimate to Non-development Tasks based on
Historical Metrics based on Quality Goals and Lifecycle
Establishing Project Control
• Identify Project Deliverables for each SLM Phase
• Build Master Project Schedule
• Assign Planned Value to Deliverables with Project
Schedule
– SW deliverables back-fire GOBs from hours estimates
Project Monitoring with Metrics
• Track % complete for Deliverables
– Use GOBs, SLM phase, status on tasks for objective
feedback on software deliverables
• Roll Up % Complete to EV
• Track EV vs. Planned Value, Actual Cost
Project Monitoring with Metrics
• Track % complete for
Development by LifeCycle Phase
DeliverablesWorkspace: Stopwatch Project Rev2.0
– Use GOBs, SLM phase for objective
feedback on software deliverables
• Roll Up % Complete to EV
Rework Design
11%
• Track EV vs. Planned Value, 19%
Actual Cost
Implementation Design
Development
by
• Manage19%Budget at Completion
Test
vs. Estimate at Completion
Development Implementation
Variance, etc. 23% Rework
Test
28%
Project Monitoring with Metrics
• Track % complete for GOBS vs Time
Stopwatch Project
Deliverables
– Use GOBs, SLM phase for
1000
objective feedback on software
800 Core Classes
GOBS (1000)
deliverables Plug Ins
600
• Roll Up % Complete to EV Database
400
Total
• Track EV vs. Planned Value,
200
Actual Cost
0
• Manage by Budget at 4 5
1 2 3 6 7 8
at
Completion vs. EstimateMonths
Completion Variance, etc.
Reuse
Coding for Reuse in LabVIEW
• Benefits of LabVIEW Reuse
• What is Required for Effective Reuse with LabVIEW?
• Reuse Tips
33
Benefits of LabVIEW Reuse
• Decreased Development Time
• Reduced Risk
• Improved Quality
• Lower Overall Cost
Requirements for Reuse
• Reuse Development Philosophy
– Reuse must be a part of the organizational culture
• Coding Standards / Style Guides
– Coding style needs to be familiar to entire team
• Training
• Source Code Control
Reuse Development Philosophy
• Always Evaluate Reuse Potential
• Generalize Code; Avoid project specific code
• Document for Reuse and Maintenance
• Have Reuse Reviews at the Beginning and the End of the
Project
36
CM and Reuse
• Avoid VIs with the Same Name
• Put LabVIEW Reuse Libraries in user.lib (treat them
similar to vi.lib)
– Organize VIs based on code categories
– Limit check-out/check-in access to reuse librarians
• Categorize LabVIEW Libraries
– General reuse, product line reuse, product specific
• Maintain Backward Compatibility
– Create a new VI before losing backward compatibility
Example Reuse Library Directory Structure in
user.lib
• ..\VIE Library.lib\ArrayTools.llb • ..\VIE Library.lib\SharedTYPE.llb
• ..\VIE Library.lib\ConfigUtilities.llb • ..\VIE Library.lib\Sound.llb
• ..\VIE Library.lib\ControlFunctions.llb • ..\VIE Library.lib\StateMachine.llb
• ..\VIE Library.lib\Dialogs.llb • ..\VIE Library.lib\StringTools.llb
• ..\VIE Library.lib\Time.llb
• ..\VIE Library.lib\ErrorTools.llb
• ..\VIE Module.lib\AI and Calibration.llb
• ..\VIE Library.lib\FileUtilities.llb • ..\VIE Module.lib\DTS Engine.llb
• ..\VIE Library.lib\GUI.llb • ..\VIE Module.lib\Formula Evaluator.llb
• ..\VIE Library.lib\GUI.llb\Soft • ..\VIE Module.lib\SecurityModule.llb
Keyboard.llb • ..\VIE Module.lib\System Integrity Module.llb
• ..\VIE Library.lib\NumericTools.llb
• ..\VIE Library.lib\OS Utilities • (.llb extensions are used for directory names
• ..\VIE Library.lib\OS Utilities\Windows.llb to be consistent if converted to VI library
files)
38
LabVIEW Style Affects Reuse
• Using a common style and good VI documentation can
significantly improve reuse
• LabVIEW Development Guidelines, chapter 6
– devstyle.pdf in ..LabVIEW\Search the LabVIEW Bookshelf…
• Use a code review checklist
• Avoid overly complex code
• Use templates to help standardize coding style
between multiple developers
39
Architecture Reuse Tips
• Use layered architectures; limit interaction to adjacent
layers
• Avoid project specific dependencies; use loose
coupling
– Use Graphical Object-oriented Programming (GOOP)
– Avoid global variables
– Use standard configuration tools
– Separate user-interface VIs from the main application
• Think in terms of reusable components
– Encapsulate functionality and data (use GOOP)
– Remember that how features specific to an architecture are
used may limit reuse to that architecture.
40
VISTA
Software engineering
approach to the
management, development
and release of LabVIEW
and TestStand software.
•Process Consulting
•Training
•Development and Process Tools
VISTA Configuration and Project Management Tool
• Workspace
• Metrics
• CM Functionality
– LV and TS integration
– Baselines
– Build with integrity (exe or native)
– File comparison
– File load check
VISTA Productivity and Reuse Tools
• Help Generator
• Renaming and
Relocation Tool
• Documentation Tool
• Reuse Library
• Style Guide
VISTA Quality Assurance
• Node/Path Tracker
• Cyclomatic Complexity
• Traceability Tool
VISTA GOOP Tools
• GOOP Inheritance Toolkit
– GOOP Wizard 3 with
Inheritance
– Icon Editor
– Development Distribution
Tool
GOOP Background
• Scalability – this non-OOP is not easy to grow or maintain
GOOP Background
• Scalability – layered, OOP approach
How Can V I Engineering Help You?
• Process Assessment
• Custom and Advanced Training
• Tools
• Project Support
• Architecture Design
Learn more at
www.vista.viengineering.com