Technology Evaluation Centers, Inc. Request for Information/Proposal Requirements Research
Instructions
RFI Rating Legend Format
● The top of the RFI tab contains six response columns. ● TEC analysts developed the RFI based on past experience and research into the widest and deepest range of possible requirements. 3RD CST FUT NS Priority Response Explanation SUP Supported as delivered "out-of-the-box" MOD Supported via modifications (screen configurations, reports, GUI tailoring, etc) Supported via a third party solution Supported via customization (changes to source code) Will be supported in a future release Not supported 0 to 10, where 10 is most important
Mandatory Yes, only for "must-have" factors
RFI Example Vendor Responses
● Complete the RFI worksheet by placing an X in the appropriate column for each criterion. ● The Xs represent the current state of a product or service. 1 1.1 1.1.1 1.1.1.1 1.1.1.2 1.1.1.3 1.1.1.4 1.1.1.5 1.1.1.6 Hierarchy Module 1 Category of Module 1 Subcategory of Category 1 Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 5 Criterion 6 Criterion
User Responses
● Use the Priority column to indicate how important a particular criterion or entire group of criteria (module or category) is for your organization. ● The Mandatory column is useful for indicating absolute requirements. Note that a "Yes" would mean a vendor/provider fails if it does not support that criterion or group of criteria. It is often useful to use "No" as the general default response and only change to a "Yes" for an absolutely critical item. Don't forget to indicate your priorities and mandatory requirements for modules, categories, subcategories, and criteria.
RFI Example
Priority (0-10) 9 5 10 1 5 5 10 7 9 Mandatory (Y/N) Y Y Y N Y N N N Y X X X X X X SUP MOD 3RD CST FUT NS
Technology Evaluation Centers, Inc. Software Test Tools Functional and Technical Worksheet
Hierarchy Criteria Priority Mandatory SUP MOD 3RD CST FUT NS (0-10) (Y/N)
1
Functionality
1 1.1
Used in Design Requirements Capture and Analysis Requirements under change control Audit trail and history of changes and versions Requirements uniquely identified and versioned Compares versions of requirements Customizable requirement template Enforceable rules on requirements Enables requirements to be linked to tests Enables requirements to be linked to code Enables requirements to be linked to documentation Link requirement to test results Tests may be linked back to requirements Requirements may be accessed and updated on-line Off-line updates may be merged back into central storage Authorization and access control Requirements may be extracted from existing documentation Changes can trigger e-mail alerts or notification of interested users Users customize views of information Changes may be proposed and published to interested parties before being accepted
1.1.1 1.1.2 1.1.3 1.1.4 1.1.5 1.1.6 1.1.7 1.1.8 1.1.9 1.1.10 1.1.11 1.1.12 1.1.13 1.1.14 1.1.15 1.1.16 1.1.17 1.1.18
1.1.19 1.1.20 1.1.21 1.1.22 1.1.23 1.1.24 1.1.25 1.1.26 1.1.27 1.1.28 1.1.29 1.1.30
Tool predicts impact of change Spelling and grammar checker Highlights ambiguities Highlights changes to requirements Graphical navigation of information Graphical display of links between information Integration with documents from MS Office On-line glossary of project definitions Sort requirements by cost, value, and priority Track requirement defects Supports and captures on-line discussion Cost/time requirements can act as constraints on other requirements Graphical display or creation of links Group of requirements at known versions can be baselined Visual Modeling This category contains 29 criteria below it. Used While Coding
1.1.31 1.1.32 1.2
2
2.1
Static Testing
2.1.1 2.1.2 2.1.3 2.1.4 2.1.5 2.1.6 2.1.7 2.1.8 2.1.9 2.1.10
Check against standard rules Check against customizable rules Suppress standards Spot undeclared variables Spot "dead" code Spot portability and compatibility problems Predicts run time errors Allows code to be corrected from within the tool Integrates with IDE and build tools Visual representation of code structure
2.1.11 2.1.12 2.1.13 2.1.14 2.1.15 2.1.16
Visual representation of module linkage Checks data flow between related modules Produce test cases for given level of code coverage Produce cyclomatic complexity metrics (code complexity) Produce Halstead metrics (data complexity) Produce code metrics: lines of code, comments, and executable lines Metrics represented visually Track metrics over time Runs unattended or in the background Overnight or batch analysis of many files or whole code set Predicts impact across multiple modules of code changes in one module Component and Unit Testing
2.1.17 2.1.18 2.1.19 2.1.20 2.1.21
2.2
2.3
This category contains 25 criteria below it. Code Coverage and Instrumentation
3
This category contains 7 criteria below it. Used While Testing
3.1
Automated Testing
3.1.1
Capture user input
3.1.1.1 3.1.1.1.1 3.1.1.1.2
3.1.1.1.3 3.1.1.1.4 3.1.1.1.5 3.1.1.1.6 3.1.1.1.7 3.1.1.1.8 3.1.1.1.9 3.1.1.1.10 3.1.1.1.11
User Activity Captured Keystrokes Keystrokes for power users--function keys, shortcuts, and tab between fields Keystrokes for special keys (shift and escape) Mouse actions for screen location Mouse actions for location relative to bitmap or object Mouse wheel Absolute timings of mouse actions and keystrokes Relative timings of mouse actions and keystrokes Contents of clipboard for copy/paste User activity on non-target applications Other user activity (pen, digitizer, joystick, non-keyboard buttons, and media insertion) Contextualize Captured Information This category contains 22 criteria below it. Optimized for Capture This category contains 47 criteria below it. Capture information and activity not triggered by the user This category contains 8 criteria below it.
3.1.1.2
3.1.1.3
3.1.2
3.1.3
Test Script Development
3.1.4
This category contains 64 criteria below it. Automated Test Execution
3.1.5
This category contains 34 criteria below it. Capture Output and Compare Results
3.1.6
This category contains 14 criteria below it. Summaries and Results
3.2
3.3
4
This category contains 11 criteria below it. Monitor Environment and Resource Usage ("Flight Recorders") This category contains 11 criteria below it. Modify Run Time Environment ("Fault Injection") This category contains 11 criteria below it. Test Support Tools
4.1 4.1.1 4.1.2
Test Planning and Management Supports manual testing Supports automated testing
4.1.3 4.1.4
4.1.5 4.1.6 4.1.7 4.1.8 4.1.9 4.1.10 4.1.11 4.1.12 4.1.13 4.1.14
Allows tests to be grouped into test suites for execution Allow tests or test suites to be planned against resources and testers Assign test execution task to a team member Assign non-test-execution task to team member Integration with test script execution tool, triggers tests Coordinates unattended automated testing Test plans held under change control Unique ID per test Unique ID per test execution Authorization required for access Customizable fields Display customized for tester or test manager, highlighting scheduled tasks and available actions Sort and filter test list Customizable reports Integration with requirements tool Supports third party or outsourced testing Allows remote access Testware held centrally Testware held on local machine only Testware held locally, but accessible from remote client Test Results Capture and Tracking This category contains 21 criteria below it. Defect and Enhancement Tracking This category contains 22 criteria below it. Test Generation This category contains 10 criteria below it. Generate and Manipulate Test Data This category contains 14 criteria below it.
4.1.15 4.1.16 4.1.17 4.1.18 4.1.19 4.1.20 4.1.21 4.1.22 4.2
4.3
4.4
4.5
5
General Functionality
5.1 5.1.1 5.1.2 5.1.3 5.1.4 5.1.5
5.1.6 5.1.7
Change Control and Configuration Management File locking functionality Check in and check out of test materials Allows and helps resolve multiple access to same file Holds and can derive full history for stored materials Access to stored materials restricted or enabled by user level and identification Materials held under change control are stored centrally Materials held under change control are distributed around network Stores text files Stores binaries Allows branches Allows sets of materials at particular versions to be tagged and seen as a group Controllability This category contains 4 criteria below it. Backup and Resilience This category contains 9 criteria below it. Access and Authorization This category contains 4 criteria below it. Infrastructure Integration with Other Tools Requirements analysis with test generation Test management with defect tracking Test management with test generation Automated execution with test management
5.1.8 5.1.9 5.1.10 5.1.11
5.2
5.3
5.4
6 6.1 6.2 6.3 6.4
6.5 6.6 6.7 6.8 6.9 7 7.1 7.1.1 7.1.2 7.1.3 7.1.4 7.1.5 7.1.6 7.1.7 7.1.8 7.1.9 7.1.10 7.1.11 7.1.12 7.1.13 7.1.14 7.1.15 7.1.16 7.1.17 7.1.18 7.1.19 7.1.20 7.1.21 7.1.22 7.1.23 7.1.24 7.1.25 7.1.26 7.1.27 7.2
Single-user automation with load/stress automation Code instrumentation with automated execution Test management with metrics gathering Integrated only with proprietary software Integrated with third party software Testable Platform Language and Protocol Support COBOL C C++ Java HTML Ada SQL .NET COM COM+ MTS Soap Objective C Visual Basic Fortran JavaScript PHP Python Perl Ruby Pascal PL/1 Natural JCL XML XSLT Other Fit with Target Deployment This category contains 7 criteria below it. Test Execution Architecture
7.3
8 8.1 8.1.1 8.1.2 8.1.3 8.1.4 8.1.5 8.1.6 8.1.7 8.1.8 8.1.9 8.2
This category contains 59 criteria below it. Ease of Use and Customizable UI Learning Curve Comprehensive on-line help Examples and samples supplied Designed for ease of use Specialist knowledge needed for dayto-day use Specialized knowledge needed to install and configure tool Technical tool for use by experienced software engineers Core functionality accessible with one hour's training Core functionality accessible with one day's training Core functionality accessible with one week's training End User Configurability This category contains 7 criteria below it. Architecture Platform Required Servers Windows Windows XP Windows 2000 Windows Server 2003 Windows 95 Windows NT 4.0 Windows Me Windows 98 Unix and Variants This category contains 7 criteria below it. Other This category contains 6 criteria below it. Client This category contains 12 criteria below it. Installation Architecture This category contains 6 criteria below it. Data Architecture This category contains 4 criteria below it.
9 9.1 9.1.1 9.1.1.1 9.1.1.1.1 9.1.1.1.2 9.1.1.1.3 9.1.1.1.4 9.1.1.1.5 9.1.1.1.6 9.1.1.1.7 9.1.1.2
9.1.1.3
9.1.2
9.2
9.3
9.4
9.5
10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9 10.1 10.11 10.12 10.13 10.14 10.15 10.16 10.17 10.18 10.19 10.2 10.21 11 11.1 11.1.1 11.1.2 11.2
Fit with Customer Development Style This category contains 7 criteria below it. Scalability This category contains 4 criteria below it. Industry Aerospace and defence Banking Education Electronics and engineering Exploration (GIS, oil, and gas) Financial Games Government Health Internet and web services Logistics Music and film Open source Publishing and media Retail Software development Telecommunications Utilities Wireless applications Academia Other Tool Characteristics Tool Age and Current Development Activity Commercial Availabilty Development Status Available Consultancy This category contains 23 criteria below it. Available Training This category contains 23 criteria below it. Licensing Options This category contains 10 criteria below it.
11.3
11.4
and Technical Worksheet
Comment/Description Tool functionality has been split by the classic "three-legged" architecture of development; design, coding, and testing--although tools used by one group can be highly useful to another, and some tools contain a range of functionality that covers many areas. Test support tools are used at all stages and levels of a project, and include tools to handle test management, test generation, defect management, and test data manipulation. The "general" section contains details of functionality that is not test-specific, but used by many tools. Design tools allow fundamental information about the system to be captured and shared, and are typically set up and used most intensely in the early stages of a project as the requirements, architecture, and designs are drawn up. Requirements will be shared with all levels of the project, and will change over time. Requirements capture and analysis tools allow project members to collaborate on defining the requirements, keep histories and definitive copies of requirements, and allow linkage between requirements and documentation, code and tests.
Visual modelling tools allow people to make models of the system, and to visualize those models in a variety of ways. Formal diagrams may be used to create models that can be converted into code structures, tests, and data schemas.
Test tools used during coding are often called "white-box" or "clear-box" tools. The tools are closely linked to the structure of the code, and tend to be used on individual pieces of code rather than integrated and configured systems. Many tools will only work with specific coding languages (covered in "infrastructure" below). They may be integrated with the build environment (IDE). Static test tools examine code without running it, and may be incorporated in the build process--running over the code at compile time, or as it is checked-in to the code base. They are an effective way of spotting a wide range of faults, which may be difficult to reproduce or diagnose at later stages. They are based on standards rather than any direct linkage to the functionality or requirements of the system, and while they will spot potential faults, they may also be configured to pass over desirable differences. Many also provide functionality to help developers visualize code structure, measure complexity, and assess impact of proposed changes.
Unit test tools allow a component of the system to be tested in isolation. They can act as a test harness (allowing code to execute as if it were supported by the system), test manager/driver (triggering automatic test execution, checking output, and recording pass or fail) and test stub (behaving like other system components that the tested component connects to). They may also monitor the component's use of resources, and provide further functionality to allow predictions of how the component will interact with a more complex integrated environment. Test-before-code techniques require a suite of unit tests that may be run on demand and typically must run without failure before changed code can be checked-in to the overall code base. These tools allow this task to be automated.
Coverage is a measure of how tested something is--if something is well-tested, most faults should have shown themselves. Statement coverage indicates the proportion of statements that have been executed in a given test. In order to measure this, the code is instrumented so that the tool can trace the lines that have been executed. Instrumentation may also be used to aid diagnosis and debugging. Statement coverage by itself is an inadequate measure of the testedness of a system, and more complex measures (branch, LCSAJ) may be required by safetycritical projects. Code instrumentation does not allow user-centric measurements of coverage (requirements, risk, and regulatory coverage).
Tools used during testing are typically "black-box" tools; they do not require information about the structure of the code to do their work. Instead, tests may be derived from a variety of sources; requirements, risks, regulations, fault models, critical-path, use cases and so on. Test automation puts a machine in charge of performing a known test and allows tests to be executed quickly, accurately, and repeated as desired--but these great advantages must be offset against higher initial costs, maintenance costs, and a reduced ability to spot unexpected problems. Flight recorders measure aspects of the system under test--the running code and the environment it is running in-and allow the measurements to be recorded, analyzed, and compared. They can give early warnings of unexpected systematic problems, and may be used in live operation as well as test. Fault injection tools simulate problems in the environment that the system is operating in. They allow tests that would otherwise be impossible or inconvenient, and can provide a fast and effective route to discovering problems that would be hard to show in an un-equipped test environment.
Testing is a broad discipline and in order to allow a machine to take the breadth of actions found in manual testing, test automation tools have to offer a similarly broad range of functions. Most test automation tools fall into three broad categories; automated unit testing, capture-replay, and load and stress tools. Unit test tools are used to support coding, and are dealt with in the coding section. Capture-replay tools seek to replicate the actions of a user, typically by generating a script that follows a user's actions and allowing that script to be modified and replayed. Load and stress tools also replicate a user's actions, but replay many users simultaneously. They are also known as volume test tools. Script construction lies at the heart of both of these approaches, and constructing a full set of automated scripts for a system may be a more complex software engineering task than constructing the system itself. Tools seek to compensate for this by processing captured user actions in the context of the application - by giving the user actions meaning in this way, the cost of writing and maintaining a script can be reduced. Although many tools place a virtual user firmly at the centre of their test, user actions are not enough by themselves to allow efective testing. Events that do not originate at the user or do not afect the graphical user interface are still important, and need to be captured by specialized tools or replayed through a A GUI often gives users a variety of ways to interact with the system--contextual menus and shortcut keys can duplicate expected mouse action, and must not only be captured, but also put into context. It is not enough to capture a "drag" between two points, if the action is a menu selection or drags an object into a container. The information must be contextualized--given meaning based on what was done to the system, not the interface. GUIs and browsers work in different ways; some tools support both, some work well only with one.
Capture tools need a context in which to work. Aside from operating systems and browsers, tools may also capture application-specific information, or be able to work with GUI delivery over a thin client or with standardized button sets.
Some tools can capture non-user activity. While not directly used for scripting and script generation, this functionality can be useful for synchronizing actions in more complex tests. Some functionality provided is similar to that from flight recorder tools.
Test script development is a software engineering task, and has its own languages, development environments, etc. Some of these are proprietary and some are in general use. Scripts may be constructed from scratch but commonly tools will translate user activity into a workable script, which may be modified, customized, and generalized. Maintenance is often an issue, particularly with GUIs and browser-based applications, and tools offer functionality to ease the pain. Many tools will also provide templates, wizards, and other assistance with common tasks. To fully test a system, the test tool may need to link with other applications. Some tools make this simpler than others.
Unattended test execution is the goal of many automated test projects. To support this, tools offer functionality to group tests into suites, to synchronize tests, and to deal robustly with failure. Load and stress tests may be closely monitored as they run, and unattended execution is less important. However, various techniques must be used to correctly simulate live operation without introducing test-related failures. Load and stress test tools will include functionality to simulate live operation more closely.
Automated test execution is useless without some way of spotting problems as they occur--and functionality to detect deviations from expected results is covered here. Expected results may be taken from many sources, but any unexpected problems will not necessarily be picked up by looking at expected results alone. Tools may have the facility to capture more than simply expected information, and compare those details with previous runs. This facility needs to be matched with a filtering capability to ensure that testing is not stopped in its tracks by an avalanche of minor differences.
Testing's primary goal is to produce information, but the large and undifferentiated raw output of automated testing may not be immediately useful at the point of production. It needs to be filtered, summarized, and stored for future retrieval--without knowing what will be important, and what will be useless.
Testing involves lots of lists, and tools to manage those lists can help the project go smothly. While some list managers are generic, many tools exist to support the most common lists found in testing; test lists and problem lists. Test management tools allow a list of tests to be grouped, sorted, and prioritized--helping the team to manage the work of testing. The results of that work (test pass/fail details, rather than test output data) need to be captured and summarized. Defect trackers hold a list of logged problems, and are used by most test teams. The tools allow problems to be classified and prioritized, and to be assigned to individuals as they pass through stages of a life cycle from detection to resolution. Test generation tools generate tests to fit certain parameters. While they are useful analytical tools for manual testing, they can be most efectively used to generate rule-based or model-based test scripts for test execution tools. Accurate and workable test data is vital to good testing, and data tools help significantly with analysis and manipulation of test data.
Software development tools have common needs. This section deals with the functionality that may be found in a range of test tools, and which is not specific to testing. Change control and configuration management is as important to test development as it is to code development. Controllability functionality allows test tools to be controlled by other applications, and is vital if the tool is to be integrated with any automated process. Backup functionality reduces the risk of data loss, whereas resilience helps ensure continuity of service from the test tool. Access and authorization protect information from unauthorized changes, and can also stop unauthorized people from seeing test results and problem logs.
This section covers technical and non-functional aspects of test tools. Test tools can support a single task, or the whole process of software development. Many vendors offer tools that can stand alone, but are more powerful when integrated into a larger suite of proprietary tools. Some vendors offer tools that are designed to integrate with third party tools.
Test tools that interact with the software under tests (generally those used while coding and testing, but not necessarily those used during design or as supporting tools) need to work with the platform on which the software is running. Clear-box tools--those used directly with code--are closely linked to the language that the code is written in. Tools that work on working code integrated into a system may not need to be linked to the language.
This section covers the way that the system under test (not the tool) is designed to be deployed.
Black-box test tools--usually used in testing--need to be able to make some sense of the environment that contains the system under test. Client-server systems typically run on two different machines--and the tool will need to cope with both. Tools that work on code alone, or those that work on the system in isolation (i.e. tools used while coding) may not need to be designed to work with the deployed system. Some tools, particularly load tools, act as a variety of clients and connect directly to middleware or the communications protocol. Some tools have specific functionality to allow efficient testing with common applications (typically CRM and ERP). If a test tool is to check the database that an application is writing to, it will need to connect to that database. Functionality to capture and parse information from various sources is dealt with above, in the functionality, automated testing section.
This section covers the users' experience of the tool. It contains two sections; learning to use the tool, and customizing the user interface to fit requirements.
This section covers the architecture of the tool as it is deployed (not the architecture of the system under test). It also contains related sections on the scalability of the tool, and its closeness of fit with structured methodologies. This section covers the platform required to run the tool
Although tools are generally not industry-specific, some tools have characteristics that are designed to support "best practices" in various industries.
This section covers aspects of the tool that are not "programmed in" but are characteristic of its maturity and place in the market.