Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Test Generation for AssP

VIEWS: 6 PAGES: 20

									                                                          CRT:
Software Diagnostics and Conformance Testing




                                               Voting system logic testing
                                                        (Votetest)


                                                         David Flater
                                                         2008-01-23
                                                                           Players and field
                                               •   Congress
                                                    –   Help America Vote Act (HAVA)
Software Diagnostics and Conformance Testing




                                                    –   National Voting Rights Act
                                                    –   Section 508
                                                    –   Americans with Disabilities Act
                                               •   Election Assistance Commission (EAC)
                                                    – Voluntary Voting System Guidelines (VVSG)
                                                    – Manufacturer registration, lab accreditation, certification, …
                                               •   Technical Guidelines Development Committee (TGDC)
                                               •   NIST
                                                    – National Voluntary Lab Accreditation Program (NVLAP)
                                               •   Voting system manufacturers
                                               •   Voting jurisdictions
                                               •   State and local election officials
                                               •   Concerned citizens
                                               •   Professional advocates
                                               •   Academics
                                               •   Reporters & bloggers                      (and they all have lawyers)
                                                                   Logic testing in context
Software Diagnostics and Conformance Testing




                                               •   Manufacturer-driven activities
                                               •   Conformity assessment
                                                    – Physical configuration audit
                                                    – Documentation and design reviews
                                                    – Electromagnetic compatibility and
                                                      environmental testing
                                                    – Logic testing (Votetest)
                                                    – Volume test (mock election)
                                                    – CRT benchmarks
                                                    – STS and HFP testing
                                               •   Election Assistance Commission (EAC)
                                                   certification
                                               •   Jurisdiction acceptance testing and
                                                   certification
                                               •   Deployment
                                               •   Monitoring
                                                                                Goals

                                               •   Status quo: test labs are on their own to
Software Diagnostics and Conformance Testing




                                                   develop conformance tests for the Voluntary
                                                   Voting System Guidelines (VVSG)
                                               •   Conservative goal: reduce variability and cost of
                                                   testing by providing test labs with tools and
                                                   materials useful in constructing test suites
                                               •   Ambitious goal: further reduce variability and
                                                   cost by providing a canonical test suite
                                                                  Choosing the right tools

                                               •   “Testing target:” the object of conformity assessment
Software Diagnostics and Conformance Testing




                                                    – A.k.a. Implementation/Device/System Under Test


                                               •   Different kinds of testing targets need different testing
                                                   approaches
                                                       Differences from other testing targets
                                               •   Automatic testing is not feasible
                                                    – Don’t have standard interfaces to get data in and results
                                                      out
Software Diagnostics and Conformance Testing




                                                    – Voters are part of the process (people in the loop)
                                                    – Unanticipated nonfatal errors must be detected
                                               •   Cost of executing tests is a major issue
                                                    – Significant time and effort to prepare election
                                                      definitions, ballot styles, and test ballots or voters for
                                                      each test case
                                                    – Labor costs for people in the loop
                                                    – Politics: any increase in total cost for certification will be
                                                      considered an unfunded mandate
                                               •   More is not better
                                                    – A vote is a vote (logically)
                                                    – As we increase the number of votes counted
                                                         • Cost of testing increases proportionally
                                                         • Return on investment diminishes rapidly
                                               •   Context: one step in a long process
                                                    – Volume test (mock election), logic verification, etc.
                                                                                                          The requirements
                                                                     •   Normative reference: the next iteration of the VVSG (in public review)
                                                                     •   Logic must correctly handle all voting variations that the manufacturer claims
                                                                         to support
Software Diagnostics and Conformance Testing




                                                                     •   Everything must work through the complete elections and voting process

                                                                                                   Configuration and   Logic and
                                                                          Election     Ballot                                        Vote
                                                                                                     calibration of    accuracy                Tabulation   Reconciliation   Reporting
                                                                         definition   definition                                   gathering
                                                                                                      equipment         testing
                                                  1 of M voting              X            X               X               X           X            X              X             X
                                                  N of M voting              X            X               X               X           X            X              X             X
                                                Cumulative voting            X            X               X               X           X            X              X             X
                                               Ranked order voting           X            X               X               X           X            X              X             X
                                                 In-person voting            X            X               X               X           X            X              X             X

                                                 Absentee voting             X            X               X               X           X            X              X             X

                                                  Provisional /
                                                                             X            X               X               X           X            X              X             X
                                                challenged ballots
                                                    Write-ins                X            X               X               X           X            X              X             X
                                                 Review-required
                                                                             X            X               X               X           X            X              X             X
                                                      ballots
                                                Primary elections            X            X               X               X           X            X              X             X
                                                  Split precincts            X            X               X               X           X            X              X             X
                                                  Ballot rotation            X            X               X               X           X            X              X             X
                                               Straight party voting         X            X               X               X           X            X              X             X
                                                   Cross-party
                                                                             X            X               X               X           X            X              X             X
                                                  endorsement
                                                                           Testing strategy
                                               •   All tests are end-to-end tests that exercise the complete elections
                                                   and voting process
Software Diagnostics and Conformance Testing




                                               •   Small number (10-100) of carefully selected tests
                                                    – Cover each voting variation with a simple, synthetic test (around 10
                                                      ballots, 1 contest)
                                                    – Similarly cover all meaningful pairs of voting variations
                                                    – Few slightly larger tests (around 100 ballots, multiple contests) based
                                                      on real sample ballots
                                                    – Few miscellaneous tests (e.g., boundary cases)
                                               •   Test scripts to be “realized” according to the specifics of the target
                                               •   Test oracle
                                               •   No big tests in this test suite
                                                    – Context: The big volume test (mock election) provides a significant test
                                                      of all supported voting variations together
                                               •   Punt devilish details
                                                    –   Some requirements are too implementation-dependent
                                                    –   Some requirements are incidental to every scenario
                                                    –   Provided test descriptions but not test cases
                                                    –   Test lab is responsible for complete coverage
                                                                   Votetest release strategy
                                               •   First release
                                                    –   Based on draft VVSG
Software Diagnostics and Conformance Testing




                                                    –   “Basic test suite”
                                                    –   Tools and materials
                                                    –   Needs review and feedback
                                               •   Second release?
                                                    – If consensus is that basic test suite is not enough
                                                    – If there are problems to correct
                                                    – Sync with finalized VVSG (if applicable)
                                               •   Maintenance and support
                                                    – Keep up with VVSG maintenance (interpretations, errata)
                                                    – Correct operational issues and coverage gaps as they arise
                                                                        Votetest contents
                                               • Data model that supports all draft VVSG voting variations
                                               • SQL* schema that realizes the data model and the
Software Diagnostics and Conformance Testing




                                                 tabulation logic specified in the draft VVSG
                                               • Test cases formalized as SQL scripts
                                                    –   We don’t know the interface to the test target
                                                    –   SQL used as surrogate language
                                                    –   Execute as written on the supplied database
                                                    –   Must be translated into whatever is required by the test target
                                               •   Report generator to display results from test oracle
                                               •   Expected test results
                                               •   Documentation
                                               •   Bonus: test generator

                                               * Schema uses extensions to ISO SQL
                                                                           Test case execution
                                                                         Votetest environment
                                                 Test case (SQL)       Database       Report generator      Expected results
Software Diagnostics and Conformance Testing




                                                        Translate                                Compare




                                               Test case (translated) Voting system      Report generator     Actual results
                                                                     Voting system environment
                                                     Usability of logic test tools and materials
                                               •   Technical expertise befitting an accredited test lab is
                                                   assumed and required
Software Diagnostics and Conformance Testing




                                               •   Test cases formalized as SQL scripts
                                                    – More precise than informal test scripts
                                                    – Automated translation is possible
                                               •   The expected output from each test case is provided as a
                                                   plain text report
                                                    – Test lab does not need to get the infrastructure to run on their
                                                      machines to use the test scripts
                                                    – Sanity check for running installations
                                               •   No huge up-front investment
                                                    – Hardware requirements: one surplus PC
                                                    – Software requirements: all free software
                                               ######################################################################

                                                               BEGIN TEST CASE OUTPUT          2007-12-27 15:52:52-05        •   Print header
                                               ######################################################################
                                                                                                                             •   Reset database to
                                               $Id: 1-basic-1ofM.sql 415 2007-12-27 16:34:15Z dflater $
                                                                                                                                 baseline state
                                               Small 1-of-M contest, no write-ins, no rejected ballots.
                                               Ballot styles: 1                                                              •   Load test data
                                               Reporting contexts: 1
                                                                                                                             •   Run integrity
Software Diagnostics and Conformance Testing




                                               [... Integrity checks deleted ...]
                                               [... View materialization log deleted ...]                                        checks
                                               -------------------------------------------------------------------------------
                                               Report for context Precinct 1 generated 2007-12-27 15:52:52-0500

                                                                                            BALLOT COUNTS

                                               Configuration                                                Read   Counted
                                               -------------                                                ----   -------
                                               Total                                                          12        12
                                                                                              Blank            1         1   •   Generate report
                                               Precinct 1 Style                                               12        12
                                                                                              Blank            1         1

                                                                                             VOTE TOTALS

                                               President, vote for at most 1
                                               Car Tay Fower                                     4
                                               Tayra Tree                                        3
                                               Beeso Tu                                          2
                                               Oona Won                                          1
                                               Nada Zayro                                        0
                                               Overvotes                                         1
                                               Undervotes                                        1
                                               Counted ballots                                  12
                                               Balance                                           0
                                               -------------------------------------------------------------------------------

                                               Report total volume: 76
                                                 - Includes optional reporting of blank ballots.
                                                 - Excludes separate reporting of ballots cast vs. read.
                                                                                                                             •   Print footer
                                               ######################################################################

                                                                 END TEST CASE OUTPUT          2007-12-27 15:52:52-05

                                               ######################################################################
                                                                              The oracle




                                                                                                      X
                                               •   Design requirement is correctness not
                                                   performance
Software Diagnostics and Conformance Testing




                                               •   Logic model of draft VVSG translated as
                                                   transparently as possible into SQL views
                                                    – Limited expressiveness of SQL means fewer
                                                      ways to introduce faults (vs. programming)
                                                    – Good news: the logic model itself translates
                                                      with minimal overhead
                                                    – Bad news: straight party voting and write-in
                                                      reconciliation add a level of complexity
                                               •   Informal verification of correctness included in
                                                   documentation
                                               •   Demonstrated scalability up to 2 million ballots
                                               •   Results of simple tests are manually confirmed
                                               •   Test suite + saved output + shell script =
                                                   automated regression test
                                                                  Status as of 2008-01-23
                                               •   3 baseline tests (no optional voting variations required)
                                               •   19 single-variation tests covering 12 optional voting variations
Software Diagnostics and Conformance Testing




                                               •   66 two-variation tests covering 63 combinations of two voting
                                                   variations
                                                    – The other 3 combinations are not meaningful
                                               •   1 three-variation test
                                               •   3 tests based on sample ballots
                                               •   Total of 92 tests

                                               •   Working on documentation and presentation
                                               •   Could improve test generator and do more samples tests
                                               •   Needs NIST internal review, integration with other test efforts
                                               •   No public release yet
                                                                             Challenges
                                               •   Can’t review prior art—everything claimed as trade secret
                                               •   Draft VVSG is a moving target—Standards and Advisory Boards
Software Diagnostics and Conformance Testing




                                               •   Accretive release strategy—pressure to get it right the first time
                                               •   Realism—no two jurisdictions are alike
                                               •   Politics
                                                                       Demo—Disclaimers
                                                 •   For demonstration purposes only, we are about to execute a test
                                                     case in an emulated environment
Software Diagnostics and Conformance Testing




                                                 •   This configuration has problems and is not recommended for
                                                     production use
                                                 •   The nonfatal error shown below should be ignored



                                               could not remove file or directory "base/55958":
                                               Directory not empty
                                                                           Test case execution
                                                                         Votetest environment
                                                 Test case (SQL)       Database       Report generator      Expected results
Software Diagnostics and Conformance Testing




                                                        Translate                                Compare




                                               Test case (translated) Voting system      Report generator     Actual results
                                                                     Voting system environment
THE DEM
Software Diagnostics and Conformance Testing
                                               End of presentation

								
To top