Testing Voting Systems

Testing Voting Systems The Georgia Model Caltech-MIT Voting Technology Project October 1, 2004 Merle S. King Georgia – Snapshot – Georgia 2000 election had more residual votes than Florida – Collection of disparate technologies: punch card, lever machines, optical scan, and paper – First state to use computers to centrally tabulate votes (Fulton Co., 1964) – First state to implement a uniform, statewide DRE system (2002) – Deployment financed by state legislature Center for Election Systems Who we are: – Housed in the Department of Computer Science and Information Systems at KSU – Located on KSU campus in north-metro Atlanta, GA – 6 full-time staff – Graduate and Undergraduate students Center for Election Systems What we do: – Audited warehouse operations of vendor (2002) – Performs acceptance testing of equipment – Training and testing of election officials – Training poll managers – State of Georgia certification testing – Ballot building – Election day support – 700+ elections Testing Issues (general) – Testing is one of many overlapping and complementary controls that ensure correctness in systems – Building quality into products vs. testing quality into products – Preventative controls vs. detective and corrective controls – Tamper evident vs. tamper proof – Cost of control vs. value of asset protected Testing Issues (voting systems) – Election law U.S. Constitution Georgia Constitution – Article II, Section 1, Paragraph 1 HAVA 1965 Voting Rights Act Department of Justice oversight of Southern states Rules and Regulations of the Secretary of State of Georgia Emerging best practices (e.g. usability issues) Testing Issues (voting systems) – Multiple levels of testing: Federal, State, County & Precinct Different criteria at each level – Costs associated with testing cannot exceed pragmatic constraints Vendors ability to absorb and pass on excessive testing fees States have constrained budgets and domain issues Counties have limited and uneven resources to implement sophisticated regimens Testing Issues (voting systems) – Windows of opportunity to conduct testing and upgrades is very small – Evolution of testing protocol to address emerging variances and threats as well as expanded functionality of systems – System vs. unit testing – Staffing for testing – Qualifications – Peak demands Testing of Voting Systems • • • • • • • • Qualification Testing Certification Testing Acceptance Testing Ballot Testing Logic and Accuracy Testing Integrity Testing Parallel Monitoring on Election Day Certification of Election Officials Testing of Voting Systems Qualification Testing Performed on system by ITA in compliance with the EAC/FEC/NASED Voting System Standards. Qualified systems are deemed to be minimally capable of collecting, tabulating, and reporting vote totals. Documentation and system robustness meet minimum standards. Testing of Voting Systems Certification Testing KSU Center for Election Systems receives Qualified system and subjects it to rigorous (4-6 weeks) testing protocol that benchmarks the system to Georgia election code. Includes security tests, volume tests, recovery, exception handling, error handling, etc. We have an NDA with vendor to permit source code review. Testing of Voting Systems Acceptance Testing Once the vendor installs the EMS and delivers DREs and ancillaries, KSU CES performs on site testing of all equipment and software. Includes confirming hash signature of installed EMS Testing of Voting Systems Acceptance Testing – 2002 • 23,596 Touch Screen Units • 417 Optical Scan Units • 10,450 Encoders • 161 GEMS servers • 159 counties – four months • Failed ~1000 pieces of equipment • Created personal relationships with county election officials Testing of Voting Systems Ballots Center builds the ballots for ~100 counties and tests the ballots of the remaining 50+ Errors in style and content (statewide contests) are relatively easy to detect Splits and name spelling require research Testing of Voting Systems Logic and Accuracy Testing Prior to the election, L&A must be advertised in official county organ. Open to the public Demonstrates the ballot styles for the jurisdiction Historical significance Testing of Voting Systems Integrity Testing County election officials can verify the hash signature on the EMS at any time. Scripts and hardware to support DRE tests are available to county election officials Every time CES staff visit a county, we test the servers Testing of Voting Systems Parallel Monitoring on Election Day On election day, units will be pulled at random from counties and brought to the Center for parallel testing. Testing of Voting Systems Certification of Election Officials Georgia requires all election officials to receive 64 hours of training and be tested and certified by January 2007. Tests cover election law, ethics, EMS, DREs, election reporting, poll worker training, etc. Overview of Relationships Election System Vendor Qualified Federal Testing Laboratory (ITA) Trusted Organizations Counties Center for Election Systems Election System Vendor • Designs and builds the Election System • Creates system capable of passing Federal Qualification and State Certification tests • Submits the Election System to the ITA to verify compliance with Federal Voting System Standards • After completing Federal and State testing and receiving approval, installs the System in the counties Qualified Federal Testing Laboratory • Reviews the System for compliance with the Federal Voting System Standards • Issues Qualification Report on Complete System • Submits the Qualified System to the KSU Center for Election Systems where State Certification is performed KSU Center for Election Systems • Reviews the System for compliance with State of Georgia Election Code and Rules and makes recommendation to Secretary of State • Conducts Acceptance tests on installed systems (and replaced/repaired units) • Builds and tests ballots • Provides training to state election officials and certifies • Parallel monitoring on election day County Election Official • Maintains, stores and protects the System • Uses the System in accordance with Georgia law and rules to conduct elections. • Conducts Logic & Accuracy Test • Performs Integrity Tests on EMS Protecting System Integrity Three distinct functions must be performed to protect the integrity of the System: 1. Verify the System at Receipt. 2. Verify the System at Installation. 3. Verify the System in Operation. Function #1 Verify the System at Receipt. Using the System as delivered from the ITA • Set up and conduct sample elections with known outcomes that are representative of Georgia general and primary elections. • Conduct high-volume tests to determine capacity limits of the System. • Conduct tests to determine the System’s ability to recover from various types of errors. Function #2 Verify the System at Installation. Ensure that the System installed in the Counties is identical to the System received from the ITA and certified by the State. • Prepare a validation program that will detect any changes to the EMS installed in the Counties. • Run the validation program against the System installed in the County (after vendor installation). • Provide the County with a copy of the validation program. Function #3 Verify the System in Operation. Ensure that the System is performing properly, that all precinct ballots are correct and that the System has not been modified in any way. • Logic and Accuracy Tests are performed prior to each election. • Performance of all System components is verified. • Specific ballot information for each memory card in each precinct is verified. • Touch screen units are set for election, locked, and sealed. • Validation program is run after any suspicious event. Testing Relationships Election System Vendor Qualified Federal Testing Laboratory Trusted Organizations Function #3 Function #2 Function #1 Counties KSU Center for Election Systems Testing - Observations • Good testing cannot replace good design – but it can improve subsequent design • Testing voting technologies requires multifaceted approach • • • • Knowledge of technology Knowledge of election law and code Project management Multi-level partnerships: State, Counties, Vendor(s), ITAs, EAC, NIST, etc. Testing - Observations • Need to improve testing methods and efficiency of testing DREs • Need to improve testing of ballots, especially compliance with district boundaries • Need improved methods of dealing with COTS issues, including upgrades and enhancements Center for Election Systems http://elections.kennesaw.edu mking@kennesaw.edu Kennesaw State University is a unit of the University System of Georgia The Center for Election Systems receives its funding from the Georgia Legislature, through the Office of the Secretary of State