Docstoc

COTS

Document Sample
COTS Powered By Docstoc
					                                Key to Workbook
        Worksheet                                    Explanation

Summary                  Summary of Comments Received by Various Categories
All Comments - Alpha     All comments received on version 5.0, arrange alphabetically by
                         submitter.
All Comments - By Sec.   All comments received on version 5.0, arrange by section.



                         The remaining sheets in the workbook arrange comments by their
    Disposition Sheet    disposition.

Editing Committee        Comments to be resolved by the editing committee.
STG Assignment           Comments grouped by topics requiring committee or Special TG
                         resolution by group.
Sug. Resolution          Comments for committee review with a suggested resolution from the
                         editing committee.
TG - Security            Comments refered to the Security TG.
TG - Reliability         Comments refered to the Reliability TG.
TG - Usability           Comments refered to the Usability TG.
TG - Environmental       Comments refered to the Environmental TG.
TG - EMC                 Comments refered to the EMC TG.
TG - Software            Comments refered to the Software TG.
Need Wording             Comments returned to submitter for lack of specific suggestion for the
                         document.
No Change                Comments for committee review that the editing committee believes
                         require no change to the document.
                     Comments by Assignment
                        Section
                           --             Total
   Assignment          Special TG       Comments   Non-Editorial   Editorial
Editing Committee                      218
TG - Security            5.1, 6.1, C   274
TG - Reliability          5.2, 6.2     52
TG - Usability            5.3, 6.3     178
TG - Environmental        5.4, 6.4     36
TG - EMC                  5.5, 6.5     15
TG - Software             5.6, 6.6     131

Special TG's                           115
                     Prov. Ballot
                     COTS
                     VVPB
                     Cross Ref
                     TDP

Returned for
Specific Wording
No Change




        Total                          1019

                       Comments by Section
                                          Total
      Section         Sub-Section       Comments   Non-Editorial   Editorial
      General                          26
      Abstract                         6
         1                             4
         2                             14
         3                             52
         4                             60
         5                             485
                            5.1        188
                            5.2        45
                            5.3        112
                            5.4        14
                            5.5        22
                            5.6        111
          6                            137
                            6.1        18
                            6.2        9
          6.3   53
          6.4   18
          6.5   0
          6.6   44
   7            83
Annex A         6
Annex B         2
Annex C         90
Annex D         8     8
Annex E         2     2

 Total          975
                                                                       IEEE P1583
                                                             BALLOT COMMENT SUBMISSION FORM

                  Document: COTS
                  related
                  comments from
                  P1583 Draft
                  5.0 August
                  2003
#   Commenter and Clause/        Paragraph       Ty Comment                                       Proposed Change
    Number        Subclause      Figure/ Table   pe
1   Alice - 001   5.6.1.1                        T "source code generated by COTS code            Delete this clause
                                                    development package and embedded in
                                                    software modules for compilation or
                                                    interpretation shall be provided in human
                                                    readable form"    Some newer programming
                                                    tools do not necessary generate
                                                    traditional source code as reference within
                                                    this clause.
2   Corry-022    5.1.1          p.20, 3rd        T COTS may be properly installed and             Notwithstanding the fact that system
                                para., 3rd          configured but still not meet requirements    certifiers can rely upon the prior
                                sentence            unless latest security patches are            validations of the individual components of
                                                    installed.                                    the system [ ] provided they are properly
                                                                                                  installed and configured [with the latest
                                                                                                  security patches], there must still be an
                                                                                                  evaluation of the integrated system to make
                                                                                                  certain that security holes have not been
                                                                                                  left or created during the integration
                                                                                                  process.
3   Corry-023    5.1.1          p. 20, last      E   Last three sentences should be separate      Start new paragraph: [As] COTS products
                                para.                paragraph.                                   require updates due to a detected security
                                                                                                  breach or vulnerability [the] voting
                                                                                                  system vendor must provide a method to
                                                                                                  assess the impact of COTS updates on the
                                                                                                  voting system, as well as a method for
                                                                                                  providing notice and distribution of
                                                                                                  updates to purchasers[, testing facilities,
                                                                                                  and election officials and boards]. Where
                                                                                                  COTS products are known to be inherently
                                                                                                  risky ([e.g.,] memory leaks in the C++
                                                                                                  language), vendors must adequately describe
                                                                                                  the control methods they have employed to
                                                                                                  ensure these risks have been mitigated.
4   Corry-139    6.4.4.1        2nd para., 1st   T   Systems that are simply cobbled together     Delete first sentence of second paragraph.
                                sentence             (kluge might be a better description) from
                                                     COTS components must not be exempted from
                                                     environmental testing. I've had too many
                                                     problems with little doohickies hung on
                                                     some piece of otherwise great equipment
                                                     that caused problems when fielded.
5   df1         3         No. 26    E   COTS - "These devices and software are        Delete sentence. I do not believe that is
                                        exempted from certain portions of the         appropriate in a reference section defining
                                        qualification testing process so long as      COTS.
                                        such products are not modified in any
                                        manner for use in the voting system."
6   Dill-35     6.6.2               T   If COTS hardware or software is in the        Specify that the COTS exclusion only
                                        trusted subset, it must be treated exactly    applies to system components outside the
                                        like software or hardware designed by the     trusted subset.
                                        vendor.
7   Dill-7      3         Def #26   T   Explanation about exemption is unnecessary,   Delete last sentence of definition.
                                        and may become inconsistent if we add
                                        change requirements on COTS
8   Lipsio-12   5.1.1     Para. 5   T   The treatment of COTS products contradicts  Change “COTS product may” to “COTS products
                                        section 5.1.2.2, “Elements of Security      shall”. Mandate compliance with section
                                        Outside of Vendor Control”.                 4.3.11 (“Previously developed or purchased
                                                                                    software”) of IEEE Std 1228-1994, “IEEE
                                                                                    Standard for Software Safety Plans”.
9   Lipsio-14   5.1.1     Para. 7   T   There is implied a lack of testing in “COTS Mandate that testing preclude any security
                                        products require updates due to a detected breach or vulnerability; mandate compliance
                                        security breach or vulnerability”; nothing with section 4.3.11 (“Previously developed
                                        that requires an update should pass         or purchased software”) of IEEE Std 1228-
                                        testing.                                    1994, “IEEE Standard for Software Safety
                                                                                    Plans”. Mandate COTS be subject to the
                                                                                    specifications of IEEE Std 1008™-1987
                                                                                    (R1993), “IEEE Standard for Software Unit
                                                                                    Testing”. Add reference to IEEE Std 982.1™-
                                                                                    1988, “IEEE Standard Dictionary of Measures
                                                                                    to Produce Reliable Software”.
10 Lipsio-15    5.1.1     Para. 7   T   “The voting system vendor must provide a    Bring into conformance with Annex D (“V&V
                                        method to assess the impact of COTS updates of reusable software“) of IEEE Std 1012-
                                        on the voting system, as well as a method   1998, “IEEE Standard for Software
                                        for providing notice and distribution of    Verification and Validation”, e.g.,
                                        updates to purchasers” is inconsistent with “Reusable software (in part or whole)
                                        IEEE Std 1012-1998.                         includes software from software libraries,
                                                                                    custom software developed for other
                                                                                    applications, legacy software, or
                                                                                    commercial-off-the-shelf (COTS) software.
                                                                                    The V&V tasks of Table 1 are applied to
                                                                                    reusable software just as they are applied
                                                                                    to newly developed software. However, the
                                                                                    inputs for these tasks may not be available
                                                                                    for reusable software, reducing visibility
                                                                                    into the software products and processes.“
11 Lipsio-16    5.1.1     Para. 7   T   Memory leaks are the result of using C++    Eliminate “(ex. memory leaks in the C++
                                        language inappropriately; they are not a    language)”
                                        risk of a COTS C++ compiler.
12 Lipsio-3E    5.6.2.2   para. 1   T   Industry standard COTS compiler and runtime Require all tools, including compilers and
                                        interpreter both is not defined and assumes interpreters, to be validated and verified
                                        that, contrary to reality, something is     in the same manner as application software.
                                        fail-safe and fool-proof by virtue of being
                                        in common use.
13 Lipsio-43   5.6.2.3   Para. 1       T   “COTS software is not required to be          Eliminate the section, or, better yet,
                                           inspected…” is contrary to such other         reverse its sense.
                                           mission-critical methodologies as those
                                           used by the FDA and FAA, and contradicts
                                           what is specified in section 5.1.3.3.2
14 Lipsio-44   5.6.2.3   Para. 1       T   There is implied a lack of testing in “COTS Mandate that testing preclude any security
                                           products require updates due to a detected  breach or vulnerability; mandate compliance
                                           security breach or vulnerability”; nothing  with section 4.3.11 (“Previously developed
                                           that requires an update should pass         or purchased software”) of IEEE Std 1228-
                                           testing.                                    1994, “IEEE Standard for Software Safety
                                                                                       Plans”. Mandate COTS be subject to the
                                                                                       specifications of IEEE Std 1008™-1987
                                                                                       (R1993), “IEEE Standard for Software Unit
                                                                                       Testing”. Add reference to IEEE Std 982.1™-
                                                                                       1988, “IEEE Standard Dictionary of Measures
                                                                                       to Produce Reliable Software”.
15 Lipsio-45   5.6.2.3   Para. 1       T   There is implied a lack of testing in “the Bring into conformance with Annex D (“V&V
                                           most recent version of the COTS product     of reusable software“) of IEEE Std 1012-
                                           incorporating all security patches” ”;      1998, “IEEE Standard for Software
                                           nothing that requires an update should pass Verification and Validation”, e.g.,
                                           testing.                                    “Reusable software (in part or whole)
                                                                                       includes software from software libraries,
                                                                                       custom software developed for other
                                                                                       applications, legacy software, or
                                                                                       commercial-off-the-shelf (COTS) software.
                                                                                       The V&V tasks of Table 1 are applied to
                                                                                       reusable software just as they are applied
                                                                                       to newly developed software. However, the
                                                                                       inputs for these tasks may not be available
                                                                                       for reusable software, reducing visibility
                                                                                       into the software products and processes.“
16 Lipsio-4B   6.4.4.1   Para. 2       T   COTS hardware must have been tested to the Change paragraph to “COTS systems or
                                           rigor required of non-COTS components; if   components must be documented by their
                                           the supplier has not done this, then COTS   suppliers to have been tested to at least
                                           hardware must be treated like any other     the same rigor as required of voting
                                           component.                                  devices as specified hereinbelow; else, the
                                                                                       said COTS components shall be tested in a
                                                                                       like manner to any other component.”
17 Lipsio-4D   6.6.2     para. 3 & 4   T   “Unmodified, general purpose COTS non-      Eliminate the sections; ensure compliance
                                           voting software ...is not subject to code   with section 4.3.11 (“Previously developed
                                           examination...is not subject to the full    or purchased software”) of IEEE Std 1228-
                                           code review and testing” is contrary to     1994, “IEEE Standard for Software Safety
                                           such other mission-critical methodologies   Plans”.
                                           as those used by the FDA and FAA, and
                                           contradicts what is specified in section
                                           5.1.3.3.2.
18 Lipsio-6D    3.26                         E    Second sentence is not part of the            Delete the second sentence.
                                                  definition. Whether or not my later
                                                  comments on COTS are accepted, “These
                                                  devices and software are exempted from
                                                  certain portions of the qualification
                                                  testing process so long as such products
                                                  are not modified in any manner for use in
                                                  the voting system” does not belong in the
                                                  definition.
19 Lipsio-7A    5.1.1       Para. 7          E    It is unclear if “vendors” means “COTS        Change vendors” to “COTS vendors” or
                                                  vendors” or “voting equipment vendors” in     “voting equipment vendors”.
                                                  “vendors must adequately describe the
                                                  control methods they have employed to
                                                  ensure these risks have been mitigated.”
20 Lipsio-80    5.1.3.6.5                    E    COTS software was already covered in 5.1.1.   Eliminate “and software” from the first
                                                                                                paragraph and eliminate item “a”.
21 Lipsio-89    5.6.2                        E  “The software used by voting systems is
                                                selected by the vendor” appears to mean
                                                “COTS is selected”; else, it contradicts
                                                the subsequent sentence. Change the
                                                opening words from “The software” to “The
                                                COTS software”.
22 MercuriD50 - 4.6         Add bullet at    Ge It needs to be specified how updates to         * Documentation describing how an update is
   013 (formerly            end              ne software are going to be supplied and           to be certified and performed, should there
   mercuri-034)                              ra performed.                                      be a declared or discovered defect in the
                                             l                                                  voting system, software, hardware, or
                                                                                                firmware, or any COTS products used in or
                                                                                                in the development of the system that could
                                                                                                compromise its operation as an election
                                                                                                device.
23 MercuriD50 - 6.6.2       Paragraphs 2-4   Ge   The decision by the FEC to exempt COTS        Remove all exemptions for COTS product
   022 (formerly                             ne   products from inspection has created a        review from this standard on the grounds
   mercuri-048)                              ra   serious security flaw. It should not be       that such pose a serious security flaw.
                                             l    imperative that the IEEE standard continue    COTS products shall be presented in their
                                                  to reflect this inappropriate practice.       entirety for open review in the same way
                                                  All exemptions for COTS product review        that vendor software is examined.
                                                  should be removed from this standard.




24 MercuriD50 - 5.6.1.1     Section          Ge Concerns addressing use of COTS products        COTS products, especially software
   064 (formerly                             ne need to be added.                               libraries, are a vulnerable attack point
   mercuri-143)                              ra                                                 and must be subject to risks assessment
                                             l                                                  prior to use in voting products.
                                                                                                Configuration management should include
                                                                                                vendor updates and alerts when flaws are
                                                                                                detected that could compromise election
                                                                                                operations or cast ballot data integrity.
                                                                                                Object code modules should be provided such
                                                                                                that compiled versions of programs can be
                                                                                                compared.
25 MercuriD50 -   7.13                       Te   Provision is made in the standard for        System changes that have resulted from
   078 (new)                                 ch   update for COTS products releases, but       identification of insecure voting system
                                             ni   there is no such provision for updating or   components must be propagated to all
                                             ca   decertifying non-COTS voting system          systems currently deployed. (This might be
                                             l    components if such have been revealed to be  more appropriate in the configuration
                                                  insecure.                                    management section, or a different section
                                                                                               under maintenance.)
26 RGH 006        5.1.1     last paragraph   E    There is a change of gears just past the     Paragraph break with the sentence beginning
                                                  middle of the paragraph.                     "COTS products require updates…"
27 RGH 007        5.1.1     last paragraph   T    Memory leaks in C++ is not an example of an More appropriate would be "security
                                                  inherent risk in COTS products.              vulnerabilities in Microsoft products".
28 RGH 072        5.4       Second          T     COTS equipment will be entrusted with        Either require COTS equipment to comply to
                            paragraph above       counting votes but is exempted from this     the same standards as all other voting
                            clause 5.4.1.         standard with a "proven record of            equipment or remove the paragraph
                                                  performance"? OEMs of voting eqipment also altogether.
                                                  have "proven" track records but must still
                                                  test to this standard? This seems
                                                  unreasonable.
29 RGH 117        5.6.2.3                    T    "…COTS software …must be the most recent     Remove this clause.
                                                  verion of the COTS product …"     The most
                                                  recent version is not always stable enough
                                                  to deploy and may not be compatible with
                                                  the other aspects of the application.    The
                                                  vendor must have the latitude to employ the
                                                  COTS versions and upgrades at the
                                                  appropriate time.
30 schneidewind -3          Pg. 10           T    COTS Hardware and software should not be     Eliminate the exemption.
   002                      Line 26               exempted from qualification testing.
                            Definition 26
                                                  This exemption should not be included in
                                                  Definitions. The exemption is not a
                                                  definition.

31 schneidewind -5.6.2.3    Pg. 70           T    Why specify that COTS software must be        Either eliminate the requirement or inspect
   005                                            designed in a modular or object oriented      for compliance.
                                                  fashion and not inspect it for compliance?

32 schneidewind -6.4.4.1    Pg. 100          T    Why exempt COTS hardware from environmental Require environmental testing of COTS
   006                                            testing?                                    hardware.

33 schneidewind -6.6.2      Pg. 107          T    COTS software must work in conjunction with Eliminate the exemption of COTS software
   007                                            the voting application software. Therefore, from the testing requirement.
                                                  it should be subjected to the same rigor of
                                                  testing as the application software.
34 Simons - 002   5.1.1       the sentence    G   This is a far too vague and does nothing to Replace sentence with the following:
                              that reads,         address the security issues.                "Underlying products, such as operating
                              "The security                                                   systems, database systems, firewalls,
                              countermeasures                                                 network devices, web browsers, smart cards,
                              implemented by                                                  biometric devices, general purpose
                              an IT system                                                    application components, libraries, and
                              typically use                                                   hardware platforms, that are crucial to the
                              functions of                                                    correct and secure operation of the entire
                              the underlying                                                  system must be thoroughly tested. This
                              products and                                                    includes COTS systems. In addition, there
                              depend upon the                                                 must be a line by line code review of ALL
                              correct                                                         software that interacts with the voting
                              operation of                                                    system in any fashion. This is required
                              those products                                                  because of the potential risk of malicious
                              and their                                                       code."
                              security
                              functions."
35 Simons - 017   5.1.3.4.2   the entire      G   There is no way to adequately test against   Add the requirement that all COTS used in
                              section             all possible bugs and malicious code in      any voting system must be open source.
                                                  COTS.
36 Sklein-007     5.6         Para 5.6.1.1   T    Unmodified COTS must be evaluated at the     Delete “Unmodified third-party software is
                                                  source code level to protect against the     not subject to code examination; however,”
                                                  threats identified in 5.3.2.1 (A).           and replace it with “All third party
                                                                                               software shall be subject to source code an
                                                                                               d other examination to preclude the
                                                                                               presence of trap doors, hard-coded
                                                                                               passwords, vulnerabilities and other non-
                                                                                               deliberate errors, deliberate errors
                                                                                               allowing the introduction of malicious
                                                                                               code, and malicious code of any kind,
                                                                                               especially malicious code intended to
                                                                                               trigger upon use of the software in voting
                                                                                               systems.”
37 Sklein-044     5.6.2.3     First paragraph T   COTS must meet the requirements of 5.1.3.1   In the second sentence, after “security
                                                                                               requirements defined in” insert “Section
                                                                                               5.1.3.1 and”.
38 Sklein-045     5.6.2.3     First paragraph T   COTS virus detection programs are not        In the second sentence, replace the comma
                                                  available for all operating systems.         after “security patches” with “and”.
                                                                                               Replace “and must be tested” by “. In
                                                                                               complying with the requirement of 5.1.3.1,
                                                                                               the vendor must document how the COTS has
                                                                                               been defended against the threats
                                                                                               identified in 5.1.2.3 (A-1), (A-3), (B-1)
                                                                                               and (B-2), such as by testing”.
39 Sklein-051     Section 3   Item 26        T    Unmodified COTS are not exempt from          Delete the second sentence of the
                                                  evaluation to preclude the threats           definition.
                                                  identified in 5.3.2.1 (A).
40 Sklein-056   5.1.3         All           T   Voter verified paper needs to be mandatory   Add to the section created under comment SK-
                                                under certain circumstances                  4 above: A voter verified paper audit
                                                                                             trail is mandatory for any system in which
                                                                                             any of the following conditions is found:
                                                                                             1. Either the system software or any COTS
                                                                                             used as either a system component or
                                                                                             development tool, including compilers,
                                                                                             libraries, and other tools, is too complex
                                                                                             to clearly and thoroughly evaluate at the
                                                                                             source code level to ensure absence of
                                                                                             backdoors and other malicious code or means
                                                                                             of introducing malicious code. 2. All
                                                                                             other security, accuracy, integrity, and
                                                                                             availability requirements are not satisfied
                                                                                             clearly, easily, and without any question
                                                                                             or requirement for interpretation. 3. -
                                                                                             There are any reports or significant
                                                                                             suspicions that similar technology may have
                                                                                             failed to record all ballots exactly as
                                                                                             cast. 4. - There is any question whatever
                                                                                             about the ability of all using
                                                                                             jurisdictiions to easily and completely
                                                                                             satisfy all assumptions regarding
                                                                                             supervision of machines and relevant
                                                                                             personnel at all times machines are in use,
                                                                                             regarding fully secure storage of machines
                                                                                             between elections, and regarding other
41 Sklein-057   5.6.2.3       5.6.1.1       T   COTS evaluated should include compilers,     COTS to be evaluated shall include
                                                libraries, and any other software tools      compilers, libraries, and any other
                                                used in system development and capable of    software tools used in system development
                                                introducing backdoors or other malicious     and capable of introducing backdoors or
                                                code.                                        other malicious code.
42 VCW-02       5.1.1         2nd to last   Ed  The COTS products may also be subject to a   delete second space before "voting system"
                              para          it  security evaluation themselves; such
                                            or  evaluations can support the voting system
                                            ia  evaluation process.
                                            l
43 wfw -001     Section 3     26            Ed COTS, whether modified or not must be         I would drop the last sentence.
                Definitions                 it tested at least to system level.
                                            or
                                            ia
                                            l
44 PPLX-001   3         Section 3.      E    In discussing the definition of COTS, this    Remove the text in quotes.
                        Definition # 26      section goes on to say, “These devices and
                                             software are exempted from certain portions
                                             of the qualification testing process so
                                             long as such products are not modified in
                                             any manner for use in the voting system.”
                                             In general it is not a good idea to discuss
                                             policy in a definition. In particular,
                                             doing so here raises the question, which
                                             portions of the testing process are
                                             “certain” portions from which testing is
                                             exempted.
45 PPLX-035   5.6.2.3   5.6.2.3          T   This section of the draft has this            This section has several problems. The
                        Software         &   language:                                     module usage should be changed to
                        Modularity and   E   “However, COTS software is not required to    subrouting or function, remove the strict
                        Programming          be inspected for compliance with this         requirement of only one exit per subroutine
                                             requirement but must be the most recent       or function. Change so the most recent
                                             version of the COTS product incorporating     version of COTS is not required.
                                             all security patches,” [emphasis added]
                                             This section may be ambiguous.     Must the
                                             latest version always be incorporated or
                                             only the latest version of security
                                             patches? What if the security patch is not
                                             relevant to the particular operation.

                                             In any case, forcing the latest version of
                                             COTS software is a configuration control
                                             nightmare and will result in endless re-
                                             qualification. One interpretation of this
                                             section is that software written to run on
                                             Windows 2000 must be rewritten and re-
                                             qualified to run on Windows XP even if it
                                             runs perfectly well on Win2000. An even
                                             worse interpretation requires vendors to
                                             update hard disk controllers with new
                                             firmware and drivers every time a new
                                             software version is available. We don’t
                                             think this is intended nor desirable.

                                             Note the term “Module”. The term Module is
                                             used here as it is used in the FEC VSS and
                                             we believe this usage to be non-standard.
                                             A module should be a collection of related
Proposed Resolution                        Reference Information   Reference Information
(by the Chair on each comment submitted)
NC - Out of scope.   This is determined by    The lack of any specification
the relevent election authority, such as      regarding updates and
NASED or the individual state officials.      configuration management is a
                                              serious security flaw that
                                              must be addressed by the
                                              standard.


NC - Only unmodified COTS is exempted.      Unmodified COTS is not exempt
This is a drastic change that permeates the from serious security flaws,
spec and cannot be considered at this time. as evidenced in the continual
Also, if required, a vendor cannot control  update patches that must be
COTS source availability which would also   downloaded for Microsoft
limit vendor choices in system design.      operating systems, for
                                            example. The exemption for
                                            COTS products was erroneous
                                            in the FEC document and is
                                            ludicrous here. This must be
                                            changed.
NC - This is covered in section 5.1.1 as    COTS products themselves
shown by the following excerpt:   "COTS     should be subject to thorough
products require updates due to a detected evaluation, not just their
security breach or vulnerability. The       updates. COTS provide a
voting system vendor must provide a method significant security risk.
to assess the impact of COTS updates on the This must be addressed by the
voting system, as well as a method for      working group.
providing notice and distribution of
updates to purchasers."
RGHKR001

RGHKR002

RGHMD010




RGHKC016
       IEEE P1583
LOT COMMENT SUBMISSION FORM

                Date: 9-30-03 P1583 Draft 5.0 August 2003
                      Document:
                 Commenter Clause/ Subclause        Paragraph             Type of
                and Number                       Figure/ Table           comment
                                                                         (General/
       #                                                                 Technical/
                                                                          Editorial)
           1    Lipsio-5D            1.1                         3 E

           2    selker-1002          1.1                         3           e

           3 Corry-006               1.2                                     E


           4 Corry-007               1.2       2nd sentence                  T



           5    Lipsio-5E             2                              E


           6    Lipsio-5F             2                              E

           7    Lipsio-60             2                              E


           8    Lipsio-61             2                              E

           9    Lipsio-62             2                              E


           10   Lipsio-63             2                              E

           11   Lipsio-64             2                              E


           12   Lipsio-65             2                              E

           13   Lipsio-66             2                              E



           14   Lipsio-67             2                              E


           15   Lipsio-68             2                              E
16   Lipsio-69            2                         E


17   Lipsio-6A            2                         E


18   Lipsio-6B            2                         E


19 bronaugh - 001         3   Definition 8                  E
20 Corry-008              3                     8           E




21 Corry-010              3                    48           E


22 Corry-012              3                    73           E



23 df1                    3   No. 26                        E




24   df2                  3   No. 58                        E

25   Lipsio-6C            3                         E

26   PPLX-001             3   Section 3.                    E
                              Definition # 26




27   MercuriD50 - 006     3   definition #93             Editorial
     (formerly mercuri-
     011)


28 wfw -001               3            26               Editorial

29   HD-001               3                     8           G

30   HD-002               3                    59           G
31   HD-003       3                75      G

32   HD-004       3                98      G

33   PPLX - 002   3   Section 3.           G
                      Definition for
                      Directr
                      Recording
                      Electronic




34   wfw - 002    3         59          General

35   Adler-016    3                        T




36   Adler-017    3                        T
37   Adler-018   3        T




38   Adler-019   3        T




39   Adler-020   3        T




40   Adler-021   3        T



41 Corry-009     3   44   T
42 Corry-011            3   66, last sentence    T




43   Dill-10            3   Def #98              T


44   Dill-11            3   Def #30              T

45   Dill-28            3   Def # 42             T




46   Dill-6             3   Def # 17             T

47   Dill-7             3   Def #26              T


48   Dill-8             3   Def #64              T

49 Dill-9               3   Def #73              T
50 schneidewind - 002   3   Pg. 10               T
                            Line 26

                            Definition 26



51   selker-1003        3                   98   t
52   PPLX-004   3   Section 3.       T
                    Paper Based
                    Voting system.
                    Definition #
                    64.
53   PPLX-005   3   Closed and Open   T
                    Primaries:
                    Definitions 25
                    and 61.
54 vwilliams - 27   3   References       T




55   PPLX-003       3   Section 3      T and E
                        Firmware.
                        Definition #
                        49.
56   PPLX-006             3     Section 3.         T and E
                                Recall Issues
                                (with Options).
                                Definition 82




57   MercuriD50 - 070     3     Definitions #21   Technical
     (formerly mercuri-
     162)




58 VCW-01                 3     def 98            Technical




59   Dill-5               3     Def #4

60 berger - 001           3.1                         E
61 Corry-013      3.1                          E




62   Lipsio-55   3.101                    G


63   Lipsio-58   3.15    3.15 and elsewhereG



64   Lipsio-6D   3.26                     E




65   Lipsio-59   3.37    3.37 and elsewhereG



66   Lipsio-5A   3.42    3.42 and elsewhereG



67   Lipsio-6E   3.49                     E




68   Lipsio-6F   3.64                     E
69   Lipsio-70      3.7                     E

70   Lipsio-71     3.72                     E



71   PPLX-007        4      Section 4           G
                            System
                            Description
72   PPLX-012       4.6     Documentation       G
                            Section 4.6




73   PPLX-037     2.6.4.1   5.6.4.1 Error
                            Messages            G
74
     Sklein-051     3.0     Item 26             T


75
     Sklein-052     3.0     Item 38             T
76
     Sklein-055        3.0        All       T




77   Lipsio-76     3.54, 3.93           E

78 Gough-004          3.7.4


                                        T
79   Lipsio-77    3.94 and 3.95         E




80   Lipsio-72        3.98              E




81
     Sklein-049        4          All       T
82   Aragon - 06          4,4,1   last bullet point      T




83   Dill-12              4.0      first sentence        E



84 Corry-014              4.0      Entire section        T


85 Corry-015              4.0      First sentence        T




86   MercuriD50 - 007     4.1      diagram            Technical
     (formerly mercuri-
     014)


87   Lipsio-5B                                       draft
                          4.2 4.2 and throughout the G
88   RGH 002             4.2                             G


89 Corry-016             4.2                             T




90   Lipsio-5C              4                        G
                        4.2.1 .2.1 and throughout the draft




91   PPLX-008           4.2.1     Sections 4.2.1         G
                                  through 4.2.4

92 Corry-017             4.3      First sentence         E

93 MercuriD50 - 008     4.4.1     bulleted item        General
   (formerly mercuri-
   023)
94 MercuriD50 - 009     4.4.1     bulleted item        General
   (formerly mercuri-
   025)
95 Aragon - 07          4.4.1    last bullet point       T
 96 Corry-018              4.4.1   15th bullet point         T




 97 Gough-008              4.4.1




                                                       T
 98   PPLX-009             4.4.1   Precinct                  T
                                   Voting.
                                   Section 4.4.1.




 99   df3                  4.4.2                             E

100   MercuriD50 - 010     4.4.3   Entire subsection       General
      (formerly mercuri-
      026)




101 Gough-009               4.5



                                                    T
102 Corry-019              4.5.1                         E
                                   5th para., 2nd sentence




103   df4                  4.5.1   Paragraph 6               E



104   Lipsio-78            4.5.1   para. 5             E
105   RGH 003                  4.5.1        7th paragraph             E



106   Lipsio-0D                4.5.1        para. 4             T




107   PPLX-010                 4.5.1        Polling Place             T
                                            Reports.
                                            Section 4.5.1




108   MercuriD50 - 011 4.5.1, 4.5.2, 4.5.3, First sentence in       General
      (formerly mercuri-      4.5.4         each section
      027)
109   RGH 004                  4.5.1.       5th paragraph             E




110   df5                      4.5.2        Paragraph 3               E



111   PPLX-011                 4.5.2        Precinct                  G
                                            Reports
                                            Section 4.5.2




112   RGH 005                  4.5.3                                  G
113   HD-005               4.5.4   1st bullet                     T




114   Lipsio-0E            4.5.4   Bullets 1 - 5           T




115 Corry-020              4.5.5                         T
                                   2nd para., last sentence



116 Gough-010              4.5.5


                                                           T
117   Lipsio-0F            4.5.5   Para. 1                 T




118   MercuriD50 - 012     4.5.5   Add to end of section       Technical
      (formerly mercuri-
      030)


119   Lipsio-73             4.6    Bullet 5.10             E



120   Lipsio-74             4.6    Bullet 5.6              E


121   MercuriD50 - 013      4.6    Add bullet at end           General
      (formerly mercuri-
      034)


122 Corry-021               4.6    First sentence                 T


123   Lipsio-01             4.6    Bullet 5.2              T
124   Lipsio-02    4.6   Bullet 5.2   T


125   Lipsio-03    4.6   Bullet 5.3   T




126   Lipsio-04    4.6   Bullet 5.3   T



127   Lipsio-05    4.6   Bullet 5.3   T




128   Lipsio-06    4.6   Bullet 5.3   T




129   Lipsio-07    4.6   Bullet 5.4   T




130   Lipsio-08    4.6   Bullet 5.6   T



131   Lipsio-09    4.6                T




132
      Sklein-054   5     All              T
133
      Sklein-005    5.1     Para 5.1.3.2.5             T




134
      Sklein-006    5.1     Para 5.1.3                 T




135 Corry-023       5.1.1   p. 20, last para.          E




136   Lipsio-79     5.1.1   Para. 6            E




137   Lipsio-7A     5.1.1   Para. 7            E




138   RGH 006       5.1.1   last paragraph             E

139   VCW-02        5.1.1   2nd to last para       Editorial



140   Alice - 016   5.1.1                              G


141   RGH 008       5.1.1   last pp                    G
142   Simons - 001         5.1.1   bulleted list               G



143   Simons - 002         5.1.1                         G
                                   the sentence that reads, "The security countermeasures i




144   MercuriD50 - 031     5.1.1   Paragraph following       General
      (formerly mercuri-           bulleted list
      087)

145   Aragon - 09          5.1.1        third bullet           T




146 Corry-022              5.1.1                         T
                                   p.20, 3rd para., 3rd sentence




147   Lipsio-10            5.1.1   Para. 1               T


148   Lipsio-11            5.1.1   Para. 5               T
149   Lipsio-12      5.1.1   Para. 5          T




150   Lipsio-13      5.1.1   Para. 6          T




151   Lipsio-14      5.1.1   Para. 7          T




152   Lipsio-15      5.1.1   Para. 7          T




153   Lipsio-16      5.1.1   Para. 7          T


154   Lipsio-17      5.1.1   Para. 7          T




155   RGH 007        5.1.1   last paragraph       T

156   Simons - 003   5.1.2   the entire section   G

157   Lipsio-18      5.1.2   Bullet 2         T
158   Lipsio-19            5.1.2     Bullet 3              T


159   Dill-13              5.1.2.1   Third bullet                E
160   Simons - 004         5.1.2.1   the entire section          G

161   RGH 009              5.1.2.2   2nd pp                      G



162   Simons - 005         5.1.2.2   the entire section          G




163   Lipsio-1A            5.1.2.2                         T



164   Dill-14              5.1.2.3   Add introductory explanation E



165   Lipsio-7B            5.1.2.3                         E

166   MercuriD50 - 032     5.1.2.3   List of Assumptions       General
      (formerly mercuri-
      090)

167   MercuriD50 - 033     5.1.2.3   E.                        General
      (formerly mercuri-
      093)
168   Adler-001            5.1.2.3   1st paragraph               T




169   Adler-002            5.1.2.3   New 1st paragraph           T




170   Adler-003            5.1.2.3   Threats                     T
171   Aragon - 10   5.1.2.3   List of Assumptions   T



172 Corry-024       5.1.2.3   No. 2                  T



173   Dill-15       5.1.2.3   A-2                   T




174   Dill-16       5.1.2.3   A                     T


175   Dill-17       5.1.2.3   Add E-5               T


176   Dill-18       5.1.2.3   E-4                   T
177
      Sklein-058    5.1.2.3   5.1.2.3               T




178
      Sklein-001    5.1.2.3   A-2(h)                T/E
179   Adler-005      5.1.3    New paragraph          T
180
      Sklein-002   5.1.3   All   T




181
      Sklein-003   5.1.3   All   T
182
      Sklein-056      5.1.3    All                  T




183   Adler-004      5.1.3.1   New 2nd paragraph    T




184   Simons - 006   5.1.3.1   the entire section   T




185 GHM - 001        5.1.3.1                        T
186   MercuriD50 - 034        5.1.3.1.1        Paragraph            General
      (formerly mercuri-
      096)




187   MercuriD50 - 035        5.1.3.1.3        End of section       General
      (formerly mercuri-
      097)
188 Corry-025              5.1.3.1.3 and *.4                          G




189   GHM - 004             5.1.3.15.1         e                      T




190   Simons - 007            5.1.3.2.1        the entire section     G




191   GHM - 002              5.1.3.2.1                                T




192   Simons - 008            5.1.3.2.2        the entire section     G

193 Corry-026                 5.1.3.2.3        Missing g. as approvedT in previous comments no. 105 on s




194   Lipsio-1B               5.1.3.2.3                         T


195   RGH 010                 5.1.3.2.3        items d & e            T




196   RGH 011                 5.1.3.2.3        a                      T
197 Corry-027        5.1.3.2.4   f.                   T



198 Corry-028        5.1.3.2.4   Missing h. as approvedT in previous comments no. 108 on




199   Simons - 009   5.1.3.2.4   item c               T


200   Dill-19        5.1.3.2.5   item a               E


201   Dill-20        5.1.3.2.5   item b               E


202   Dill-21        5.1.3.2.5   item d               E

203   Dill-22        5.1.3.2.5   item e               E



204   RGH 012        5.1.3.2.5   item a               E


205   HD-006         5.1.3.2.5   e                    G


206   Simons - 010   5.1.3.2.5   a)                   G




207   Simons - 011   5.1.3.2.5   b)                   G




208   Simons - 012   5.1.3.2.5   c)                   G




209   Simons - 013   5.1.3.2.5   d)                   G
210   Simons - 014        5.1.3.2.5   e)                   G




211   PPLX-013           5.1.3.2.5    Vote Secrecy         G
                                      (DRE Systems)
                                      Section
                                      5.1.3.2.5




212 MercuriD50 - 036      5.1.3.2.5   d.                General
    (formerly mercuri-
    099)
213 Adler-015             5.1.3.2.5   a) - d)              T




214 Corry-029             5.1.3.2.5                         as
                                      Missing changes to c. T approved   in previous comments




215   RGH 013             5.1.3.2.5   d.                   T

216   GHM - 003          5.1.3.2.5                         T
217   PPLX-014   5.1.3.2.5    Vote Secrecy     T and E
                              (DRE Systems)
                              Section
                              5.1.3.2.5
                              Subsection e




218   vcw-04      5.1.3.2.5   a) - d)         Technical




219   vcw-05      5.1.3.2.5   e)              Technical
220   wfw - 003             5.1.3.2.5             e.           Technical




221 Gough-011




                           5.1.3.2.5.d                     T
222   wfw - 004              5.1.3.3                           Technical




223   Simons - 015          5.1.3.4.1    a)                        G

224   Simons - 016          5.1.3.4.1    e)                        G




225   MercuriD50 - 005      5.1.3.4.1    d.                     General
      (formerly mercuri-
      009)
226   MercuriD50 - 037      5.1.3.4.1    e.                     General
      (formerly mercuri-
      100)


227   MercuriD50 - 071      5.1.3.4.1    first paragraph        General
      (formerly mercuri-
      165)




228   Lipsio-1C             5.1.3.4.1    Item “a”          T

229   Lipsio-1D             5.1.3.4.1    Item “a”          T
230   Lipsio-1E            5.1.3.4.1   Item “a”         T




231   Lipsio-1F            5.1.3.4.1   Item “a”         T




232   Lipsio-20            5.1.3.4.1   Item “b”         T
233   Lipsio-21            5.1.3.4.1   Item “c”         T




234   Lipsio-22            5.1.3.4.1   Item “d”         T




235   Lipsio-23            5.1.3.4.1   Item “e”         T

236   RGH 014              5.1.3.4.1   item d                 T


237   df6                  5.1.3.4.2   Paragraphs 3-4         E

238   Simons - 017         5.1.3.4.2   the entire section     G


239   RGH 015              5.1.3.4.2   par. 3                G/T



240   MercuriD50 - 038     5.1.3.4.2   next to last         General
      (formerly mercuri-               paragraph
      101)


241   MercuriD50 - 039     5.1.3.4.2   last paragraph       General
      (formerly mercuri-
      102)
242   Lipsio-24            5.1.3.4.2   Para. 1           T




243   Lipsio-25            5.1.3.4.2   Para. 1           T




244   Lipsio-26            5.1.3.4.2   Para. 1           T




245   Lipsio-27            5.1.3.4.2   Para. 2           T

246   Dill-23              5.1.3.4.3                            E

247   Simons - 018         5.1.3.4.3   the entire section       G


248   MercuriD50 - 040     5.1.3.4.3   paragraph              General
      (formerly mercuri-
      103)


249   wfw - 005            5.1.3.4.3   First paragraph       General




250 Corry-030              5.1.3.4.3   2nd sentence             T
251   Lipsio-28   5.1.3.4.3                   T




252   Lipsio-29   5.1.3.4.4   Item “c”        T




253   Lipsio-2A   5.1.3.4.4   Item “f”        T



254   Lipsio-7C   5.1.3.4.5   Item “a”        E

255   Lipsio-7D   5.1.3.4.6   Item “b”        E
256   Lipsio-2B   5.1.3.4.6   Item “b”        T



257   Lipsio-2C   5.1.3.4.6   Item “b”        T




258 Corry-031     5.1.3.5.1                         E
                              3rd para., 2nd sentence


259   RGH 016     5.1.3.5.1   4th pp               E

260   Adler-022   5.1.3.5.1   1st paragraph        G
261   df7             5.1.3.5.1   Paragraph 4          G




262   HD-007          5.1.3.5.1   3rd paragraph        G




263   PPLX-015       5.1.3.5.1    System Audit         G
                                  Purpose and
                                  Context.
                                  Section
                                  5.1.3.5.1




264   Simons - 019    5.1.3.5.1                         T
                                  the first and second sentences




265
      Sklein-053      5.1.3.5.1   First paragraph     T/E
266   RGH 017        5.1.3.5.1 & 5.1.3.5.3 entire clause        E

267   Simons - 020        5.1.3.5.3    the first sentence       G



268   GHM - 005         5.1.3.5.3                               G




269   RGH 018             5.1.3.5.4    item a              note to Bob

270   HD-021              5.1.3.5.4    b.                       T




271   RGH 019             5.1.3.5.4    item b                   T


272   RGH 020             5.1.3.5.4    item c                   T



273   RGH 021             5.1.3.5.4         b.                  T




274   GHM - 006         5.1.3.5.4                               T
275   vcw-06                5.1.3.5.4      b)                      Technical




276   HD-008                5.1.3.5.5      b.3                         T




277   RGH 024               5.1.3.5.5      b.2                         T


278   RGH 022               5.1.3.5.5      items b1 & b3              T/E



279   Jhulshof-001          5.1.3.5.5      b5                      technical

280   RGH 023              5.1.3.5.5.a.3                               T


281   MercuriD50 - 041      5.1.3.5.6      between d) and e)        General
      (formerly mercuri-
      105)
282 MercuriD50 - 042        5.1.3.5.6      f)                       General
    (formerly mercuri-
    106)
283 RGH 025                 5.1.3.5.6      item d              note to Bob




284   Dill-41               5.1.3.5.6      item h                     T


285   Jhulshof-002          5.1.3.5.6      b5                      technical

286   Jhulshof-003          5.1.3.5.6      d                       technical

287 Gough-012

                           5.1.3.5.6.a                         T
288   Lipsio-2D            5.1.3.6                        T




289   RGH 026              5.1.3.6     last 3 bullets             T



290
      Sklein-004           5.1.3.6     Several                   T/E




291 Corry-032              5.1.3.6.1   2nd sentence               T



292   RGH 027              5.1.3.6.2   entire clause             G/T



293   MercuriD50 - 043     5.1.3.6.2   paragraph               General
      (formerly mercuri-
      107)
294 MercuriD50 - 046       5.1.3.6.2   end of paragraph        General
    (formerly mercuri-
    111)
295 Lipsio-7E              5.1.3.6.3   Item “a”           E

296   vcw-07               5.1.3.6.3   a)                     Editorial




297   vcw-08               5.1.3.6.3   d)                     Editorial
298   HD-010               5.1.3.6.3   d.                G

299   Lipsio-2E            5.1.3.6.3   Item “b”    T


300   RGH 028              5.1.3.6.3   items a-d         T




301   RGH 029              5.1.3.6.3   b,d               T


302   HD-009               5.1.3.6.3   a.                T

303   Lipsio-7F            5.1.3.6.4               E

304   Lipsio-2F            5.1.3.6.4               T




305   Lipsio-30            5.1.3.6.4               T




306   Lipsio-80            5.1.3.6.5               E

307 MercuriD50 - 044       5.1.3.6.5   e.              General
    (formerly mercuri-
    109)
308 Lipsio-81              5.1.3.6.7               E




309   MercuriD50 - 045     5.1.3.6.7   d. 2)           General
      (formerly mercuri-
      110)




310   Dill-24              5.1.3.6.8                     E
311   Lipsio-82            5.1.3.6.8               E
312   Lipsio-83            5.1.3.6.8               E

313   RGH 030              5.1.3.7.1   item c           G/E
314   RGH 031     5.1.3.7.1     a              T


315 Corry-033     5.1.3.7.2     2nd sentence   T



316   Dill-44     5.1.3.7.2                    T




317   Adler-006    5.1.3.8      New section    T




318   Adler-007   5.1.3.8.1     New section    T




319   Adler-008   5.1.3.8.1.1   New section    T




320   Adler-009   5.1.3.8.1.2   New section    T




321   Adler-010   5.1.3.8.2     New section    T
322   Adler-011          5.1.3.8.2.1   New section             T



323   Adler-012          5.1.3.8.2.2   New section             T




324   Adler-013          5.1.3.8.2.3   New section             T



325   vcw-03              5.1.5.2      4th paragraph       Editorial




326   RGH 032             5.2.1.1      item a                  G



327 MercuriD50 - 047      5.2.1.1      c.                   General
    (formerly mercuri-
    112)
328 MercuriD50 - 072      5.2.1.2      paragraph            General
    (formerly mercuri-
    166)

329 Corry-034             5.2.1.2      Add sentence            T



330 Gough-013             5.2.1.2
                                                       T
331   PPLX-016             5.2.1.2     5.2.1.2 DRE        T and E
                                       System
                                       Standards




332   Jhulshof-004         5.2.1.2                       technical
333   vcw-09               5.2.1.2                       Technical




334   selker-1004          5.2.1.2.3   #1                    e



335   selker-1005          5.2.1.2.3   #1                    t




336   selker-1006          5.2.1.2.5   b                     t

337   selker-1007          5.2.1.2.5   c                     t

338   selker-1008          5.2.1.2.5   d                     t

339   RGH 033               5.2.2      par. 12              G/T



340   MercuriD50 - 048      5.2.2      first paragraph    General
      (formerly mercuri-
      114)
341   MercuriD50 - 049         5.2.2       next to last             General
      (formerly mercuri-                   paragraph
      115)
342 Corry-035                  5.2.2       b.2                        T



343   Dill-25                  5.2.2       Last paragraph on page     T


344   Dill-45                  5.2.2                                  T



345   PPLX-017                 5.2.2       5.2.2 Accuracy             T
                                           Requirements




346 Gough-005                 5.2.2.1.b


                                                                T
347   Alice - 017              5.2.2.2                                G



348   Aragon - 11              5.2.2.2         First sentence         T




349 Gough-014                  5.2.2.2
                                                                T
350   RGH 034                  5.2.2.2                                T


351   RGH 035                              First setence of each.
                           5.2.2.2 & 5.2.2.3                          T
352   Lipsio-84            5.2.2.3                    E

353   MercuriD50 - 050     5.2.2.3   paragraph             General
      (formerly mercuri-
      116)
354   PPLX-018             5.2.3.1   5.2.3.1 Common           E
                                     Standards




355   Dill-26              5.2.3.1   item a                  T


356   PPLX-019             5.2.3.2




                                     5.2.3.2 DRE
                                     Systems
                                     Standards               G
357   MercuriD50 - 073     5.2.3.2   b.                    General
      (formerly mercuri-
      167)

358   Aragon - 08          5.2.3.2      new item d.           T




359   Dill-27              5.2.3.2   last sentence           T




360   vcw-10               5.2.3.2                        Technical
361 MercuriD50 - 051     5.2.3.3   a.                  General
    (formerly mercuri-
    117)
362 Lipsio-85            5.2.5                     E

363   Lipsio-86           5.2.5                    E
364   PPLX-020           5.2.5     5.2.5
                                   Reliability




                                                         E
365 MercuriD50 - 052     5.2.5     Last sentence       General
    (formerly mercuri-
    119)
366 Lipsio-31            5.2.5     Last line       T
367 schneidewind - 003     5.2.5     Pg. 43                   T




368
      Sklein-046           5.2.5     Second paragraph         T




369
      Sklein-047           5.2.5     First paragraph          T




370 Corry-036              5.2.6     Last sentence            T




371   Lipsio-32            5.2.6.2                      T



372   Lipsio-33            5.2.6.2                      T



373   MercuriD50 - 053     5.2.7     section                General
      (formerly mercuri-
      122)


374 Gough-016              5.2.7.b



                                                        T
375   RGH 036                   5.2.7.b.3                              T


376 berger - 005                   5.3        Para 1                   E

377 berger - 006                   5.3        4th bullet               E
378 berger - 009                   5.3        para 2                   E
379 berger - 010                   5.3                                 E

380   RGH 038                      5.3                  5              E


381   Jhulshof-005                 5.3        all section          editorial
382   HD-011                       5.3        7th bullet               G


383   RGH 037                      5.3                  3              G


384   RGH 039                      5.3            7th bullet           G

385   RGH 040                      5.3           Table 5.3-1           G




386   Aragon - 01                  5.3         First paragraph         T


387 Corry-051                      5.3        Figures 5.3-2,3,4,5,6 T



388
      Sklein-048                   5.3        First paragraph          T




389 berger - 007                    5.3          5 places              E
                             also 5.4.4, 5.4.6,
                              5.6.2, 5.6.2.6,
                           5.6.7.2.1, 5.6.8.1.2,
                           5.6.8.1.3, and many
390   Jhulshof-006             other places
                                   5.3.1                       2    general


391   MercuriD50 - 054            5.3.1       Bulleted List         General
      (formerly mercuri-
      125)
392   MercuriD50 - 055       5.3.1          2    General
      (formerly mercuri-
      126)
393   Aragon - 03          5.3.10.1     3           T




394   RGH 041              5.3.10.1                 T



395   Aragon - 04          5.3.10.1     1          T




396   Jhulshof-016         5.3.10.1         2   technical

397   RGH 113              5.3.10.1.3               E


398   RGH 042              5.3.10.1-1               G
399   PPLX-027             5.3.10.2




                                        5.3.10.2
                                        Accessibility
                                        for Voters with
                                        No Vision,
                                        Limited Vision,
                                        Reading
                                        Problems, or
                                        Print
                                        Blindness
                                                               E
400   Jhulshof-018          5.3.10.2    3                   general
                                            dot 3 from bottom



401   RGH 114               5.3.10.2                                T




402   Aragon - 05           5.3.10.2    3, fourth bullet           T




403   Jhulshof-017          5.3.10.2                        1   technical



404   MercuriD50 - 076      5.3.10.2    end of subsection       Technical
      (formerly mercuri-
      171)




405   wfw - 009            5.3.10.2.1       .1 (second          Technical
                                             sentence)
406   wfw - 010            5.3.10.2.2           0.1             Editorial
407   RGH 043              5.3.10.2-1                           G


408   RGH 046              5.3.10.2-3     4th bullet            E




409   RGH 044              5.3.10.2-3     12th bullet           G

410   RGH 045              5.3.10.2-3     13th bullet           G

411   RGH 047              5.3.10.2-3                           G

412   Jhulshof-020         5.3.10.3                     2   editorial
413   MercuriD50 - 061     5.3.10.3                     1     General
      (formerly mercuri-
      139)
414   Jhulshof-019         5.3.10.3                     1   technical

415   Jhulshof-021         5.3.10.4                     1   technical


416 Corry-043              5.3.10.5     Title                   E


417   Jhulshof-022         5.3.10.5                     3    general



418   RGH 048              5.3.10.5                             T



419   df8                  5.3.10.6     1                       E
420   df9                  5.3.10.6     5.3-7                   E
421   Jhulshof-023         5.3.10.6                     1    general




422   Jhulshof-025         5.3.10.6     figures              general



423   MercuriD50 - 062     5.3.10.6                     1    General
      (formerly mercuri-
      140)
424 Corry-044        5.3.10.6            1       T



425 Corry-045        5.3.10.6   1.a.             T




426 Corry-046        5.3.10.6   1.b              T




427 Corry-047        5.3.10.6   1.c              T




428 Corry-048        5.3.10.6   1.d              T




429 Corry-049        5.3.10.6   1.e              T




430 Corry-050        5.3.10.6   1.g              T




431   Jhulshof-024   5.3.10.6   1 g          technical


432   RGH 049        5.3.1-2                     E




433   vc1-11          5.3.3     9 & 10       Editorial
434   HD-013             5.3.3     1. Last bullet                G




435   HD-014              5.3.3    9. & 10.                      G
436   PPLX-021           5.3.3




                                   5.3.3
                                   Information
                                   Presentation         G
437   Jhulshof-007       5.3.3                       general
                                   table charactor size 5.3-1



438   Jhulshof-008       5.3.3                           7    general

439 MercuriD50 - 056     5.3.3     Second bullet under        General
    (formerly mercuri-             2.
    129)
440 MercuriD50 - 057     5.3.3     Add bullet under 5         General
    (formerly mercuri-
    132)
441   HD-012             5.3.3     Table 5.3-1                   T




442   RGH 050            5.3.3     Table 5.3-1                   T

443   wfw -006           5.3.3.7    Last paragraph           Editorial
                                     and sentence.
444   RGH 051        5.3.3.-7          2               G



445   RGH 052        5.3.3.-7     2nd bullet           G


446   RGH 053        5.3.3-1      2nd bullet           G


447   RGH 054        5.3.3-3                           G


448   RGH 055        5.3.3-3                           G

449   RGH 057        5.3.3-5      7th bullet           E




450   RGH 056        5.3.3-5       4th bullet          G
451   RGH 058         5.3.4       footnote 14          E

452   PPLX-022       5.3.4




                                5.3.4 Vote
                                Selection
                                Mechanisms          E and G
453   Jhulshof-009    5.3.4                     3   general



454   Jhulshof-010    5.3.4                     8   general




455   Jhulshof-011    5.3.4     9 last dot          general
456   MercuriD50 - 058      5.3.4    footnote #17          General
      (formerly mercuri-
      135)



457 MercuriD50 - 059        5.3.4                      8   General
    (formerly mercuri-
    136)
458 MercuriD50 - 074        5.3.4    7                     General
    (formerly mercuri-
    169)

459   MercuriD50 - 075      5.3.4    footnote #17          General
      (formerly mercuri-
      170)



460   RGH 059              5.3.4-3                            G


461   RGH 060              5.3.4-8                            E
462   RGH 061              5.3.4-9       4th bullet           G

463   Aragon - 02           5.3.5             5               E

464   RGH 115               5.3.5                             E




465   HD-015                5.3.5                             G


466   Jhulshof-012          5.3.5                     13   general


467 Corry-037               5.3.5                      5      T




468 Corry-038               5.3.5                      6      T

469 Corry-039               5.3.5                     11      T
470 Corry-040                     5.3.5           12, 2nd bullet          T



471   RGH 062                   5.3.5.10          first bullet            T


472   RGH 064                   5.3.5-11             3rd bullet           E




473   RGH 063                   5.3.5-11             1st bullet           G




474   PPLX-023                   5.3.6




                                                  5.3.6
                                                  Navigation and
                                                  Interaction
                                                  with the System         E
475   Jhulshof-013                5.3.6                          1     general



476   MercuriD50 - 060            5.3.6                            1   General
      (formerly mercuri-
      137)




477   RGH 116              5.3.6.1 and 5.3.7.6.                           T




478   RGH 065                    5.3.6-1                                  G

479   RGH 066                    5.3.6-3                                  G

480   RGH 067                    5.3.6-6                                  E
481 Corry-041         5.3.7                    5       T




482   HD-016          5.3.7                    2       T




483   PPLX-024       5.3.7




                               5.3.7 System
                               Response Time
                               and Feedback         T and E
484   Jhulshof-014    5.3.7                    2   technical


485   RGH 068        5.3.7.4                           T



486 RGH 069          5.3.7-4                           G
487 Corry-042         5.3.8                    2       E
488   PPLX-025       5.3.8




                               5.3.8
                               Preventing and
                               Minimizing
                               Voter Errors             G
489   Jhulshof-015    5.3.8                     3   technical



490   RGH 070        5.3.8.5                            T

491   wfw -007       5.3.8.5                        Technical


492   RGH 071        5.3.8-5                            G

493   PPLX-026       5.3.9     5.3.9 Help and
                               Indications of
                               Degraded
                               Conditions               G
494   wfw - 008      5.3.9.2         All            Technical




495 Corry-052         5.4      First sentence           E




496   RGH 072                                        T
                     Second paragraph above clause 5.4.1.
                      5.4
497 Corry-053               5.4.1   Entire section       T




498   PPLX-031             5.4.10   5.4.10 Safety




                                                         G
499 Corry-054               5.4.4   a.                   E


500 Corry-055               5.4.4   Should be c.         E


501   MercuriD50 - 063      5.4.4   c.                General
      (formerly mercuri-
      141)
502   Jhulshof-026          5.4.4   b                technical




503   Jhulshof-027          5.4.4   last sentence    technical


504 Corry-056               5.4.5                        E


505   PPLX-028             5.4.5




                                    5.4.5
                                    Environmental
                                    Control -
                                    Operating
                                    Environment          G
506   RGH 073               5.4.5      Humidity range       T




507   RGH 074               5.4.5       Temp. ranges        T




508   PPLX-029             5.4.6




                                      5.4.6
                                      Environmental
                                      Control -
                                      Transit and
                                      Storage               G
509 Corry-057               5.4.6     First sentence        T



510 Corry-058               5.4.6                           T




511   PPLX-030             5.4.8


                                      5.4.8 Product
                                      Marking               G
512 Corry-059               5.4.9     b)                    T




513   Jhulshof-028          5.5.1     last sentence     editorial

514   RGH 075                                               T
                     Last sentence "Electrical power increases of 10%….
                            5.5.1
515 Corry-060       5.5.2      a)                         E
516 Corry-061       5.5.2      b) and c)                  T

517 Corry-062       5.5.3      c), d), and e)             T


518 Corry-063       5.5.3      c)                         T

519 Corry-064       5.5.3      d)                         T

520 Corry-065       5.5.3      e)                         T

521 Gough-015      5.5.3.2.a
                                                  T
522 Corry-066       5.5.4      First sentence             E
523 Corry-068       5.5.4      Last sentence              E




524 df11            5.5.4                                 E
525 Jhulshof-029    5.5.4      second sentence        editorial
526 Corry-067       5.5.4      Footnote 26                T




527   RGH 076       5.5.4       Superscript 26            T



528   RGH 077       5.5.4                                 T

529   PPLX-032      5.5.5




                               5.5.5
                               Electromagnetic
                               Radiation                 E
530 brook - 001     5.5.6                                T

531 Corry-069       5.5.7      f) and g)                  E
532 Jhulshof-032    5.5.7      3 and 4 sentence       editorial
533 Corry-070                 5.5.7            f) and g)                 T

534   Jhulshof-030            5.5.7            Second sentence       technical



535   Jhulshof-031            5.5.7            g)                    technical
536   PPLX-033               5.5.8


                                               5.5.8 Magnetic
                                               Fields Immunity           T
537   Lipsio-56                5.6                               G




538 Corry-071                  5.6             Entire section            T


539   Lipsio-0A                5.6                               T




540   Lipsio-0B                5.6                               T


541
      Sklein-007               5.6             Para 5.6.1.1              T




542   Lipsio-34      5.6 , 6.6.4.1 , 6.6.4.1.2                   T




543   Lipsio-35      5.6, 6.6.4.1, 6.6.4.1.2                     T
544   Lipsio-36            5.6, 6.6.4.1, 6.6.4.1.2             T



545   Lipsio-37            5.6, 6.6.4.1, 6.6.4.1.2             T




546   Lipsio-87                    5.6.1             Para. 3   E




547   Lipsio-38                    5.6.1             para. 2   T




548   Lipsio-88                   5.6.1.1            para. 2   E


549   MercuriD50 - 064            5.6.1.1            Section       General
      (formerly mercuri-
      143)




550   Alice - 001                 5.6.1.1                            T




551   Lipsio-39                   5.6.1.1            Para. 2   T




552   Lipsio-3A                   5.6.1.1            para. 3   T
553   Lipsio-3B     5.6.1.1   para. 3         T




554   Lipsio-3C     5.6.1.1   Para. 3         T




555   RGH 078       5.6.1.1                       T

556   Alice - 005   5.6.1.1   2nd Paragraph       T

557 berger - 008    5.6.1.2                       E

558 berger - 011    5.6.1.2   last para           E

559   RGH 079       5.6.1.2                       G




560   Lipsio-3D     5.6.1.2                   T
561 MercuriD50 - 068         5.6.10.2        i             General
    (formerly mercuri-
    160)
562 MercuriD50 - 069         5.6.10.2        list          General
    (formerly mercuri-
    161)
563 Lipsio-89                 5.6.2                    E




564   Lipsio-8A              5.6.2.1                   E

565   Alice - 008            5.6.2.1                         G


566   Alice - 002            5.6.2.1                         T




567   HD-022                 5.6.2.1                         T




568   Lipsio-8B      5.6.2.1, 5.6.2.3- 5.6.2.7         E

569   Lipsio-3E              5.6.2.2         para. 1   T




570   Lipsio-3F              5.6.2.2                   T
571   PPLX-034      5.6.2.2




                               5.6.2.2
                               Software
                               Integrity        T
572   Lipsio-40     5.6.2.2.   bullet 3     T




573   Lipsio-41     5.6.2.2.   bullets      T




574   Lipsio-42     5.6.2.2.   para. 1      T



575   Lipsio-8C     5.6.2.3    Bullet “c”   E
576   Lipsio-8D     5.6.2.3    Bullet “c”   E




577   Lipsio-8E     5.6.2.3                 E


578   Alice - 010   5.6.2.3    c                G

579   Alice - 009   5.6.2.3    a                T



580   Lipsio-43     5.6.2.3    Para. 1      T




581   Lipsio-44     5.6.2.3    Para. 1      T
582   Lipsio-45          5.6.2.3   Para. 1           T




583   Lipsio-46          5.6.2.3                     T



584   RGH 117            5.6.2.3                         T




585 schneidewind - 005   5.6.2.3   Pg. 70                T



586
      Sklein-044         5.6.2.3   First paragraph       T


587
      Sklein-045         5.6.2.3   First paragraph       T




588
      Sklein-057         5.6.2.3   5.6.1.1               T
589   PPLX-035               5.6.2.3




                                              5.6.2.3
                                              Software
                                              Modularity and
                                              Programming         T & E
590   Lipsio-47            5.6.2.3, 5.6.2.7                    T
                                              Bullet “c”, Bullet “a”




591   MercuriD50 - 065         5.6.2.4        control constructs list   General
      (formerly mercuri-
      147)




592   Alice - 003              5.6.2.4                                    T
593   PPLX-036      5.6.2.4




                                5.6.2.4 Control
                                Constructs            T and E
594   Alice - 006   5.6.2.4     a




595   Alice - 011   5.6.2.5     c                        G




596   Lipsio-8F     5.6.2.7     Bullet “d”        E


597 Gough-017        5.6.3




                                                  T
598 Corry-072       5.6.4.1     a)                       T




599 Corry-073       5.6.4.1     c)                       T




600   RGH 080       5.6.4.1.c                            T
601 Corry-074              5.6.4.2   After 1st sentence              T




602 Corry-075              5.6.4.2   2nd sentence                    T




603 Corry-076              5.6.5.1   c)                              E

604 Gough-018              5.6.5.1



                                                               T
605   Lipsio-90            5.6.5.2                             E




606   PPLX-038             5.6.5.2   5.6.5.2 Voting
                                     Variations                      G
607   MercuriD50 - 066     5.6.5.2   Voting variations list.       General
      (formerly mercuri-
      153)




608 Gough-006              5.6.5.2


                                                               T
609 Gough-019              5.6.5.2


                                                               T
610 Gough-020               5.6.6




                                                               T
611   Lipsio-48             5.6.6    Bullet “a”                T
612   Jhulshof-033     5.6.6       opening                technical




613   Jhulshof-034     5.6.6       a                      technical

614 Corry-077         5.6.7.1      First sentence             E


615   RGH 081         5.6.7.1.c                               T


616 Corry-078        5.6.7.2.1     f) and g)                  E




617 Corry-079        5.6.7.2.1     Second a) and b)           E


618 Corry-080        5.6.7.2.1     Second a)                  T




619   Lipsio-49      5.6.7.2.1                        T




620   RGH 082        5.6.7.2.1.e                              T


621 Gough-021          5.6.8
                                                      T
622   Lipsio-91      5.6.8.1.1     Bullet “b”         E


623   wfw - 011      5.6.8.1.2           a)               Editorial

624   Lipsio-4A      5.6.8.1.3     Bullet “a”         T



625 Gough-022        5.6.8.2.b
                                                      T
626   RGH 083         5.6.8.2.c                               T


627   RGH 084        5.6.8.2.d                               E
628   Dill-29         5.6.8.3                                E
629 Corry-081              5.6.8.3.2   j)                        E




630   Jhulshof-035         5.6.8.3.2   b                     editorial
631   wfw -012             5.6.8.3.2         b)              Editorial

632   Jhulshof-036         5.6.8.3.2   c                      general
633   MercuriD50 - 067     5.6.8.3.2   f                       General
      (formerly mercuri-
      158)

634 Corry-082              5.6.8.3.3   i)                        E




635 Corry-083              5.6.8.3.3   o)                        E




636 Gough-023              5.6.8.3.3
                                                         T
637   Jhulshof-037         5.6.8.3.3   p                     technical

638 Corry-084               5.6.9      First sentence            E




639 Corry-085               5.6.9      Second sentence           T


640 Gough-024               5.6.9
                                                         T
641   RGH 085              5.6.9.1.e                             T
642   Simons - 021          5.6.9.2    the list                  G

643 Corry-086              5.6.9.2     k)                        T



644 Gough-025              5.6.9.2




                                                         T
645
      Sklein-037   6     Second bullet          T



646 JL - 005       6.0                          E




647   Dill-30      6.1                          T

648   Dill-31      6.1   Paragraph 3            T


649   Dill-32      6.1   Paragraph 3            T


650   Dill-33      6.1   Paragraph 4            T
651   Lipsio-0C    6.1   Para. 3            T




652
      Sklein-038   6.1   Fourth paragraph       T


653
      Sklein-039   6.1   All                    T
654
      Sklein-059                  6.1           All                 T




655   RGH 086              6.1 par. 3 & 6.1.2                     G/T/E
                                 par. 1


656
      Sklein-040                 6.1.1          Last paragraph      T



657   RGH 087                    6.1.2          items e & f        G/T



658   HD-017                     6.1.3          e)                  G


659   RGH 088                    6.1.3          item e             G/E



660   MercuriD50 - 014           6.1.3          e)                General
      (formerly mercuri-
      038)

661   Dill-34                    6.1.3          Item c              T




662
      Sklein-041                 6.1.4          First paragraph     T

663
      Sklein-042                 6.1.4          First paragraph     T
664   Adler-014            6.1.5     New section          T




665   MercuriD50 - 015      6.2      main section       General
      (formerly mercuri-
      039)

666   MercuriD50 - 016     6.2.1     Bulleted list      General
      (formerly mercuri-
      040)

667   MercuriD50 - 017     6.2.1     End of section     General
      (formerly mercuri-
      041)




668 Corry-087              6.2.1.1                         T
                                     4th para., 2nd sentence




669 schneidewind - 004     6.2.1.1   Pg. 86               T


670 Corry-088              6.2.2                          E


671 Corry-089              6.2.2.1                         T
                                     2nd para., 2nd sentence


672   MercuriD50 - 018     6.2.3     list               General
      (formerly mercuri-
      042)


673   RGH 089              6.2.4               c.         G
674   df10                   6.3      All Sections                E



675   MercuriD50 - 019       6.3      Bulleted list            General
      (formerly mercuri-
      044)
676 Corry-125                6.3                                  T



677 Corry-126                6.3                                  T



678 Corry-096              6.3..3.3                               T

679   MercuriD50 -          6.3.1     Entire            Technical
      079 (new)                       subsection
680 Corry-090              6.3.1.1                          E
                                      2nd para., 2nd sentence




681   RGH 090               6.3.2                                 G




682   RGH 091               6.3.3                                 E

683 Corry-091               6.3.3     Entire section              G




684   MercuriD50 - 077      6.3.3     Test matrix - 5.3.10.2   Technical
      (formerly mercuri-
      172)


685 Corry-092              6.3.3.1                                T
686 Corry-093            6.3.3.1                               T

687 Corry-094            6.3.3.1                               T

688 Corry-124          6.3.3.10.1         Item 3               E



689 Corry-122          6.3.3.10.1                              T

690 Corry-123          6.3.3.10.1                              T


691   RGH 092          6.3.3.10.2           5.3.10.2-4         G




692 Corry-127          6.3.3.10.2                              T




693 Corry-128          6.3.3.10.2                              T


694 Corry-129          6.3.3.10.3                              T



695 Corry-130          6.3.3.10.3                              T

696 Corry-131          6.3.3.10.4                              T




697 Corry-132          6.3.3.10.5                              T



698 Corry-133          6.3.3.10.5                              T



699 Corry-134          6.3.3.10.5                              T


700 Corry-095            6.3.3.2                               T


701 berger - 004   6.3.3.2 - 6.3.3.10.3                        E

702 Corry-099            6.3.3.3                                E
                                          Para. 2., first sentence
703   wfw - 013    6.3.3.3      3       Editorial




704   RGH 093      6.3.3.3    5.3.3-7       G




705 Corry-097      6.3.3.3                  T

706 Corry-098      6.3.3.3                  T


707   RGH 094     6.3.3.3-1                 G


708   RGH 095     6.3.3.3-1                 G




709   RGH 096      6.3.3.4    5.3.4-1       G




710 Corry-100      6.3.3.4                  T

711 Corry-101      6.3.3.4                  T




712 Corry-102      6.3.3.4                  T
713 Corry-103   6.3.3.4              T




714 Corry-108   6.3.3.5              E


715   RGH 097   6.3.3.5   5.3.5-10   G




716   RGH 098   6.3.3.5   5.3.5-11   G




717   RGH 099   6.3.3.5   5.3.5-12   G




718   RGH 100   6.3.3.5   5.3.5-13   G




719   RGH 101   6.3.3.5   5.3.5-14   G




720   RGH 102   6.3.3.5   5.3.5-9    G




721 Corry-104   6.3.3.5              T



722 Corry-105   6.3.3.5              T


723 Corry-106   6.3.3.5              T
724 Corry-107   6.3.3.5                 T

725 Corry-109   6.3.3.6                 T


726 Corry-110   6.3.3.6                 T

727 Corry-111   6.3.3.6                 T


728 Corry-115   6.3.3.7   Item 6        E

729 Corry-116   6.3.3.7   Item 7        E

730 Corry-112   6.3.3.7                 T


731 Corry-113   6.3.3.7                 T


732 Corry-114   6.3.3.7                 T




733   RGH 103   6.3.3.8       5.3.8-1   G


734   RGH 104   6.3.3.8       5.3.8-2   G




735 Corry-117   6.3.3.8                 T


736 Corry-118   6.3.3.8                 T



737 Corry-119   6.3.3.9   Title         E



738   RGH 105   6.3.3.9       5.3.9-1   G




739   RGH 106   6.3.3.9       5.3.9-3   G
740   RGH 107              6.3.3.9        5.3.9-3         G




741 Corry-120              6.3.3.9                        T



742 Corry-121              6.3.3.9                        T


743 Corry-135               6.4      Last sentence        E




744   RGH 108              6.4.1      First paragraph     G
745   RGH 109              6.4.1     Second paragraph     T



746 Corry-136              6.4.3     First sentence       E




747   MercuriD50 - 020     6.4.3     End of section     General
      (formerly mercuri-
      045)




748 Corry-137              6.4.3     Last three paragraphs T




749 Corry-138              6.4.4.1                         T
                                     1st para., 1st sentence
750 Corry-139            6.4.4.1                           T
                                     2nd para., 1st sentence




751   Lipsio-4B          6.4.4.1     Para. 2         T




752 New - 001            6.4.4.1                          T




753 schneidewind - 006   6.4.4.1     Pg. 100              T


754 Corry-140            6.4.4.1.2   Last sentence        E




755 Corry-141            6.4.4.1.6                        T




756 Corry-142            6.4.4.3.2   Step 5               E


757 Corry-143            6.4.4.4.2                        E



758 Corry-144            6.4.4.5.2                        E

759 Corry-145            6.4.4.5.2   Step 3               E
760   RGH 110              6.4.5.1   Steps 4, 9, 10 and 11.      T
761   Lipsio-75              6.6                         E

762   Lipsio-57             6.6                            G




763 Corry-146               6.6         Entire section           T




764   Alice - 004          6.6..2                                G


765   MercuriD50 - 021     6.6.1        End of paragraph       General
      (formerly mercuri-
      047)




766   MercuriD50 - 022     6.6.2        Paragraphs 2-4         General
      (formerly mercuri-
      048)




767   Dill-35              6.6.2                                 T


768   Dill-36              6.6.2        Paragraph 3              T

769   Lipsio-4C            6.6.2        para. 1            T



770   Lipsio-4D            6.6.2        para. 3 & 4        T
771   Lipsio-4E               6.6.2                            T


772   Lipsio-4F               6.6.2                            T




773   Lipsio-50               6.6.2                            T


774 schneidewind - 007        6.6.2       Pg. 107                    T




775 schneidewind - 001        6.6.2       Pg. 107                    T




776   Lipsio-51            6.6.3, 6.6.4                        T

777   MercuriD50 - 023        6.6.4       End of paragraph         General
      (formerly mercuri-
      049)




778   Dill-37                6.6.4.1      Replacement rule discussion E



779   Lipsio-92              6.6.4.1                           E


780   Alice - 012            6.6.4.1                                 G



781   Alice - 007            6.6.4.1                                 T




782   Lipsio-52             6.6.4.1.2                          T

783 Corry-147                6.6.4.2      r.                         E
784   Lipsio-93            6.6.4.2                   E


785   Alice - 013          6.6.4.2                           T




786   Dill-38              6.6.4.2   item c                  T
787   HD-018               6.6.4.2   I.                      T



788   Lipsio-53            6.6.4.2   bullet “n”      T



789   Lipsio-54            6.6.4.2                   T




790   MercuriD50 - 024     6.6.4.2   i.                   Technical
      (formerly mercuri-
      056)




791   Dill-39              6.6.4.5   item w                  T




792 Corry-148              6.6.5.1   Last two sentences      E
793   MercuriD50 - 025     6.6.5.1.2   Last paragraph     General
      (formerly mercuri-
      061)




794   Alice - 014          6.6.6.1.1                        G

795   Dill-40              6.6.6.1.2                        T


796   HD-019               6.6.6.3                          G


797   HD-020               6.6.6.4                          G


798 Corry-149                7.1                             T
                                       2nd para., last sentence




799 Corry-150               7.1.1      Last diamond         T




800 Corry-151              7.1.1.1     d.                   T

801 Corry-152              7.1.1.1     New para.            T


802   RGH 118              7.1.1.1.a                        E




803 Corry-153              7.1.1.3     2nd para.            E




804   RGH 119              7.1.1.3                          G



805   Dill-42              7.1.1.3                          T
806   MercuriD50 - 026      7.1.2    last sentence       General
      (formerly mercuri-
      066)

807   Dill-43               7.1.3                          T



808   RGH 120              7.10.1                          G



809 Corry-190              7.10.1    First paragraph       T




810 Corry-191              7.10.1    Last paragraph        T



811   RGH 121              7.10.2                          G

812 Corry-192              7.10.2    First sentence        T



813 Corry-193               7.11     5th diamond           E


814   RGH 112               7.11                           E

815   MercuriD50 - 029      7.11     between bullets 3   General
      (formerly mercuri-             and 4
      079)




816 Corry-195              7.11.1    2nd paragraph         E




817 Corry-194              7.11.1    1st para., c.         T

818 Corry-201              7.11.12   Last sentence         E


819 Corry-196              7.11.2    2nd para., c          T
820 Corry-197        7.11.7    1st sentence              E



821 Corry-198        7.11.7    a.                        T

822 Corry-199        7.11.7    b.                        T



823 Corry-200        7.11.7    c.                        T

824 Corry-204        7.12..3   c.                        T

825 Corry-202        7.12.2    a. 1st sentence           E


826 Corry-203        7.12.2    b.                        E



827   MercuriD50 -    7.13                           Technical
      078 (new)



828   RGH 122        7.13.b                              G

829 Corry-154        7.2.2     a.                        T




830 Gough-007        7.2.2




                                                 T
831 Corry-155         7.3      First sentence            E




832 Corry-156         7.3      c.                        T



833 Corry-157         7.3      d.                        T
834 Corry-158    7.3      e.                    T




835 Corry-159   7.4.2     Add e.                T


836 Corry-160   7.5.1     First sentence        T




837 Corry-168   7.5.10    a. Glossary           E




838 Corry-169   7.5.10    c. Program analysis   E




839   RGH 123   7.5.3.d                         E




840 Corry-161   7.5.6.1   Add d)                T


841 Corry-162   7.5.6.2   Add h)                T

842 Corry-163   7.5.7.1   First sentence        T




843   RGH 124   7.5.7.2   f3                    G


844 Corry-164   7.5.7.2   a.                    T
845   RGH 125              7.5.8     d                    E

846 Corry-165              7.5.8     c.                   T




847   RGH 111              7.5.9.1                        T

848 Corry-166              7.5.9.2   b.4)                 T




849 Corry-167              7.5.9.2   c.2)                  T
850 Corry-170                7.6                           E
                                     3rd para., 1st sentence



851   Dill-3                7.6                           T




852   Dill-4                7.6                           T




853
      Sklein-043            7.6      All                  T




854   MercuriD50 - 027     7.6.1     Add sentences at   General
      (formerly mercuri-             end
      073)

855 Corry-171              7.6.2     2nd para.            E



856   RGH 126              7.6.2                          G
857 Gough-026   7.6.2
                                         T
858 Corry-172   7.6.5   b.2)                 E



859 Corry-174   7.6.5   b. 6)                E




860   RGH 127   7.6.5                        G



861 Corry-173   7.6.5   b. 4)                T




862 Corry-175   7.6.6   First sentence       E




863 Corry-176   7.6.6   Last paragraph       E




864 Corry-177   7.7.1   Numbering             E
865 Corry-178   7.7.1                         T
                        Last para., 2nd sentence



866 Corry-179   7.8.5   b)                   E




867   RGH 128   7.8.5   g)                   G
868 Corry-180   7.8.5     c)        T




869 Corry-181   7.8.6     a)        E




870 Corry-182   7.8.6     b         T




871 Corry-183   7.8.7     c)        T




872 Corry-184   7.8.7     d)        T




873 Corry-185   7.9.1     d)        T


874   RGH 129   7.9.2.1             G




875   RGH 130   7.9.2.2   a and b   G



876 Corry-186   7.9.4.1   b)        T

877 Corry-189   7.9.5     c)        E
878   MercuriD50 - 028     7.9.5   end of list        General
      (formerly mercuri-
      078)


879 Corry-187              7.9.5   a)                     T




880 Corry-188              7.9.5   b)                     T




881   Lipsio-94             A                       A-1
                                   title of section E




882 Corry-205              A.1                           E
                                   1st para., 2nd sentence


883 Corry-206              A.1     2nd para.              E
884 Corry-208          A.1      4th paragraph           E




885 Corry-209          A.1      Last paragraph          E



886 Corry-207          A.1      3rd para.               T




887   MercuriD50 -   Abstract   page ii                 E
      004 (new)
888 Peterson-1       Abstract   Section 1 or abstract   G
889 Corry-002       Abstract                        T



890 Corry-003       Abstract                    T
                               1st para., 2nd sentence




891 Corry-004       Abstract                         T
                               2nd para., last sentence




892   selker-1001   Abstract   Pg. ii               t
                               Keywords
893 Corry-210          B       2nd para.            E




894 Corry-211          B       2nd para., add bullet T




895 MercuriD50 -       C       Entire section       E
    002 (new)                  C.7.1
896 wfw -014           C             all         General


897 Corry-212          C       Title                T

898 Corry-213          C       Entire section       T
899 MercuriD50 - 001   C   Entire Annex         T
      (new)




900   MercuriD50 -     C   C.7.3                T
      003 (new)




901
      Sklein-050       C   All                  T




902   vcw-26           C   Table 4 and beyondTechnical
903
      Sklein-008   C.3.1.1   A.CONNECT       T




904
      Sklein-034   C.3.1.1   All             T

905
      Sklein-009   C.3.1.3   A.MANAGE        T




906
      Sklein-010   C.3.1.4   A.LOCATE        T




907
      Sklein-011   C.3.1.4   A.PROTECT       T




908
      Sklein-012   C.3.3.1   All             T




909   vcw-12       C.3.3.2               Editorial
910 diaz - 001     C.3.3.2                        T




911
      Sklein-013   C.3.3.3   All                  T


912
      Sklein-014   C.3.4     P.ACCESS_LEVEL       T

913
      Sklein-015   C.3.4     P.ADMINISTRATOR      T


914
      Sklein-016   C.3.4     P.ALERT              T




915
      Sklein-017   C.3.4     P.ASSURANCE          T

916
      Sklein-018   C.3.4     P.DATA_AUTHENTICATION T

917
      Sklein-019   C.3.4     P.PHI_ACCESS         T




918
      Sklein-020   C.3.4     P.SECURE_TRANSMISSION T

919
      Sklein-021   C.3.4     P.SESSION_TERMINATION T




920
      Sklein-022   C.3.4     P.VOTER_ANONYMITY    T

921 diaz - 002     C.4.1     O.INSTALL            T
922
      Sklein-023   C.4.1     O.ABORT_SESSION      T

923
      Sklein-025   C.4.1     O.AUDIT_RECORD       T
924
      Sklein-026   C.4.1   O.AUDIT_REVIEW          T




925
      Sklein-027   C.4.1   O.EVENT                 T




926
      Sklein-028   C.4.1   O.INSTALL               T




927
      Sklein-029   C.4.1   O.VOTE_VALIDATION       T



928
      Sklein-030   C.4.1   All                     T



929
      Sklein-024   C.4.1   O.AUTH_VOTER           T/E
930   vcw-13       C.4.1   O.Install           Technical
931 diaz - 003     C.4.2   Para 1                  T

932 diaz - 004     C.4.2   OE.PHYSICAL            T

933
      Sklein-031   C.4.2   OE_PHI_ACCESS           T
934
      Sklein-032       C.4.2         OE.USER_TERMINATION   T


935 diaz - 005         C.5.1         FCF.VA                T




936 diaz - 006         C.5.1                               T



937
      Sklein-033       C.5.1         All                   T




938   vcw-14           C.5.1                           Technical
                                     Security Requirments
939 diaz - 007      C.5.1.1.1.2.1                          T

940 diaz - 008      C.5.1.1.2.1.1                          T




941 diaz - 019     C.5.1.5.2.3.2




942   vcw-15          C.5.1.5.3                        Technical

943   vcw-16         C.5.1.5.3.1                       Technical

944   vcw-17        C.5.1.5.3.1.1                       Technical
945   vcw-18        C.5.1.5.3.1.1   after C.5.1.5.3.1.1 Technical



946   vcw-20        C.5.1.6.4.1.3                      editorial



947 diaz - 009         C.5.2                               T
948 diaz - 010     C.5.2                   T




949 diaz - 011     C.5.2                   T




950 diaz - 012     C.5.2                   T




951
      Sklein-035   C.5.2   All              T




952   vcw.23       C.5.2                Technical
953   vcw-21       C.5.2   Class AGD:   Technical
                           Guidance
                           documents
954   vcw-22           C.5.2                        Technical



955 diaz - 020     C.5.2.1.1.1.5




956   vcw-24          C.5.2.1.2                     Technical



957   vcw-25        c.5.2.2.2.1.1                   Technical




958 diaz - 021     C.5.2.2.2.1.1




959
      Sklein-036     C.5.6.2.6.1    All                 T




960 diaz - 013          C.7                            T



961 diaz - 014         C.7.1        FAU_GEN.1          T
                                    P.ACCOUNTABIL
                                    ITY
962 diaz - 015         C.7.1        FAU_GEN.2
963 diaz - 016         C.7.1        FAU_GEN.2
964 diaz - 017   C.7.1   FAU_GEN.2
965 diaz - 025   C.7.1   FCS_CKM.1

966 diaz - 026   C.7.1   FCS_CKM.1

967 diaz - 027   C.7.1   FDP_DAU.1
968 diaz - 028   C.7.1   FMT_MSA.1




969 diaz - 029   C.7.1   FMT_MSA.2




970 diaz - 030   C.7.1   FMT_MSA.3




971 diaz - 031   C.7.1   FMT_MSA.3

972 diaz - 032   C.7.1   FMT_SMR.1
973 diaz - 033   C.7.1   FMT_SMR.1
974 diaz - 034   C.7.1   FTP_ITC.1

975 diaz - 035   C.7.2

976 diaz - 036   C.7.2   table

977 diaz - 037   C.7.2   table


978 diaz - 038   C.7.2   table


979 diaz - 039   C.7.2   table



980 diaz - 040   C.7.2   table
981 diaz - 018             C.7.3




982 diaz - 022             C.7.3   Para 2




983 diaz - 023             C.7.3   Para 3

984 diaz - 024             C.7.3   Para 4


985 berger - 002            D      Tables 1 & 2     E




986   df12                  D                       E




987 JL - 008                D                       E


988   df13                  D                       G




989   HD-023                D                       G

990   MercuriD50 - 030      D      entire list    General
      (formerly mercuri-
      085)
 991   Jhulshof-038      D        Ann. D page 213   technical
                                  VSS clause
                                  3.2.2.8.
                                  last colunm
 992 berger - 012     D.2.2.7.2   a, b.7, b.8           E

 993 berger - 003        E        Table 1               E

 994 JL - 006            E                              E
 995 JL - 001           Gen                             E


 996 JL - 002           Gen                             E

 997 JL - 003           Gen                             E



 998 JL - 004           Gen                             E

 999 JL - 007           Gen                             E



1000 Corry-001          Gen       Document              G




1001 Corry-005          Gen       Document              G
1002   Dill – 1   Gen        G




1003 Dill-001     Gen        G




1004 Dill-002     Gen        G



1005   Dill-2     Gen       G



1006 Gough-001    Gen   G




1007   RGH 001    Gen        G


1008 DC - 001     Gen       G/T
1009   DC - 002   Gen   G/T




1010   DC - 003   Gen   G/T




1011   DC - 004   Gen   G/T




1012   DC - 005   Gen   G/T
1013   DC - 006      Gen       G/T




1014   DC - 007      Gen       G/T




1015   Alice - 015   Gen        T




1016 Gough-002       Gen




                           T
1017 Gough-003     Gen




                         T
1018 diaz - 019    Gen


1019   HD-024      Gen




1020   GHM - 007   Gen       G
                 Comment



Change “i.e.” (“id est” = “that    is”) to
“e.g.” (“exempli gratia” = “for    example”)
we do not have to presupose the    nature of
the changes for internet voting
This standard does provide, not    will provide
tech specs.

No mention of system integrity.



MIL-STD-1521 is referenced, first in section
6.1.3, bullet “a”.

MIL-STD-498 is referenced, first in section
7.7.1
IEEE Std. 610.12-1990 is referenced, first
in section 3.33. Add reference for IEEE
Std. 610.12-1990.
Need reference for IEEE Std 1063-2001.

Reference IEEE Std 1228-1994 Add reference
for IEEE Std 1228-1994, “IEEE Standard for
Software Safety Plans”.
Reference IEEE Std 829™-1998

Reference IEEE Std 1063™-2001


Reference IEEE Std 1028™-1997

Reference IEEE Std 1471™-2000



Reference IEEE Std 1016™-1998


Reference IEEE Std 14143.1™-2000
Reference IEEE Std 1061™-1998


Reference IEEE Std 1008™-1987 (R1993)


Reference IEEE Std 982.1™-1988


Last line is incomplete.
Incomplete sentence.




Punctuation missing at end of sentence.


Incomplete sentence.



COTS - "These devices        and software are
exempted from certain        portions of the
qualification testing        process so long as
such products are not        modified in any manner
for use in the voting        system."
NIAP

Add definition for RTOS (which, presumably,
is the sort of OS a voting device uses).
In discussing the definition of COTS, this
section goes on to say, “These devices and
software are exempted from certain portions
of the qualification testing process so long
as such products are not modified in any
manner for use in the voting system.” In
general it is not a good idea to discuss
policy in a definition. In particular,
doing so here raises the question, which
portions of the testing process are
“certain” portions from which testing is
exempted.
Election laws may change regarding federal recalls, although
recall can be performed using impeachment proceedings,.




COTS, whether modified or not must be tested
at least to system level.
The definition has apparently been
truncated. "formats that"
Non-partisan should not be hyphenated
Public Counter is similar to ballot counter.
Cross reference them.
The definition doesn't identify that this is
DRE related.
This definition is different from the
definition given in the FEC Voting System
Standards adopted in 2002.   That definition
is:

“A Direct Record Electronic (DRE) Voting
System records votes by means of a ballot
display provided with mechanical or electro-
optical components that can be activated by
the voter; that processes data by means of a
computer program; and that records voting
data and ballot images in memory components.
It produces a tabulation of the voting data
stored in a removable memory component and
as printed copy. The system may also provide
a means for transmitting individual ballots
or vote totals to a central location for
consolidating and reporting results from
precincts at the central location.

The proposed definition is:

“A voting system that records votes by means
of a ballot display provided with mechanical
or electro-optical components that can be
actuated by the voter; that processes the
data by means of a computer program; and
that records voting data and cast vote
records in internal and/or external memory
components. It produces a tabulation of the
votingis notstored inanywhere thatmemory
CCEVS data defined a removable I can
find. What is it?
Add definition for "Ballot"




Add definition for "Election Auditor" and,
equivalently, "Election Observer"
Add definition for "Election Verification"




Add definition for "Sealed Ballot Box"




Add definition for "Tabulation Rules"




Add definition for "Tally"



It is a database that is being referred to
here, not a data file.
Does not specify where build is to occur.




Definition is too narrow


Can components be software? All the examples are
hardware.
Is a device that prints a paper ballot for reading by another
device a DRE? Perhaps we should say "no" and call it paper-
based. I'm also concerned about the "removable memory
component". Later, there is discussion of DREs that transfer
votes over a network. bar codes, OCR, etc. in this definition
I would like to include

Explanation about exemption is unnecessary, and may
become inconsistent if we add change requirements on
COTS
Is this meant to exclude voting systems based on computer-
printed paper ballots?
Nitpic
COTS Hardware and software should not be
exempted from qualification testing.

This exemption should not be included in
Definitions. The exemption is not a
definition.

voter verified audit record is a user
interface statement indicating that there is
an ability for voters to recognise and judge
an audit. Either this should be established
as a completeable voter responsibility or we
should leave out the word voter
We urge against defining this important term
in any other documents, more particularly in
a document that has been superseded.     The
problem is repeated in the definition: “A
voting system referred to in the 1990
Standards as a Punchcard and
Marksense (P&M) Voting System that records
[emphasis added].” Paper based voting
systems do not have to be punchcard or
Marksense.   We have the following
suggestion to improve the definition:
“Voting System that records votes, counts
votes, and produces a tabulation of the vote
count, using one or more ballot cards or
sheets of paper or a written list of
choices.” We suggest adding the italicized
words.

Furthermore, we note that the FEC 2002 VSS
has this to say about Paper Based Systems:
“Additionally, a paper based system may, or
records votes using other approaches whereby
the voter’s selections are indicated by
marks made on a paper ballot by an
electronic input device based on input from
the voter, as long as such an input device
does not independently record, store, or
tabulate the voters selections...”
These definitions rely too heavily on a
specific implementation and the term
“affiliation.”   While the definition of
Open Primary doesn’t seem to present a
problem, the definition of Closed Primary
does: “Closed Primary: A primary election
in which voters receive a ballot listing
only those candidates running for office in
the political party with which the voters
are affiliated”. One can imagine a DRE
machine that presents a first question as to
the party’s primary upon which the voter
chooses to vote. Following such a decision,
even those made within the privacy of the
voting booth, the voter is given a choice of
only one party’s candidates. While this
could be considered an Open Primary, it
falls within the proposed definition of
Closed Primary.

Finally, there is no definition for what
some call a “Blanket Primary”. In this
case, a voter may vote for candidates of one
party in some races and for another parties
candidates in another race. While there
have been court cases about the legitimacy
of such a type of Primary, there is no harm
describing it. Whether or not it is legal
is a totally different question.
Addition to References

Several NIST security guidlines are
available and should be referenced in this
standard.




We recommend that this definition be
improved. Not only is the definition
problematic, it causes significant problems
later when the Draft Standard calls for
source code for firmware. Much firmware
found in today’s electronic voting systems
will not have any source code since that
firmware (usually found in device
controllers, BIOS and other discrete
hardware units) is usually provided by third
parties who do not make source code
available.

The definition itself is problematic. For
example, is software stored on a CD-ROM
firmware? Is software stored on a hard
drive that is write-protected firmware? Is
software stored on a flash memory that can
only be modified by removal from the voting
machine firmware?
The problem here is mixing two different
ideas. The first is recall. The second is
Options. The danger of mixing is borne out
by recent court decisions in California
which allow a recall, but do not allow any
restriction on Options. We suggest two
different definitions: Recall and Recall
Options. Does a system support Recall
(according to Definition 82) if it allows
voters voting No on the recall to vote for a
successor? Recent law seems to suggest that
such a possiblity should not even exist.
Separating these shields the definition from
specific legal findings.
Ballot Scanner definition does not incorporate voter verified
products.




       Definition "Voter Verifiable Audit Record: A
       human-readable printed record of all of a voters
       selections presented to the voter before the vote
       is cast. Also called Voter Verifiable Record."
       is specifically limited to one design, paper, and
       needs to focus on performance.
Accessibility --by this definition, no equipment can be
accessible.
Add VSS & HAVA.
Add acronyms used in this draft but
presently undefined. Page where referenced
is given in parentheses.




Use of “their” as a singular pronoun of
common gender is slang; also, the draft
often, probably usually, uses “his/her”.
“States” excludes U.S. Territories, does not
allow for the possibility of more local
control, and makes the standard gratuitously
U.S.A.-specific.
Second sentence is not part of the
definition. Whether or not my later
comments on COTS are accepted, “These
devices and software are exempted from
certain portions of the qualification
testing process so long as such products are
not modified in any manner for use in the
voting system” does not belong in the
definition.
URLs, being subject to continuous change,
need to have an associated date when used as
a reference. Note that this was agreed upon
at the meeting in January, 2002.
“and/or” is inappropriate in formal English
where “or” denotes an inclusive or and
exclusive or is expressed by added “but not
both”.
The distinguishing feature of “firmware” is
not its inalterability during operation
(which can be accomplished with RAM by
various methods such as disabling the write-
enable signal), but its inalterability
without electrical or mechanical means.
“The 1990 standards” are undefined.
The word “Agency” is used in its own
definition.
This definition does not apply to all
states; in Florida, for example, there is a
“Presidential Preferential” which has
nothing to do with choosing delegates.
This section and its figure describes a DRE
system. But it is equally applicable to a
paper based system.
“This section Software and firmware
documentation, information, and materials,
including the following:

the release software, firmware, utilities,
hardware, and instructions required to
install, operate and test the voting
system.”

While this is a good idea, this requirement
is utterly impractical. There is “firmware”
in device controllers and the Operating
System BIOS. There is likely to be firmware
in places not even suspected by any vendor.
Such copies of the “source” is simply not
possible.


The word “appropriated” is a typo
Unmodified COTS are not exempt from
evaluation to preclude the threats
identified in 5.3.2.1 (A).
The definition of “datagram” should be
dropped and the definition of whatever word
is substituted for datagram added to allow
for both TCP and UDP protocols (or, more
generally, for connection and connectionless
protocols) to be covered by statements
relative to communications in the standard.
Add definition of “Voter-Verifiable Paper
Audit Record”




Change “i.e.” (“id est” = “that is”) to
“e.g.” (“exempli gratia” = “for example”)
Provisional ballots are defined on page 8.
However, the definition for 3.24 on page 5,
"Challenged Ballot," also refers to provisional
ballots.
Both these sections define “software
verification and validation”.




Voter-verified has been used in an IEEE
publication while "Voter-verified" has not,
giving the former precedence. This is not
merely semantic because it was a source of
confusion in the non-resolution of a number
of comments made to the previous draft.

This section should provide information
regarding the likely sequence of events in
using the system. This sequence of events
includes storage between elections, pre-
election activities, delivery to a polling
place, use during an election, return to a
central facility, post-election activities,
and return to storage. Anticipated
conditions at these sites should be
described.
Insufficient antecedent support for
"separate method" (separate from what? the
Cast Vote Record?). Clarify that the audit
trail starts from the voter's decision
(voter intent being the legal standard in
many jurisdictions), and is a "separate
method" from that point forward rather than
merely an alternative output mode for the
Cast Vote Record.
Sentence mentions "components" which are not hardware,
and inappropriately restricts scope to DREs. Optical scan
systems are presumably included in the discussion, and the
Much of this section is not of scope of
Election Management System is outpart of the DRE. the
current standards.

System description goes beyond an EMS used
with DRE.




Is it true that the single-arrow flows are enforced as
unidirectional? Also, do the dotted-lines indicate the fact that
there are times when communications are prohibited or
restricted?

Most pf the document is not written as a
standard; according to “IEEE Standards
Companion”
(http://standards.ieee.org/guides/companion/
part1.html#how ), ‘One of the major
difficulties in splitting up the work (the
divide-and-conquer school of thought) is
that there is often an inconsistency of
tone in the document as a result. One way to
avoid this problem is to remember to use
standards verbs (shall, should, and may) as
the primary means of conveying the tone of
your document. Standards primarily use
"shall," recommended practices primarily use
"should," and guides primarily use "may."
Remember, however, that this is not an
exclusive definition. Standards can use
"may" every once in a while, just as guides
can use "shall." Indeed, this kind of use is
almost inevitable. What needs to be
attained is an overall consistency of tone.
If a guide uses "shall" almost all the time,
with a few "mays" sprinkled in, is it
really a guide? The overall tone is
mandatory, and that can cause a problem. So
consistency in the use of verbs, and the use
of proper standards verbs, can help to
We need to eliminate the Election Management
System information, it is not in scope for
the standard
The EMS described is out of scope of current
standards.




“IEEE Standards Companion”
(http://standards.ieee.org/guides/companion/
part1.html#how ), ‘General violation of One
other aspect of standards writing that is
very confusing for working groups is the use
of the word "must." Traditionally, "must" is
frowned upon in standards writing because
its mandatory nature can be confused with
"shall." In other words, when you say a user
must do something, are you mandating this?
Or are you saying it's an inevitable result
of the situation they are in? Remember,
"must" is not a defined standards verb in
standards organizations. Therefore, the
mandatory nature of a statement with "must"
in a standard could be called into question
in a court of law, and there would be no
existing practice or rules to back up its
meaning (keep in mind what was discussed
earlier, the quasi-legal nature of standards
and the need for a clear understanding of a
standard's intent). For this reason, "must"
should be avoided unless it is being used in
a descriptive fashion (if it is raining, the
sky must be gray). Stick to the defined
standards verbs for the sake of clarity
between you and the users of your standard.’
Describe material that should be excluded
according to the Document’s Scope.

That defines, which expands. Correct
grammar.
Add to bulleted item #13


Add new bulleted item to bottom of list


Same general comment as Aragon - 04.
Address known misconceptions about
interaction between this "separate record"
and the requirement for accessibility under
HAVA and 5.3.10.
No mention of audio secrecy.




On page 15 under Precinct Voting, would need to add
provisional voting (PV) information to include the recording of
the cast ballot in a separate memory file, linked to the
provisional voter identification number. The provisional voter
ballot counter (PVBC) would be incremented for every record
added to this memory file. The provisional voters' cast ballot
information would be accumulated on one memory card, but
would not be tabulated, nor would the votes be transmitted.
. If you take this section and follow the
bullets listed under it, you arrive at:
“The precinct voting stations present the
ballot to the voter and provide capabilities
for:

has been cast after the vote is stored
successfully”
A paper based system does not have a “voting
station” that signifies that the vote has
been stored successfully.”

1st sentence "described in section 4.1" -
not described in section 4.1.
The reporting of absentee votes as "separate precincts" has
created considerable confusion in end-of-day tallying. These
are known as "ghost precincts" and it makes it difficult, if not
impossible (in some court cases) to ascertain which precinct
the ballots should be attributed to. All votes should be
attributed to the precinct in which the voter was authorized to
Beginning on to separate precincts.
vote, not sent page 16 under Reporting Subsystem, would
need to add PV information to include the printing of the
PVBC number only. No PV voting results would be tabulated
or printed until after the post election PV verification process.
"This means" is undefined.




Specifies 22 months for data retention. A
spec is not appropriate in section 4, but
should be in section 5 which it is --
5.2.2.2, 5.2.2.3, and 5.1.3.5.6.h
The second sentence is difficult to parse
because in “This means must”, it is not
clear at first that “means” is a noun
referring to the means in the first
sentence.
"rolling back" - needs to be more specific



It would be better if the device audit log
were mandated to log exceptions at all times
to have a baseline for showing that the
device behaves similarly in actual use as in
test use, and to record activity outside of
actual voting use as evidence against
tampering in the event of a recount or
suspicion of tampering or malfunction.
“A voting system provides a means for
obtaining a printed report of the votes
counted on each voting device.” This
section is too prescriptive. One can easily
imagine a DRE environment in which all
voting stations are networked and for which
there is no need for a report from each
voting device. Similarly, one can imagine a
paper based system in which there are no
voting records stored on the voting device.

Reporting sybsystems should provide vendor-independent
means for producing polling place, precinct, consolidated and
audit log reports.
change "more than one voting device" - this
is a marketable feature, NOT a standard



Specifies 22 months for data retention. A
spec is not appropriate in section 4, but
should be in section 5 which it is --
5.2.2.2, 5.2.2.3, and 5.1.3.5.6.h
The section says, “The printed report shall
contain all information generated by the
system audit log.” There may be more than
one log. Also, the printing of a log report
should be optional. A good log should be
cumulative and printing it every time may
result in reams of paper.
Consolidated reports is not in scope of this
draft
Requires logging of OS and version, hardware
and peripherals. This is not always
feasible in a custom hardware environment
and might not even have an OS. Peripherals
may be temporarily attached as part of the
election phase and it's use and not it's
presence should be included in the audit
log. (e.g. The attachement of an external
printer or modem would not be logged but
rather the printing of a report or the
transmission of results.
Identification without proof of authenticity
or integrity is meaningless.




Does not prohibit alteration or modification
of the alternate data file. This is a gaping
security hole in the current Diebold GEMS
software.
4.5.5 On page 17 under Access to Election Data, would need
to include information on limited access to the PV memory
files, which would be necessary in order to authorize the
tabulation of the valid cast provisional ballots.
Changes to election results can not be
allowed not alter these results.




Add need for consistency of alternate files with primary files.




None of this should needed; the presence of
this is evidence that this standard does not
specify a secure or mission-critical system.

Tools and components are mixed together;
however, components were specified above in
bullet 3.
It needs to be specified how updates to software are going to
be supplied and performed.




Documentation is a requirement, not an
option.

Should give general term, not specific
examples.
Need a copy of the makefile or equivalent,
with directions, in order for this to be
useful.
Flowcharts are only a specific type of
software documentation, and a mostly archaic
method.




Specification is insufficient.



Specification is insufficient.




Specification is insufficient.




Identification without proof of authenticity
or integrity is meaningless.



Need a copy of tools in order for this to be
useful; the system can not be rebuilt
without the tools and specific versions of
tools may become unavailable.
Bring into conformance with IEEE Std 1063-
2001, “IEEE Standard for Software User
Documentation”



There needs to be a statement that the
system shall not automatically shut down or
time out during any official part of a
voting-related procedure, such as filling
out a ballot by a voter, except in
accordance with the requirements of voting
law and procedure. This is needed to make
clear that features such as implied by
O.ABORT_SESSION in the current draft of
C.4.1 are prohibited except as specifically
provided by law.
Randomization used for privacy protection
must be based on random events or random
noise




Need to incorporate security-relevant
requirements on operating system features
and software style that had been previously
contained in software section. There
especially needs to be protection against
improper input or buffer overflow allowing a
malicious perpetrator to input data and or
executable code.
Last three sentences should be separate
paragraph.




“Using validated products can significantly
reduce the cost ... by providing information
on how to securely configure a particular IT
product within a system” makes no sense.

It is unclear if “vendors” means “COTS
vendors” or “voting equipment vendors” in
“vendors must adequately describe the
control methods they have employed to ensure
these risks have been mitigated.”
There is a change of gears just past the
middle of the paragraph.
The COTS products may also be subject to a security
evaluation themselves; such evaluations can support the
voting system evaluation process.
How can Challenge/Provisional ballots be
verified if not associated to a voter?

"risks have been mitigated" is incomplete
The standard is supposed to address
unauthorized modification of the system.
But there is no full proof way to detect any
unauthorized modification.
This is a far too vague and does nothing to
address the security issues.




Need to add material at end of paragraph.




Original comment language "independent auditability" should
be acceptable. (mercuri-086 in 4.3 comments). The word
"audit" in common parlance is so frequently preceded by
"independent" (or some synonym) as to indicate common
practice, and therefore does not "make policy". The originally
suggested language is permissive: If the equipment
facilitates an independent audit, policymakers would still be
free to permit collusion in the audit, so policy is not
constrained; whereas if it does not provide that capability,
jurisdictions will lack it even if policy permits or requires it.
Therefore to the extent that the original comment does go to
COTS that be properly than discourages
policy, may fact favors rather installed and the inclusion
configured but still not meet requirements
unless latest security patches are
installed.




“acceptable” is not quantifiable and,
therefore, does not belong in a
specification.
In a dedicated device, the operating system
should be, specifically, a real-time
operating system.
The treatment of COTS products contradicts
section 5.1.2.2, “Elements of Security
Outside of Vendor Control”.


A voting machine is device in its own right
(or at least must be tested as such), not an
“IT system” made up of sundry components
thrown together. “specific requirements”
refer to what this draft endeavors to
specify.
There is implied a lack of testing in “COTS
products require updates due to a detected
security breach or vulnerability”; nothing
that requires an update should pass testing.




“The voting system vendor must provide a
method to assess the impact of COTS updates
on the voting system, as well as a method
for providing notice and distribution of
updates to purchasers” is inconsistent with
IEEE Std 1012-1998.




Memory leaks are the result of using C++
language inappropriately; they are not a
risk of a COTS C++ compiler.
Memory leaks can not be tolerated in a real-
time system; normal design of real-time
systems prohibits dynamically allocating
memory after initialization or freeing
dynamically allocated memory.
Memory leaks in C++ is not an example of an
inherent risk in COTS products.
The section talks about requirements, but
there are none!
“General Purpose Computing Equipment” is not
controllable, that is, there is no guarantee
that such a device is equivalent to the
device used in V&V, and therefore should be
prohibited (unless each individual unit is
fully tested).
“Any components developed by a voting
jurisdiction” fall outside of this
standard’s scope
What does "to include vendor or contractor facilities" mean ?
This section talks about requirements, but
there are none!
Need to consider security certification in
the context of full system (procedures and
technology)

While these elements may not be under the
direct control of the vendor, it is still
necessary to include security requirements.
Otherwise, the entire system could be
compromised.
“Data communications security” and “Risk,
response and recovery” are not “outside of
vendor control” and do not, therefore,
belong here.
What is the scope? How is it intended to be used?



Some cross-reference of how the standard
addresses each threat would be useful.
Add additional assumptions.




Add activation of malicious code



Add additional threat




Add paragraph on assets (i.e., the goal of
the attack)




Threats are based on the stage when a
vulnerability could be exploited instead of
the asset goal. I believe a comment was made
at the Denver meeting to "follow the vote"
which aligns with this philosophy.
Rework of 4.3 comment (mercuri-090). The duty cycle of
voting equipment differs from many other computer systems.
The difference is legitimate matter to consider in the threat
model and should not be dismissed as policy. a
Motivation alone isn't necessarily
security risk. One needs training and
ability as well.

System administrators or others could modify the source
tree.




Development phase could actually include anyone who has
custody of the software prior to installation. Depending on the
procedures in use, this could include the ITAs.



Others may have access to voting machines
Add a threat that a voter or election
official surreptitiously connects an
external device to a voting machine and
tampers with the machine or its data using
functiionality resident on the external
device.




“correctly” should be “incorrectly”
Election Verification
It needs to be made clear that voting
equipment should be dedicated to voting, or
if not dedicated should be converted to
voting by a procedure similar to conversion
between unclassified and classified
processing under “System High Mode” of
chapter 8 (Automated Information Systems) of
the National Industrial Security Program
Operation Manual.




There needs to be a section expressing the
controversy about the suitability of all-
electronic systems to provide protection
against combinations of malicious activity,
human error, and/or equipment failure that
could cause a ballot to be recorded that
differs from the intent of the voter. The
section should also point out that HAVA can
be interpreted as requiring a voter
verifiable paper audit trail.
Voter verified paper needs to be mandatory
under certain circumstances




Requirements assume that all threats can be
prevented instead of allowing for, in
addition, detection and deterrence.




The basic requirement that the vendor design
to counter and defend against the threats is
inadequate, since there is no way to protect
against all of the threats without a voter
verifiable ballot.

---- vendor provides documentation explaining how each of
the threats in Section 5.1.2.3 is countered by the system
design. --- This documentation should be checked by
computer professionals not associated with the companies
who make detailed checks of the software used in the voting
machines. The computer companies should then provide the
more improved software to the county election officials..
Wireless connectivity should specifically be precluded here,
or somewhere else appropriate in this section.




Need to note audit trail for access



These sub-sub-subsections were apparently
deleted and/or renumbered.




System audit generated creation and maintenance of audit
records
reduces the chance of error associated with audit records ---
this is true if the software has been programmed properly.
However, if problems arise due to programming errors or
Instead of allowing the vendor be specify
stealth programming the inaccuracies can to much greater.
the features and capabilities of the access
control mechanisms, a set of strong vendor
requirements should be included in this
document.
The vendors shall specify the features and capabilities of the
access control mechanisms to provide effective voting
system security. --- This information should be checked by
computer professionals not associated with the companies to
There are validity.
guarantee its no minimal general access
requirements given.
Passwords cannot be nulls or common words,
e.g., password, secret, family names, etc.,
and shall contain at least one numeric or
non-alpha character, e.g., ~ ! @ # $ % ^ & *
( ).

Insufficient security.


Authentication of components which
communicate with each other REQUIRES the use
of unencrypted key storage (unless an
operator supplies a key at the time of each
exchange), rendering items d & e
incompatible.
Can't identify each person who has access
granted, it violates secrecy of ballot
Reason needed for role-based accounts.



Default superuser accounts, e.g., root, in
all software or operating systems must be
disabled.




There should be minimal security algorithmic
and key length requirements for the
encryption.
Paper record specification s overly prescriptive.


Paper record specification s overly
prescriptive

Instructions for paper ballots are unclear. Does the printer
have to randomize the order of the ballots?
Item is unclear. What does "multiple-language presentation"
mean? If it means voter can selection English + their native
language, it may be ineffective at concealing voter identity,
The acronym "CVR" e used
as well as lengthening thisballot without
definition. A definition can be found 9
pages further on in 5.2.1.2.
text includes "Vote Verifiable Audit Record"
which is inconsisent with the previous
paragraphs.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.

This section says, “and if a paper copy of
voter selections is printed for voter
verification, deposit the detached paper
copy into a sealed, opaque “ballot box”.

This section mixes a couple of ideas and is
too prescriptive. First, we point out that
that a printed paper copy may be printed as
the ballot, and not as a later
“verification” or audit. In such a case,
there is strong reason for the voter to take
the ballot away from the voting booth (but
not out of the polling place) for deposit on
one central ballot box. The language of
this section prohibits that. The important
point is that this section must allow either
for the voting station or the voter to
deposit the printed record.

Add ballot integrity.


Prescriptive language (e.g., "a paper copy
of voter selections is printed for voter
verification") which is inappropriate in a
performance standard. The language should,
and already does, indicate that any feature
shall "ensure vote secrecy."
Ballot image may contain embedded code that
could identify sequence even though it does
not have a specific field that does so.


Should also include provisional ballots

Voter Secrecy - DRE System section e should read ----
Voter Verification Audit Record must be provided (not may
be provided) so the voter is sure his vote is counted properly
“In systems providing voter interaction in
multiple languages, the CVR and all copies
thereof,
and the Voter Verifiable Audit Record if
provided, shall be free of indications of
the language
selected by the voter, except where the
system provides and the voter explicitly
selects a
multiple-language presentation of the Voter
Verifiable Audit Record.”

This section has an ambiguity in it.

What is meant by “the voter explicitly
selects a multiple-language presentation?”
Does this mean that voter chooses to see a
presentation in more than one language
(“multiple language”) or does it mean that
the voting system supports more than one
language but the voter only sees the one
language the voter selected? We think the
latter is what is intended and suggest that
clarification. Also, we are not sure what
the intent of this element is. If it is
intended to help protect privacy we support
that intent. On the other hand, the ability
to determine what language a group of voters
voted in could be important. One reason for
this is to find out if there are any
systematic problems with the ballot based on
"Immediately after the voter chooses to cast his or her
ballot, record all voter selections as a CVR in the
memory to be used for vote counting and audit data
(including CVRs), erase selections from all visual
indicators, buffers and all other temporary storage, and
if a paper copy of voter selections is printed for voter
verification, deposit the detached paper copy into a
sealed, opaque “ballot box”; and" The highlighted
portion indicates only one possible of "voter
verification". Placing a "paper copy" of the ballot into a
In systems providing voter interaction in multiple
languages, the CVR and all copies thereof, and the
Voter Verifiable Audit Record if provided, shall be free
of indications of the language selected by the voter,
except where the system provides and the voter
explicitly selects a multiple-language presentation of
the Voter Verifiable Audit Record.
Paragraph implies that voter verifiable
ballot might be available. Current
technology does not support a cost effective
solution to provide this capability. In
other words it is not reasonalble to provide
this capability.
On page 26 under Vote Secrecy (DRE Systems), processing
provisional ballots after an election would rely on a linked
association between voter identification number and the
CVR. The security issues raised in this section would need
The modified.
to be last sentence is totally open ended and
not useable in its current form. There are
physical tamper resistant methodologies
available on most election systems, but this
sentence is unsupportable or testable.
The only way that the firmware can be
verified is by making it open source.
The vendor is told to provide a mechanism
for security access to the voting devices
after successful completion of election day
testing and the method to identify operaion
of an override feature during the election
cycle if provided. This is not adequate;
minimal security requirements should be
stated in this document.
Some manufacturers allow firmware to be changed between
elections, this could occur during system operation must be
prohibited from being changed during system operation
during an election.
Allowance of an override feature is a significant security risk
that should not be permitted under any circumstances.




Add a paragraph that includes the ability of the audit trail to
be used for checking tallying and data collection.




What applies to firmware applies to all
software.
For security reasons, the identification of
software level must be automatic.
For security reasons, the identification of
software level must be validated.




For security reasons, the software must not
be able to be altered without authorization
.


Ambiguity in “acceptable by NIST”.
Bootstraps must and monitors always reside
permanently as firmware. (Unless we want to
provide for using toggle switches to enter
the bootstrap each time the device is
powered up.)
This makes no sense and is contrary to
common practice in embedded systems; the
operating system and application code
(“election-specific programming” here) are
linked together and installed on the same
ROM(s); when resident on unalterable ROMs,
the bootstrap typically is likewise so
linked in.
Unnecessary and unacceptable security risk
in allowing an override feature.
What is the purpose of keeping election-
specific programming separate from bootstrap
or operating system code?
Paragraphs 3-4 The 3 points should be
indented and numbered 1-3.
There is no way to adequately test against
all possible bugs and malicious code in
COTS.
Some systems may not be capable of auditing
"…all process executions and terminations,
and…the alteration or deletion of any
memory, file or database object or entry."
The voter actually alters election data -- this must not be
audited because of violation of ballot privacy. Why would
anyone be allowed to alter or delete election data?



Add additional phrase pertaining to voter initiated deletions, if
recorded.
This contradicts section 5.1.3.6.8. Allowing
unrelated tasks to run simultaneously with
the election software is an absurdly
unacceptable risk that negates and V&V
performed.


Use of “servers and workstations” can not be
permitted, both because of unknown other
software that may be present, and because
the hardware is not controllable, that is,
there is no guarantee that such a device is
equivalent to the device used in V&V, and
therefore should be prohibited (unless each
individual unit is fully tested).
“Unauthorized network connections” must be
disallowed by design.




The system shall be designed such that these
precautions should not be needed.
How is the system going to protect against Trojan horses,
etc.? I don't think there is any way to do it.
The only way to protect against malicious
software is by using voter verifiable
ballots
Add time to logic bombs and also additional material
pertaining to any actions by voters or administrators that
could activate malicious code.


This far reaching all purpose statement adds
confusion rather than entlightening the
reader. Protections described may or may not
be potential threats based on the voting
system design.
Need to specify single or combination of
keypresses and to specify other types of
processes.
By mandated that a voting machine be
designed as a secure system and by
disallowing it to be constructed of sundry
other parts (such as general-purpose
computers, Microsoft Windows, discarded
bleach bottles, pipe cleaners, Microsoft
Access, et cetera), this section is not only
not needed but is a travesty.
Ambiguous (“Prior to” could mean at V&V),
insufficient (any memory used could affect
the performance) and no specification for
what to do in the event of a failure.




“Public network” is not defined.



Sub-items 2-5 all contain the words “in
human readable format” and item 1 should.
“Time and data” is a typo.
There are numerous problems with using
“local time and date”.


“A means to verify that date and time are
correctly set prior to any election” is
inadequate.




Correct sentence structure and section
reference.

System Operating Manual referred to
redundantly
Define "indestructible" since it could be
interpreted conservatively to mandate
physical medium that is infeasible.

What's the performance standard for the
audit trail? Should it be single (double,
triple) fault tolerant?
There is a possible conflict between the
last sentence in paragraph 4 of 5.1.3.5.1
"The Technical Data Package shall be subject
to public scrutiny and may not be regarded
as proprietary." and section 7.1.2 which
states the need for putting the same
information into escrow. The implication in
7.1.2 is that the information is not open to
public scrutiny .
Document states "described in section 4" .
Should identify specific section 4.5.4.
(Could also reference section 5.6.4 in
Software and functionality for more
application audit details.
The technical data package, as currently
defined includes database layout
descriptions and program logic flow.
Furthermore, some may consider that the
source code is part of the TDP.   Even if
the source code is not considered part of
the TDP vendors must disclose a great deal
of proprietary information in the TDP.
Making the TDP public will permit anyone to
duplicate a vendor’s system. Doing so
removes the incentive to create good
election systems and this requirement will
have the consequence of degrading the
quality of our electoral systems.
The second sentence says that election audit
trails "present a concrete, indestructible
archival record of all system activity
related to the vote tally, and are essential
for public confidence in the accuracy of the
tally, for recounts, and for evidence in the
event of criminal or civil litigation."
This is a false claim. The audit trail is
meaningless in as far as a recount is
concerned, because there is no way of
knowing if the vote recorded in the machine
accurately reflects the intent of the voter
unless there is a voter verifiable ballot.
Also, the existence of an audit trail should
not be sufficient for generating public
confidence in the accuracy of the tally.

The paragraph should reflect the proper
access control requirements for audit data
that were erroneously stated in Appendix C.
This text is redundant, duplicating a small
part of text found previously in 5.1.3.5.1.
The sentence makes a false claim that audit
trails are essential to ensure the integrity
of a voting system and to retain public
confidence in the election process.
 ---- vendors shall supplement it ( they are referring to the
design construction) with information relevant to the
operation of their specific systems. ---- This supplements
should be conducted by professional computer scientists not
association with the vendor company before the company
provides the voting machines instructions to the county
The M650 conduct the voting and to the operators.
officials whodoes not include the "readiness
report" in its audit log…
The restriction in parenthesis prevents a
function that is provided in many DRE
systems today that is required for ITA
stress testing and is used by jurisdictions
in election specific L&A testing. Note that
this section is in conflict with 5.6.7.2.1
(section f and the concluding paragraphs
below) which allow automatic ballot
generation as long as it can't occur during
election day voting. THIS INCLUSION IS THE
REASON FOR MY NO VOTE. My vote would be Yes
if this were changed.

This is not possible. Correct counting logic
can only be verified by human operators.

With any moderately advanced operating
system, it is not possible for software to
check "all data paths and memory locations"
which might be used in vote recording.
It says that " The ballot interpretation logic shall test and
record the correct installation of ballot styles or formats on
voting devices for the voting precincts at the polling location
and that the ballot logic produces a correct count for each
candidate and issue on the ballot (NOTE: The system shall
not automatically generate voted ballots)".Note that this
section is in conflict with 5.6.7.2.1f) and the paragraphs
below which allow it as long as it can't happen with real
voting. This restriction would severly hamper ITA as well as
---- Prior to opening of polls, a system process shall verify
hardware and software status and general readiness for
audit status. ---- The lack of readiness in certain Florida
districts lead many people to leave without voting because
they had other important things to do such as going to work.
Two hour delay in order to vote due to equipment failure is
inexcusable for equipment not working and appropriate
solutions for this condition must be made available.
"The ballot interpretation logic shall test and record the
correct installation of ballot styles or formats on voting
devices for the voting precincts at the polling location
and that the ballot logic produces a correct count for
each candidate and issue on the ballot (NOTE: The
system shall not automatically generate voted ballots);"
unreasonably restricts systems from conducting test
The auditto verify election readiness and reliability.
ballot runs log cannot contain a verification
that the voting system is in the location
for which it was programmed. This must be a
pollworker function. It can only identify
the location for which it was programmed.

Unless you are talking GPS, the voting
system cannot verify it is in the proper
location
How can the system verify that date/time and
location are correct? Must there be an
atomic clock and GPS receiver attached to
each system?
this registration may not have a timestamp
voter privacy
Codes may be all an audit log contains so
that printing tapes can be an efficient
consideration
The use of a secure time/date stamp protocol should
supplement the human readable format.


replace practical with possible


The audit record "shall be available for
review at all times by authorized election
officials" -- is the iVo audit log available
on screen to an "authorized" operator?

Does this require that electronic copies be made of paper
records?

no real time clock can activate fraudulent
code
audit trail shall not include cvr

On page 30 under Time, Sequence, and Preservation of
Audit Trails, the provisional voting concerns in this paragraph
are the same as 5.1.3.2.5.d.
Missing requirement.




This is for a "Local Area" Network,
presumably isolated from any external
connection. These security concerns are
ridiculous.
The term “datagram” normally refers to a
message transmitted using the Universal
Datagram Protocol (UDP). Unless it is the
intent of the standard to require use of UDP
(e.g., as distinct from TCP) or to limit its
statements to messages transmitted using
UDP, another synonym for “message” should be
selected and used in this section.

Devices connected to the voting system must
be certified as well. I also question how
this might be enforced?

Voting systems using sonic or physical (via
liquid, for instance) transmission of data
are exempted.

The use of wireless and open air transmissions leaves data
available for monitoring, corruption, and/or jamming,
regardless of error detection etc.. This practice should be
prohibited.
Need to add to access control end of paragraph.


“Datagram” is not the correct term here as
it implies use of UDP.
"The network shall be so configured that only
datagrams authorized and required by the voting system
appear on the physical network medium and that
datagrams from the voting system are not transmitted to
non-voting systems." is not gramatically correct based
on the preceeding sentence. keys will be managed to
               "Encryption
                ensure the keys are not compromised and
                that the keys are changed on a periodic
                basis." is not gramatically correct.
Wording not consistent with introductory
sentence "…systems shall:
Ambiguity in “for sensitive data”.


Item a essentially requires an isolated
network. Why not state that directly? In
either case, items b-d are unnecessary
overkill once the item a requirement is met.

References to encryption and keys are
specific technology implementations

Wording not consistent with introductory
sentence "…systems shall:
This is redundant to verbiage in section
5.1.1.
Additional requirement to eliminate damage
that could fall through from the existing
requirements.




Additional requirement to eliminate damage
that could fall through from the existing
requirements.



COTS software was already covered in 5.1.1.

Eliminate "antennas"


This is unnecessary if all communications
are initiated by the voting device.




Reporting of breaches needs to be added.




Item is unclear.
This is a requirement.
This contradicts section 5.1.3.4.2.

Grammar be confused.
References to encryption and keys are
specific technology implementations

Devices connected to the voting system must
be certified as well. I also question how
this might be enforced?




Election Verification




Voter Verification (cast-as-intended)




Intent Capture




Casting in the Ballot Box




Election Verification (counted-as-cast)
Sealing the Ballot Box



Scrutiny of Sealed Ballot Box




Results Reproduction



Missing "to" before "require additional security
measures"




Does this really need to be enumerated?
Presumably, this precludes any form of
encoding the data, including compression and
encryption.
Report of votes cast must not be in proprietary format.


Computerized ballot casting systems that provide voter
verified ballots should not be required to necessarily maintain
an electronic copy or tally of the results.

While a voter-verified paper copy may be
optional, if a paper copy is created and
verified by the voter then it must be the
official or correct copy.
On page 35 under DRE System Standards, the voting
devices would also need to retain a redundant copy of the
provisional ballot CVRs.
The wording of this section relates to our
earlier comment about the change of
definition of a DRE. We suggest the
following change:

As an additional means of ensuring accuracy
in systems in which votes are counted on the
voting device itself, voting devices shall
record and retain redundant copies of the
original CVR. This is a requirement whether
or not a paper copy of voter selections is
printed for voter verification. (A CVR is an
electronic record of all votes cast by the
voter, including undervotes.) There shall be
a pre-defined hierarchy to determine which
copy is to be deemed the "official" or
"correct" one, in the case redundant copies
do not agree in data content (due to
corruption or loss of information).


CVR can be stored in memory modules
"As an additional means of ensuring accuracy in DRE
systems, voting devices shall record and retain
redundant copies of the original CVR. This is a
requirement whether or not a paper copy of voter
selections is printed for voter verification." specifies
only not convinced that this is the the CVR.
I amone purpose for creating a paper copy ofbest way
for a well funded operation to through an
election. This retoric does not belong in a
standards paper
new assumtion




we do not know that voters can reliably
verify
we do not know that voters can reliably
verify
we do not know that voters can reliably
verify
Why two different rates? Only the lower rate
is important. This is equivalent to
requiring "a 10-digit calculator on which
the first 6 digits are correct."
Since even a single misplaced vote could determine the
outcome of the election, the last sentence must be changed.
There must be a way to differentiate between deliberate
undervotes and votes lost by the system.

No mention of optional paper copies with DRE
systems.


Are these accuracy requirements realistic for optical scan
systems? I've heard of error rates on the order of 1%. I
don't understand what it means to "achieve a target rate".
Accuracy requirements seem to apply to touch screens.
Testing 1/10,000,000 on a touch screen would seem to be
very difficult (and it is unlikely that that level of accuracy can
be achieved). marks are placed by, and later
when those
read by a computer, may be located anywhere
on the paper ballot. The text as drafted
could be improved without any loss of
generality and accuracy by dropping the
words “ballot position”.


On page 36 under Processing Accuracy, it states
that following an election a consolidated report
would be generated containing absentee,
provisional and other votes.
How would this be know? Is there any
methodology to determine the "ERROR FREE"
requirement? If anyone simply states it's
"ERROR FREE" is that sufficient.
Rework of 4.3 comments (Corry and aragon-014). This
section in 4.3 referred to device-level redundancy as a
means of achieving the functional goal of stability. Corry
comment was to apply functional requirement to all voting
equipment, not only DRE. Aragon comment was to clarify
that functional requirement did not depend on design choice
of redundancy (requirement applies to module as a whole
regardless of internals). Resolution was to introduce a
design requirement, for DRE only (opposite of Corry),
specifying a particular type of redundancy, and then
reversing application of aragon-014 to the changed sense of
redundancy (data record level vs. device level). Thus it has
lost ground from 4.3 (relative to the comments made against
4.3), and has also lost the reference to redundancy at the
device level which appears to have been the original
5.2.2.2 On page 36 under Memory Stability, the DRE
systems would also use redundant memories to store
provisional ballot CVRs.
"demonstrated error free data retention for
22 mos" - are you going to have 22+ month
certifications
Willl this require a 22 month certification
process?
Paper ballots are a type of removable
storage media.
Magnetic media can be altered.



The proposed draft maintains cautions about
electromagnetic radiation. Many touch
screens are made for commercial environments
and therefore meet FCC Part 15 Class A
requirements. It is generally not
admissible to allow TV sets or radio
broadcasts into a polling place. Therefore,
Class A requirements should be admissible.
This particular section does not rule out
Class A, but the electromagnetic radiation
language seems to set the stage for the more
expensive and less inclusive Class B
requirement.
What does it mean to "protect against a single point of
failure." Are you proposing redundant power supplies?
Redundant touch screens?!
The draft states:
In addition to the common standards, DRE
systems shall:
a. Maintain a record of each ballot cast as
a CVR using a process and storage location
that differs
from the main vote detection,
interpretation, processing, and reporting
path;

The problem in this section is similar to
the problem in 5.2.1.2.
Should allow voter verification to be applied here (This item is
similar but not the same as the one that may appear in David
Aragon's comment set.)

Retention of CVR: in 5,2,2,3 we required
that the medium can hold the vote for some
amount of time, but if it's a rewriteable
medium, it could change anyway. This
concern was brought to me regarding flash
memories, but is also applicable to
writeable media generally, including paper.
Clarification: Suppose a touch screen machine prints an
optical scan ballot that the voter then feeds into the optical
scanner in the usual way? Is it a DRE, or a paper based
system? I hope it is not required to keep redundant
electronic records of the votes.
        "These are requirements whether or not a paper
        copy of voter selections is printed for voter
        verification." limits the purpose of a paper CRV
        to voter verification.
The event prior to the error or failure may have caused it.


Grammatical error

Repeated word “consist … of consisting”of
We question the requirement of an MTBF
163 hours for which a failure is any outage
is a “Degradation of performance such that
the device is unable to perform its intended
function for longer than 10 seconds”. Such
a requirement runs counter to some modern
software philosophies that point out that
recovery is a viable strategy to ensure
overall system readiness. We call the
Working Committee’s attention to the work of
Professors Armando Fox and David Patterson.
In a recently published article in
Scientific American (Scientific American,
Volume 288 Number 6; June 2003 pp 54-61),
Professors Fox and Patterson write: “Rather
than trying to eliminate computer crashes –
probably an impossible task – our team
concentrates on designing systems that
recover rapidly when mishaps do occur.”    We
suggest looking at the problem not from a
MTBF point of view, but from a system
availability point of view as section
5.2.6.2 does. Outages can be caused by many
factors outside of the system designer’s
control. This includes alpha particles
found within the plastic packages on any
semi-conductor to stray electromagnetic
fields that penetrate reasonable shielding.

Looking at the effect on a voter, 10 seconds
of outage of 163 hours represents 10 seconds
Early voting districts may require longer than 163 hours of
MTBF.
An MTBF of 163 hours is ludicrously sloppy,
suggesting that in a jurisdiction with
50,000 machines where the polls are open for
12 hours, there will be more than one
failure per second.
1. An average like MTBF is not a good
measure of reliability for a mission
critical system. A specification like time
to next failure would be much more
meaningful to the voter!

2. If MTBF is to be used, there should be an
explanation of how 163 hours was determined.
suggested_remedy = 1. Specify that there
shall be no non-recoverable failure that
impairs the accuracy of voting during the
time the voting equipment is in operating at
a polling place on a given election day.
Recovery from failures shall include a paper
trail of votes cast.

3. Explain the basis of 163 hours.


The 163 hour MTBF specified implies a 9.2%
probability of failure per machine during an
election. This is unacceptably high.


The scenario omits the storage portion of
the system usage cycle and fails to specify
the equivalent of a maximum failure rate
during storage.


"mean" is undefined in sentence.




“Shall achieve at least nine-nine percent
availability” seems rather permissive and
would erode public confidence in the device
if not exceeded.
“Recommended …spare … components” suggests
that the machines be repairable and,
therefore, modifiable during an election.

No where does the standard reflect the accuracy and
reliability of the scanners used at the precincts to alert voters
using mark-sense ballots to their correctness in preparation.



On page 39 under Availability for DRE Systems, the same
listed considerations would be applied to provisional ballots,
except that the votes from the consolidation of provisional
ballot CVRs from multiple units would not be tabulated, and
only the total PVBC number would be reported.
consolidated vote data should be referenced
only for a given equipment type, not all
types in the precinct
Remove 'DRE' in first sentence to be
consistent with document scope.
Change 'catch errors' to 'identify errors'.
Replace 'such' with 'these'
Add to acronyms list - HFE - Human Factors
Engineering
ref: Embedded in the DRE is software... This
is not true in all DREs

dots are not usable for reference
Position of parenthesis and its contents
detracts from readalbility of the sentence.

ref: information is properly grouped. why is
this a standard? This is the responsibility
of the county
why is this a standard? This is the
responsibility of the county
The required size is not defined in this
table but the required size is in section
5.3.10.2-1. I don't understand what this
table is telling me. Do we have to have the
ability to display a different font size for
someone that is 6'9" tall because they are
viewing the ballot at a greater distance?

This section can be applied to non-DRE systems also, as it
relates to input while the definition of a DRE relates to its
output and tallying capabilities.
Dimesions should not be in millimeters as
that accuracy is not realistic. Also, inches
are normally related to centimeters in
metric units.
The first sentence states that the section
is intended to apply only to DRE systems.
However, some of the principles stated in
the section apply to non-DRE systems as
well. There needs to be something said
about usability for non-DRE systems
Change 'standards' to 'requirements'.




speech input is thread to privacy poll
workers can hear and count number of yes and
no
Add recountability.
Add ability of election officials to ensure between voter uses
that equipment is still secure and functioning properly.

"should be tabulated" -- to this
recommendation for how the machine is used
in combination with others, add an analogous
requirement on the individual machine.

NO! Can't require jurisdiction to provide
accessibility assistive device (eg.,
disposable sip 'n puff)

5.3.10 still does not give adequate guidance
to manufacturers as to how an audio
interface can be made to offer "the same
voting capabilities and options" per
5.3.10.2 (and HAVA), particularly with
respect to VVAR. Public debate has included
statements that a VVAR cannot be implemented
accessibly, or that it need not be. One
proposal and prototype from a major vendor
requires use of a personal assitive device
(hand-held scanner), which State of
California would reject on HAVA concerns.
Standard should pro-actively address this
confusion to facilitate development of
compliant systems.

In 5.3.10.1, should clarify applicability of
point 1 to VVAR.
it must not be possible to connect two
headphone thread to privacy
…so that it is not possible to differentiate
votes from the "accessible" voting machines.

I don't see how this can be a standard. All
voting systems including paper based system
would fail if this standard is put into
place. How will a quadriplegic use any
voting system without an assistive device?
The following bullet: “Provide auditory
output using two distinguishable voices. One
voice should be used
exclusively for communicating instructions
and the other should be used exclusively for
communicating content” is overly
prescriptive. We suggest removing it.


The requirements make no provision for
multiple languages.

Furthermore, a missing, but important
requirement allows voters who have visual
disabilities to have ballots verified or
audit trails handled in the language in
which they voted.

display choice is discriminating solution
against full face machines with printed
ballot lay out. Privacy must be guarentied
inanother way
…with graphic elements scaled
proportionately. Scaling graphical elements
is much more difficult than scaling text,
and in some cases clarity is decreased
rather than improved.
Same general comment as Aragon - 04.
Clarify that this requirement in 5.3.10.2 is
applicable to VVAR, and that a VVAR
complying with this requirement is
realizable.
fond size is a solution discriminating other
solutions as magnifying glasses, mirrors
increased light and high definition printing

Clarification that voters with visual impairment have same
voter verifiability information as sighted voters (This item is
similar to but not the same as one in David Aragon's
comment set.)




Should not indicate a more preferable
approach.
We should not require or even recommend
Braille be used for the reasons stated in
the paragraph
the font height stated in this item is
defined as an option in table 5.3-1 and now
as a requirement?
ref: the use of the word option. The audio
ballot mode was designed as a stand-alone
voting mode - separate from the visual
ballot mode and not all options are valid in
both the audio or visual mode. For example,
the write-in keyboard is a single image on
the visual ballot but on the audio ballot it
is 40 separate sound files.
ref: normal display. Does this mean the
visual ballot?
ref: audio and /or visual feedback. Is this
a combined audio and visual ballot?
does this standard apply when a voter
receipt is part of the system?
point 2 is missing
Wireless is a security and privacy risk.



wireless communication does not offer
guarantied privacy
speech input is thread to privacy poll
workers can hear and count number of yes and
no
No provision for voters with prosthetic
devices.

this is a solution discriminating other
equal or better solutions like highly
accesable build in controls or controls on a
swiffle.
Additional controls to place in a voter's
lap? Cost?


Figures 5.3-7 through 5.3-9 are wrong.
Figure number missing for 5.3-7
this is a solution discriminating other
equal or better solutions like highly
accesable build in controls or controls on a
swiffle.


this is a solution discriminating other
equal or better solutions like highly
accesable build in controls or controls on a
swiffle.
Add privacy screen note.
All subsections need to be rewritten.
Dimenstions need to be in centimeters and
inches, not millimeters and inches.
Precision without accuracy error.
122 cm or 48 inches is further than a person
can reach sitting down. Figure 5.3.6 shows
the 122 cm (48 in) dimension as the distance
to the rear of the wheelchair wheels, not
the distance a person can to reach in front
of them. Forward reach for a person sitting
down should not exceed 60 cm (24 in).

Again, a person sitting down cannot reach
122 cm (48 in) over an obstacle sitting
down.




A person sitting cannot reach a control that
is 1.2 m (48 in) away from them.




A person in a wheelchair cannot easily reach
a control that is only 25 cm (15 in) above
the floor


Again, a person sitting down cannot reach a
control that is only 38 cm (15 in) above the
ground.



The horizontal surface referenced is
undefined.



there should only be a minimum titlt


I don't understand how information
presentation and large font affect privacy.
If it is referring to these as increasing
voter independence then the wording should
change.
Indents are inconsistent
Indicates that text should be left justified
and ragged right. Should this not depend on
where the selection target is located.
Agree that this is correct for a left
target. But should this be optionally
different for a right target (vote
indicator)?
These paragraphs and incorrectly indented.

The proposed text reads:
“2. Instructions should ascribe to the
following design practices:
· The voter should have the option to choose
from available languages (as required by the
Voting Rights Act of 1965). Translations
should be independently verified to ensure
correctness.”

The problem is that the term “available
languages” is somewhat vague.

We suggest clarification of “available
languages”.
Systems may only be used in jurisdictions
for which the systems support languages as
specified in the Voting Rights Act of 1965
as amended.

table is a solution discriminating other
solutions as magnifying glasses, mirrors
increased light and high definition printing

there is a big array of coulor blindness

Need to address tallying under language options.


Color bias



The table indicates 3 sets of values with
columns for Minimum, Preferred and
Recommended Option. (the column headers
also don't line up). It is confusing as to
what is the required specification
Too many options to test or require

The sentence lacks specificity. Does this
mean the luminace and contrast requirements
can be met thru multiple ballot choices or
machine adjustibility. Previous paragraphs
state that "if display is user adjustible."
why is this a requirement. The ballot design
should be such that all people regardless of
visual impairment should be able to read the
display.
contrast ratio range is too wide. Also if a
vendor meets the minimum are they considered
in compliance?
ref: bold should be limited. Bolding when
used in a deliberate fashion should be
acceptable
ref: well-known graphics. What are well-
known graphics? This could be completely
divergent things to vendors.
focus testing not well know graphics. Isn't
this taken care of in section 6.3.3.1
ref: # of colors to be used. Is this
referring to the number of colors on a
single screen or the total numbers of colors
used in the interface. If it is the total
number of colors - this is too low.
redundant to 8th bullet
number 3 - 5 in the process sequence are
wrong.
In Paragraph 7, the draft says, “After
voters submit their votes, the system shall
inform the voter that their votes have been
properly registered and the voting process
has been concluded.” Paper ballot systems
have voters deposit votes in a passive
ballot box.. A passive ballot box may not
comply with the requirement that the
“system” inform the voter….” We suggest the
following change.

“Through the use of signs, displays or other
alerts, the voter should be assured that the
voter’s vote has been cast after the voter
submits his or her vote.”

Similarly, paragraph 10 as drafted may be
impossible for paper based systems. T

the voter shall always see the full set of
options within a contest, if not the options
not seen in the first instant are
discriminated
this is a solution discriminating other
solutions. The existing systems work
typically with the same activator to enter
and to erase a choice there is no reason to
change this. After the choice is cancelled
the voter will be informed and can reenter
the choice if desired
unnecessary error messages will hold up the
voting process
Spoiling ballots needs to be clarified.




Ensure deleted data is not counted.


Can be implemented through voter verification




Spoiling ballots needs to be clarified. (This item appeared in
an earlier comment set for Chapter 5 as mercuri-D049.)




this can not be mandated. For counties in
Pennsylvania where a vote for 20 is common
the DRE will not comply.
the use of the word data is incorrect.
ref: an error message should be presented.
This is not standard practice
In "eliminate the accidental actuation",
remove "the".
This section should be clarified that it
pertains to "Voter" Input/Control Devices
and Feedback. Such definition should not be
misconstrued to pertain to other users such
as pollworkers, warehouse workers, and
election office administrative staff.
For alphabetic keyboard entry (as for write-
ins), is querty or alphabetic sequence of
keys preferred?
speech input is thread to privacy poll
workers can hear and count number of yes and
no
I've never seen equipment that it was
impossible to accidentally actuate. Design
should minimize possibility.


Use of acronym "SPL" may not be correct.

A height and width of 1.5 cm and separation
of 1.9 cm between touch screen areas is too
small and too close together for my fingers.
Also, such a small area and distance makes
it a difficult target for elderly who are
shaky.
Return to home position should be obvious to
voter.


labels on button or keys, s/b braille for
blind voter and no need for labels if that's
only purpose
the inherent issue of averaging (when 2
points are selected on a touch screen and
the driver averaged the distance between the
touches and activates the nearest area)
needs to be addressed in this item.

the width of the minimum separation is too
large. If this is mandated than the number
of ballot pages will increase and have an
undue affect on voters especially ones with
short term memory issues.
The last sentence of paragraph 1 states:
“Where there are time limits imposed on the
total time spent voting, the system shall
indicate the amount of time the voter has
remaining to complete the voting process.”

We have no specific recommendation here but
wonder if this is a good idea. Voters may
spend more time in a voting booth due to
extenuating circumstances. This may best be
controlled by poll workers.

time limits are state rules whether or not
the voter should be informed on the time
left has no place in the federal
specification
Many states have a time limit for voting.




If there is a maximum time to vote, it does
not necessarily need to be solved with an
integrated timer in the voting system.
Such a feature may be useful, but could and
should be left to the vendor and customer.
For example, a separate "egg-timer"
alongside the terminal could be a simple
solution.
Must the time indicator be displayed to the
voter at all times within the interface?
Must the task status be displayed on every
page?
giving the voter the option to cast a blank
ballot is not always applicable and is
defined by state law.
Type ahead capability must not be provided,
not an option.



The text indentifies "feedback (within 0.1
seconds.." yet the next section discusses
more than 1 second. I expect that this is a
typo. Also move the parenthesis since it
misleads reader into thinking the time is
mandatory yet following sections indicate
accommodation for longer times.

Paragraph 2 states:
“The system shall provide feedback (within
0.1 second, but preferably less) in response
to user
actions.”

While well intentioned, this paragraph can
lead to deleterious operation. For example,
voters who have a tremor may touch an active
area more than once within 0.1 seconds. If
the voter trembles, the selection may be
selected then deselected and the voter may
not understand why or the voter may not
realize that this happened. This problem
can be solved by “click debouncing.” Doing
so means that response should not occur and
feedback should not be given so quickly.

Section 12.7 of the FEC 2002 VSS provides
guidelines that anticipate both of these
issues. We think that the VSS has been well
thought-out on this issue.

button can be used to enter and delete a
choice time must be longer to prevend
bouncing
too prescriptive of a solution



redundant to 5.3.7-3
Correct typographical error.
Paragraph 2 has two problems. As proposed,
this paragraph states:
“The system shall check user inputs for
acceptability, e.g., check for inputs that
seem to be in error
(such as putting a Arabic number in a name
field) and alert voters to the error.”

A write-in candidate with the name of John
Doe 3rd will have a serious problem with
this requirement. Furthermore, paper based
systems that allow voters to write-in in
longhand will be unduly burdened to meet
this requirement.

Various state laws forbit accepting overvote



reversible too narrow

Choosing language on most systems is
currently not reversible, without spoiling
the ballot and reactivating the machine.
are all actions defined as reversible by the
voter? OR just reversible in general?


Paragraph 6 specifies colors. We think that
red should also be allowed for warnings.
Help and instructions are ususally provided
on the opening screen and also printed
within the voting booth itself. Providing
additional context sensitive help within the
voting device screen themselves will add to
the confusion. These are not PC's!
Facilities must be secure.




COTS equipment will be entrusted with
counting votes but is exempted from this
standard with a "proven record of
performance"? OEMs of voting eqipment also
have "proven" track records but must still
test to this standard? This seems
unreasonable.
Security of storage areas and containers not
addressed.




All voting systems shall meet the
requirements for safety of IEC 60950-1.

In cases where an entire third party
specification is mentioned, there is a
strong risk that the specifications conflict
with each other. It is best to state
directly what the requirements are.

Incorrect use of abbreviations.


Third item not correctly labeled. Also add
that there is no need for backup power to
light the voting system either.
Length of time to retain contents of memory.



 Insert the drawing from fec 2002 where the
wheelchair voter approaches the voting
station paralllel


 Insert the drawing from fec 2002 where the
wheelchair voter approaches the voting
station paralllel
Typographical error on operating
temperature. 441° to 104° Fahrenheit should
be 44° to 104° Fahrenheit

The proposed language is:
“All voting equipment shall be capable of
operation in temperatures ranging from +5°C
to +40°C degrees
(441° [sic] to 104° Fahrenheit) and relative
humidity from 5% to 85%, non-condensing.

This is a change from the VSS Operating
Environment. Many vendors have designed
their systems to meet VSS rules. This
change places an undue burden on those
vendors. In any event, 441° is a typo.
Same situation as with the temp range only
that in the old standard no humidity range
was called out. Beyond that the range is set
at 5-85% RH non-condensing. One our our CF
manf.'s makes some of the most rugged
industrial CF's on the market and they only
spec a range of 8-85% RH non-condensing.
This seems a little unreasonable.
Temp range went from 50F-95F to 41F-104F. We
already meet 2002 standards, are we to be
required to re-test all environmentals but
the temp range increased by eighteen degrees
total? This hardly seems reasonable for such
a small change.
The proposed language quotes MIL STD 810D.
We have heard, but do not vouch for the
accuracy, that these MIL Standards have
either been superseded or are difficult to
acquire. Since these standards affect both
design and certification testing, they
should be directly quoted here.
Incorporating a requirement by reference
creates risk that the incorporated
requirement conflicts with the specifically
enumerated requirement.
Calls for testing physical shock but no
standards for shock are specified.


Except for a footnote about electrostatic
discharge there are no standards about
altitude or air shipping requirements, e.g.,
must the equipment be air shipped in a
pressurized aircraft, must the shipping
container be vented or not, what altitude
must the equipment be certified to work at
(a lot of electronic equipment starts having
problems above 3 km or 10,000 ft above sea
level, a common elevation in Colorado)

This requirement is at odds with FEC 2002
VSS Section 2.3.1.3.1 Common Standards. As
we understand it, that section states that
no logo be allowed on any product in the
polling place.
Nothing about workmanship that introduces
hazards when equipment is in use.



item "e" is missing

This is a change from 7.5% in the old
standard. Again, we will be required to re-
certify equipment that already meets 2002
standards?
Incomplete sentence.
I have no idea what is meant by these two
transient requirements.
Given the distances between lines specified
0.5 kV lightning surges seems an extremely
small requirement.
Incomplete sentence.

Incomplete sentence.

Incomplete sentence.

On page 37 under DRE Systems Standards, the same
considerations would be applied to provisional ballots.
Footnote 26 needs to be superscripted.
Incomplete sentence.




"air discharge26 and…"
26 is low case
Footnote is incomplete.




I really question leaving the option to
raise the air discharge to + 25kV to a
jurisdiction, are they really qualified to
make this type of decision?
What about preserving votes but not normal
operation
The draft states: Equipment covered by this
standard shall comply with the Rules and
Regulations of the Federal Communications
Commission, Part 15, Class B or the CISPR
22, Class B requirements for both radiated
and conducted emissions.

We note this is the same requirement quoted
in the FEC 2002 VSS. Most touch screens
that are commercially available meet Class
A, not Class B, requirements. We recommend
relaxing this requirement. We see no
compelling reason to force Class B.



Should be numbered a) and b)
 f) and g) wrong
The meaning of these standards escapes me.

100 KHz is wrong acc. IEC 61000-4-6



20 is wrong
The draft uses a unit of measure for
magnetic fields that it has not defined.
While the reader can probably understand m
for meter, it is not clear what the unit A
for magnetic field is.
As agreed upon in the past with Steven
Berger, I am rewriting this section, and so
am not commenting on every detail that
methinks should be changed, but only on a
few sundry points.
This section requires rewriting from stem to
stern.

This section does not address stack
overflow.




This section does not address runtime
exceptions.

Unmodified COTS must be evaluated at the
source code level to protect against the
threats identified in 5.3.2.1 (A).




The section is written assuming a single-
threaded application and, as such, prohibits
interrupt service routines as well as multi-
tasking and also does not address the
serious consequences that can arise from the
illicit interaction of the threads.


This section is written assuming a single-
threaded application which is assumed
elsewhere throughout the document.
This section is written assuming a single-
threaded application and, therefore, does
not address the serious problems that can
arise from sharing of data by threads.
This section is written assuming a single-
threaded application and, therefore, does
not address the serious problems well-known
in real-time multi-tasking systems.


“accepted and proven industry standard
software design methods and tools” should
also reference appropriate IEEE standards.




The definition of “firmware” should also
refer to its being fixed and irremovable
without opening the machine; since self-
modifying code is explicitly excluded, there
is otherwise no meaningful reason to
distinguish firmware from other software.
“Industry standard” is not defined.


Concerns addressing use of COTS products need to be
added.




"source code generated by COTS code
development package and embedded in software
modules for compilation or interpretation
shall be provided in human readable form"
Some newer programming tools do not
necessary generate traditional source code
as reference within this clause.

“Unmodified third-party software is not
subject to code examination” is very risky
and ” is contrary to such other mission-
critical methodologies as those used by the
FDA and FAA, and contradicts what is
specified in section 5.1.3.3.2.
“formal tests” needs to be explained
“Unmodified third-party software is not
subject to code examination” is contrary to
such other mission-critical methodologies as
those used by the FDA and FAA, and
contradicts what is specified in section
5.1.3.3.2
“The vendors shall submit … a record of all
user selections made during software
installation … [and] a record of all
configuration changes made to the software
following its installation” implies that the
code is variant, excluding the possibility
of V&V having been performed on the entire
software image.
Exclude db scripts from coding
considerations
Besides it is not realistic to translate the program language
source code to be plain English.
Change 'the standards' to 'this standard' or
in some cases to 'this standard'.
Add space in the last paragraph between
"5.1of".
Add comments indicating purposes and
instructions/intent related to source review




“examination... to verify that the code is
unmodified and that the default
configuration options have not been changed”
should be expanded to validate all legal
configurations.
Also provide for separate information from individual voting
machines, especially useful in case of malfunction.

Add item


“The software used by voting systems is
selected by the vendor” appears to mean
“COTS is selected”; else, it contradicts the
subsequent sentence. Change the opening
words from “The software” to “The COTS
software”.
 “operating system software may be designed
in assembly language” makes no sense.
Why is the writing of the operating system
in this manual? Aren't there already IEEE
Standards for Coimputer operating systems.




This section, which requires the use of
high-level for logical operations thus
precluding the use of assembly language, was
ported from the FEC 2002 standards. This
restriction was not included in the FEC 1990
standards. It assumes that good
maintainable code that meets all the other
standards requirements cannot be implemented
in assembly language. I don't believe this
is true and I've been told that the reviewer
of code for Wyle indicates that he's
reviewed prefectly good code that meets all
requirements implemented in assembly
language.
These are not requirements; they can not be
tested to.
Industry standard COTS compiler and runtime
interpreter both is not defined and assumes
that, contrary to reality, something is fail-
safe and fool-proof by virtue of being in
common use.
“to prevent accidental or deliberate
attempts to replace or modify executable
code” can be accomplished in a fool-proof
manner by disallowing writes to memory
containing the code.
The draft states: “Self-modifying, remotely
or dynamically loaded code” We do not
believe that any vendor can comply with the
section in boldface. Almost all operating
systems use some form of demand paging. We
believe that demand paging could be
construed as a form of dynamically loaded
code. Demand paging is transparent to, and
out of the control of, system designers.
“Dynamic memory” is rarely or never used in
real-time or mission critical systems
because it is indeterminate, that is, its
use can lead to a crash at unpredictable
times under circumstances that may or may
not have been tested for.
Insufficient features in “Where the
development environment … includes the
following features”


The control that the software provides
should apply not only where “Where the
development environment … includes the
following features”
“imbedded” should be “embedded”
“Library modules…” makes no sense in the
most common use of the term “library” as a
precompiled and partially linked collection
of modules; it is not useful or generally
possible to add a header to an object file.
Numerous requirements are merely suggestions
that do not conform to current industrial
practice.
Are these header comments in the coding for
whom?
All modules of a program may not be able to
stand alone. Calculations or table builds
may be performed prior to a module being
called.
“COTS software is not required to be
inspected…” is contrary to such other
mission-critical methodologies as those used
by the FDA and FAA, and contradicts what is
specified in section 5.1.3.3.2
There is implied a lack of testing in “COTS
products require updates due to a detected
security breach or vulnerability”; nothing
that requires an update should pass testing.
There is implied a lack of testing in “the
most recent version of the COTS product
incorporating all security patches” ”;
nothing that requires an update should pass
testing.




Requirements and suggestions are intermixed.



"…COTS software …must be the most recent
verion of the COTS product …"     The most
recent version is not always stable enough
to deploy and may not be compatible with the
other aspects of the application.   The
vendor must have the latitude to employ the
COTS versions and upgrades at the
appropriate time.
Why specify that COTS software must be
designed in a modular or object oriented
fashion and not inspect it for compliance?

COTS must meet the requirements of 5.1.3.1


COTS virus detection programs are not
available for all operating systems.




COTS evaluated should include compilers,
libraries, and any other software tools used
in system development and capable of
introducing backdoors or other malicious
code.
This section of the draft has this language:
“However, COTS software is not required to
be inspected for compliance with this
requirement but must be the most recent
version of the COTS product incorporating
all security patches,” [emphasis added]
This section may be ambiguous.    Must the
latest version always be incorporated or
only the latest version of security patches?
What if the security patch is not relevant
to the particular operation.

In any case, forcing the latest version of
COTS software is a configuration control
nightmare and will result in endless re-
qualification. One interpretation of this
section is that software written to run on
Windows 2000 must be rewritten and re-
qualified to run on Windows XP even if it
runs perfectly well on Win2000. An even
worse interpretation requires vendors to
update hard disk controllers with new
firmware and drivers every time a new
software version is available. We don’t
think this is intended nor desirable.

Note the term “Module”. The term Module is
used here as it is used in the FEC VSS and
we believe this usage to be non-standard. A
module should be a collection of related
subroutines and functions. A module may
“Headers are optional for modules of fewer
than ten executable lines” is inconsistent
with common practice in mission-critical
software and, for example, violate FAA
rules (at least as of 1993) where all
modules must contain 20 or fewer lines of
code, have a full header, and no more than 1
module is allowed in a source file.
Additions as described.




What does "No other constructs be used to
control program logic and execution" mean?
This specification prohibits the GoTo
construct; a good idea. However, some
programming languages require GoTo for so-
called ON conditions. For example consider
Visual Basic (a language expressly allowed):

On Error GoTo ErrorHander

Other languages may have other “On”
conditions:

On Timer GoTo TimeoutRoutine



Should emphasize that only the structured
programming method, meaning controlsof
Sequential, Conditional and Loop, can be
accepted. Instead of using the specific
command IF-THEN-ELSE, DO-WHILE, CASE
Why should names differ by more than a
single character? What does this do for the
logic? If there are multiple porgrams that
have the same general function, but
reference different tables, changing the
name would make more sense and relate the
programs.
“so that its executable lines can be clearly
understood” implies that the rest of the
lines do not need to be understood
5.6.3 On page 68 under Data and Document Retention, the
Board would need the ability to enter data into the third linked
field (AF) of every stored provisional ballot in order to
tabulate the valid provisional ballots. However, the first
linked field (identification number) would be read-only
information and the second field (provisional voter CVR)
Specifies reporting errors to election
would be protected from being altered, read or printed.
officials and voters but no requirement to
put these messages in audit log.



What is an "easily understood language" is
undefined.




eliminate or printed - may not be for
performance reasons
Status messages for the voter must be in the
language selected by the voter.




Messages need to be stored in audit log as
well as memory. Also, it may not be
necessary to store them in memory if they
are stored in the audit log.

No location is "indestructible" Suggest use
of "permanent."
On page 69 under Functions, the provisional ballot
information would also need to be stored in an indestructible
location. However, the votes would not be tabulated and the
information would be stored in the previously mentioned
“of the 50 states” format.
three-linked-memory-field excludes US Territories
or the possibility pf election laws being
more local (as is the case in many places
where local elections use proportional
representation or ranked-order voting).

Please add Cumulative Voting.
Additions as described.




On page 70 under Voting Variations, provisional
ballots are listed as a variation that may not be
supported by a voting system due to the different
laws in the 50 states.
5.6.5.2 On page 69 under Variations, this document should
address all variations under this section, especially
provisional ballots, in order to have a working solution to
accommodate the different laws in the 50 states.
5.6.6 On page 70 under Ballot Counter, each voting unit
would need to have their own ballot counter for their
redundant storage of the CVRs, and there would be one
master ballot counter for the device that consolidates the
information from all voting units. Separate counters following
this same logic would be provided for provisional ballots and
The ballot and storage of this information.
the processing counter need only be recorded at
commencement of an election; it need not be
0. It may be a good security feature if it
is never reset so that both testing and
tampering are recorded.
a system is registering and not tabulating




a system is registering and not tabulating

Programs must be certified before being
installed.

how does fw match ballot styles
automatically to poll place intended?

Correct English. Ensure rather than assure.




These sub-sub-subsections should be numbered
g) and h) to be consistent with 5.6.7.2.1
Common Standards
Elements need to be certified as well as
proven.



For security and to detect tampering, a
checksum of the code’s image must be emitted
and validated.



consolidated reports within a poll place but
for a specific equipment type - can have
mixed equipment
5.6.8 On page 72 under Voting Functions, this section would
include the previously mentioned voting functions for
provisional ballots.
Wording leaves something to be desired.


Standard should not include any reference to
punch cards
Need to log all attempted actuations,
authorized or not, successful or not.


On page 74 under Activating the Ballot (DRE Systems),
provisional ballots would be added to the ballot type list.
machine may not automatically prevent ballot
not entitled, some of it is procedural

can be automatic or procedural
This subsection seems to duplicate information that is
discussed elsewhere. This lengthens the standard and risks
inconsistency.
Incomplete sentence. Doesn't specify who
should get the message if a jam.




hava does not allow punch card
Standard should not include any reference to
punch cards
voting secrecy is abused
Add "appropriately"




No provision for paper verified ballots.




Missing right parentheses.




5.6.8.3.3 On page 75 under DRE Systems Standards, this
section would incorporate previously outlined provisional
voting standards.
ballot position registers are for mechanical
machienes not supported by HAVA
Sentence structure and punctuation.




Applies only if results transmitted to
another facility.

On page 76 under Post-Voting Functions, processing the
provisional ballots would be listed under this section.
can be automatic or procedural
The requirement of Voter Verifiable Paper
Ballots is missing.
Applies only to DRE machines.



On page 77 under Producing Reports, the only Election Day
report that would be generated for provisional ballots would
be the total number of ballots cast. Following the post-
election verification process, provisional vote totals would be
generated as well as a report of all provisional voter
identification numbers along with the corresponding
information entered in the AF.
Expert-based analytic evaluation should
apply as well to information security


The quality of some figures in the draft is poor. For example,
Figure C-1 in subclause 6.2.2.1. Many of the other figures in
Clause 6 are also not very good. They will need to be fixed prior to
submitting the draft for approval by the Standards Board. Failure to
provide publishable figures may result in a disapproval or
conditional approval by the Standards Board and will delay
The standard needs to include a design review and source
code review.
The last sentence of this paragraph would seem to apply to
all systems, not just systems that use public
telecommications networks to transmit official results.
Why is this restricted to threats as of the time the system
was submitted for qualification? Serious new threats are
often exploited almost immediately upon being discovered.
What attacks should be simulated?
Missing requirement (cf. section 5.1.3.6).




However, this does not relieve the testing
authority from addressing the threats
identified in 5.1.2.3 (A-1) and (A-3)
The requirement of 5.1.3.1 needs to be
evaluated by expert analysis
If the system does not include a voter-
verifiable paper audit trail, the test
procedure needs to include evaluation of the
suitability of the system to operate without
such a function.




6.1 states "public" telecommunications
networks, but 6.1.2 doesn't make this
distinction. Only public networks require
these measures.
Protection during storage between elections
also needs to be included here.


It is not possible, in most cases, to
distribute or install new system releases in
response to threats, because certification
is usually required.
Both sentences use the phrase "be
reexamination." The sentence doesn't read
properly.
Grammatical errors: "All subsequent
changes…shall be reexamination. All
changes…shall also be reexamination." Is
this a typo? What's the intended meaning?
Any changes should require reexamination.




We can be more specific about the design information for
custom chips.



EAL-2 is insufficient for the needed
protection
The “generic PP” has no obvious purpose.
P1583 is the draft standard for evaluating
voting systems. The evaluation process does
not provide an opportunity for some entity
to take a “generic PP” defined in this
standard and develop a “specific PP” to be
responded to by a machine vendor.
Design Review Board




Accuracy is not the same as integrity. The omission of any
material in this section on Data Integrity is a serious detail
that needs to be added.

Add additional material to clarify testing procedures




It is important to clarify the fact that automated testing is not
the same as user testing.




Needs reference for SPRT and is this Wald's
SPRT?




The parameters "a" and "b" have not been
defined.

Talking about Mean Time Between Failures
(MTBF) in this paragraph. Don't know what
MBTF is?
Lacks reference for probability calculated
using an exponential distribution and
sentence is incorrectly worded.
Alert must be provided for unavailability.




Vague
An extra "3" is in the numbering. For
example 6.3.3.5 refers to the specification
in 5.3.5.

Add Recountably



Need section to evaluate whether password
protection has been implemented and common
words cannot be used, e.g., password,
secret, dictionary words.
Need to evaluate whether roles have been
implemented and default superuser accounts,
e.g. root, sysadmin, sa, ora, etc. have been
disabled.
I cannot see how these items can be
validated without first inspecting them.
Entire subsection needs to be reviewed for correctness.

Public confidence is dependent on access to
the bases for why testing standards are, or
are not used.Therefore it is essential to
state specifically that such reasoning is to
be in the public record.
if methodology used should address the ANSI
(NNSI NCITS 354-2001) common industry Format
for Usability reports the document should be
included in this standards document for
comment.
it would be helpful for the testing types
had accompanying definitions
I can see no reason in most cases why an
inspection should not be required by the
procedures in these tables. Also, in most
cases a usability test should be required as
a check on what may simply be someone's
opinion. When something like luminance and
contrast, or voter privacy are to be
evaluated then an expert opinion is required
as those are far beyond what most
technicians can evaluate. Thus, these tables
need extensive additions to be satisfactory
(see following comments).Also, it would be
less confusing if the tables were labeled as
such and not as sub-sub-subsections.

Test support for new paragraph of 5.3.10.2. Propose to test
by "I" (inspection) because the requirement relates to the
logic and data path from which the data is presented, so
verification requires inspection of source code but not
expertise in human factors. (This item is the same as the one
in David Aragon's comment set.)
Expert evaluation is required for many of
these.
Voter privacy must be tested.

Usability tests must be performed to ensure
the requisite functionality.
The meaning of this sentence isn't clear.



Accessibilty for disabled voters should be
subjected to an inspection as a first step.
The use and need, or lack thereof for
personal assistive technology will require
an expert evaluation.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
usability testing are left to determine
compliance, I have concerns.
I am at a loss to understand how one would
know whether the accessibility features for
voters with limited or no vision are present
without an inspection of the equipment.

One can't know whether the large text,
audio, or Braille are useful or usable
without some form of usability testing.
Whether or not wireless coupling for hearing
aids maintains voter privacy and avoids EM
interference will require expert evaluation.

Wireless coupling for hearing aids requires
usability testing.
Whether or not the voting system is usable
by voters with no or limited speech and
other probable impairments requires and
expert evaluation as well as usability
testing.
Whether the voting system is accessible to
voters with limited movement and
coordination must be subject to an
inspection.
I really can't see how the accessibility of
the system to voters with limited movement
and coordination can be evaluated without
usability testing.
Whether or not wheelchair users can use the
voting system in the same orientation
requires some usability testing.
Both testing and usability testing are
required of the means used to activate the
ballot.
Pull the introductory sentence out of the
first cell in the table.
Footnote is numbered one. Should be no. 28.
I still dissagree with this requirement. It
is unacceptable to provide the voter the
means to adjust color or contrast on a color
DRE. It will only add to the time to vote,
provide the opportunity for malicious use by
the voter and cost and complexity to the
device. It also misinterprets the intent of
Section 508, which has this as requirement
only if the device provides for it.

missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
Adequate luminance and contrast require an
expert evaluation.
Usability testing is essential for such
features as graphics, color, luminace and
contrast, flicker, flashing, etc.
ref: the use of the word One at the
beginning of the sentence. Should this not
be "The color ballot presentation…"
It's stated in 6.3.3.3-1 footnote 28 that
the contrast ratio is 6:1 or greater. This
contradicts the statement in section 5.3.3-7
which is the contrast ratio minimum is 3 - 1
but the preferred is 7- 1.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
All the evaluation methods require an
inspection, if nothing else.
There is nothing intuitive about intuition
and only an expert can evaluate such
interactions. The multiple rules for
undervoting also require an expert
evaluation of the system capabilities. And
are we to trust a simple test to confirm
votes are correctly registered and voting is
complete? I think not!
Certainly such features as intuitive
interactions require testing. Anyone who is
married knows that what is conspicuous and
obvious to the wife is often obscure to the
husband. So that feature must be tested as
well. And feedback is oftern overlooked,
ignored, or misunderstood. So again testing
is required.
Now certainly we need to insist on testing
the usability of such features as functional
relationships, feedback, undervoting, and
vote review. Otherwise we are simply
trusting to luck that such features work for
the voter. And what works for me certainly
won't work for many others.

Items 11, 12, and 13 may or may not apply to
a given voting system. Put in disclaimer "if
applicable."
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
Without an inspection how does one ensure
controls are conveniently located, that
users get feedback, that the knobs and
switches are suitable, etc.?
Dexterity required to operate controls and
ruggedness are matters for expert
determination.
Whether controls and input devices are
conveniently located, whether the controls
are familiar to most users, whether inputs
can be triggered accidentally or
unintentionally, and whether the system is
sufficiently rugged to withstand momma's
babe in arms while she votes can only be
determined through testing.
Usability testing is essential for all items
in this list, not just the last five.
Again, an inspection is a first, essential
step to evaluating navigation and system
interaction.
Testing of the distinctions between
navigation controls appears necessary.
Whether any of these functions works for
even half the voters requires usability
testing of all listed functions.
Sentence structure and voter, not user.

Voter is who needs to be alerted.

A simple, straightforward inspection of the
system for the usability requirements listed
would save much time and effort.
Whether the voting system adequately alerts
a voter to problems and resolutions is a
matter for an expert evaluation.
This section is titled evaluation methods
for usability requirements of system
response time and feedback but not usability
testing is required. Seems a pretty basic
requirement here.
missing checkmark in testing column. The
testing authority might have additional
items on procedural items
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
If minimizing errors is the objective then
the system must be inspected for such
potential errors.
Minimizing errors requires usability
testing. What works for one often generates
errors by another. Let us keep Murphy's Laws
firmly in mind.
This is exactly the same title as for
section 6.3.3.8.


I disagree with the testing types identified
for this item. An expert does not need to
validate that this item is available. This
should be the role of the test and
inspection
I disagree with the testing types identified
for this item. An expert does not need to
validate that this item is available. This
should be the role of the test and
inspection
I disagree with the testing types identified
for this item. An expert does not need to
validate that this item is available. This
should be the role of the test and
inspection
A simple, straightforward inspection of the
system for preventing and minimizing errors
would be a first step in evaluating the
equipment.
How these functions can be adequately
evaluated without a usability test escapes
me. Seems a basic requirement.
Incomplete sentence.




This is very vague.
Current (2002) standard call out a lab test
range of 68F-75F but this document makes no
clear definition of what "standard',
"ambient" and "nominal" are or should be.
See comment 370 from v. 4.3. Wording was
changed at my request but I think the word
"encouraged" goes a bit too far. Recommend
the use of test fixtures be "permitted"
rather than "encouraged."
Pristine data sets do not reflect real voting situations. Add
paragraph at end.




See comment 371 from v. 4.3. The revision of
the first paragraph help but I think it
makes the last three paragraphs of this
section even more superfluous and an attempt
at micromanagement. My recommendation, as
before, is to delete these three paragraphs.

Sentence includes physical shock tests but
there are no specifications for shock except
a bench test. See comment 254 on v. 4.3 and
response. Issue is still unresolved. It is
still my recommendation that shock and drop
standards be incorporated. These are common
causes of failure and we can certainly
expect voting equipment to receive rough
handling.
Systems that are simply cobbled together
(kluge might be a better description) from
COTS components must not be exempted from
environmental testing. I've had too many
problems with little doohickies hung on some
piece of otherwise great equipment that
caused problems when fielded.
COTS hardware must have been tested to the
rigor required of non-COTS components; if
the supplier has not done this, then COTS
hardware must be treated like any other
component.


The mere idea of a voting system running on
an inherently vulnerable operating system
such as Windows that need not be evaluated
because it is already is certified at some
EAL-x level is absurd. We are also about to
have ATMs running on Windows software, which
will open up huge cans of worms. I am
STRONGLY OPPOSED to multipurpose systems.
Long ago someone proposed using ATMs on
election day, because people are familiar
with them. This is a HORRENDOUS IDEA. Of
course, I am also strongly opposed to voting
systems with essentially ZERO accountability
that your vote is correctly recorded and
counted, as is the case with the existing
certified DREs.
Why exempt COTS hardware from environmental
testing?

Make the last sentence a separate paragraph.




How many times can the system be retested
before the system is deemed to have failed
the environmental tests? I would suggest a
total of three failures during the complete
process with two consecutive failures
sufficient to reject the voting system.
Reference is made to Figures 514.3-2 adn
514.3-3 but couldn't find them.

Use metric units consistently. Degrees
Celsius not Fahrenheit. If English units are
used they should follow SI units in
parentheses.
Units problem again. Degrees C rather than
degrees F.
Stabilization required for minimum of 4
hours, not an exact period.
See comment for #2.
Much of the verbiage in this section is
redundant to wording in section 5.6.
As agreed upon in the past with Steven
Berger, I am rewriting this section, and so
am not commenting on every detail that
methinks should be changed, but only on a
few sundry points.
This section should be completely rewritten
to more exacting standards. Vince Lipsion
has offered to do that and I will defer
detailed comments and acceptance until I see
his version. Some minor comments follow on
existing version.
What is human readable form? Why is it
required?

The paragraph does not reflect current computer science
theory regarding inability of confirm proper functionality via
source code review. Clarification needs to be added. Citation
to Ken Thompson 1984 CACM "Reflections on Trusting
Trust" paper could be added here.




The decision by the FEC to exempt COTS products from
inspection has created a serious security flaw. It should not
be imperative that the IEEE standard continue to reflect this
inappropriate practice. All exemptions for COTS product
review should be removed from this standard.




If COTS hardware or software is in the trusted subset, it must
be treated exactly like software or hardware designed by the
vendor.
It is not possible to find all malicious code or back doors.

“All software components ... shall be tested
… after every update or modification is
completed” allows a loophole or is
ambiguous.
“Unmodified, general purpose COTS non-voting
software ...is not subject to code
examination...is not subject to the full
code review and testing” is contrary to such
other mission-critical methodologies as
those used by the FDA and FAA, and
contradicts what is specified in section
5.1.3.3.2.
Insufficient specification.


Insufficient specification.




Insufficient specification.


COTS software must work in conjunction with
the voting application software. Therefore,
it should be subjected to the same rigor of
testing as the application software.

How do you know that the COTS software has
not been modified?
COTS software should not be exempt from code
inspection.

Insufficient specification.

Add statement regarding review for malicious code.




This kind of thing was popular in the 60's and 70's, but looks
really archaic to me. I think everyone knows the basic
control constructs now. This just unnecessarily lengthens
the document. noting to do with testing and so
These have
belong with their less verbose redundancies
in section 5.6.2.4.
Flowcharts not necessary for persons with
programming knowledge.


Control Constructs should not limited to the
listed programming language commands and
should not be so detailed. Because some
newer high level computer languages use
different commands but do the similar thing.
For example PL/SQL uses FOR lopp instead of
DO-WHILE, JAVA uses SWITCH instead of CASE.

“Unless obvious from the process” is too
subjective to predicate a requirement on.
Put illustration of levels into table so
that it stays together.
These have noting to do with testing and so
belong with their less verbose redundancies
in section 5.6.2.6.
Hoew does PL/SQL,SQL,Oracle Developer fit
into the coding convention?




I don't know what a macro "exit point"
The last sentence "The vendor shall justify
any module lengths…" does not identify who
determines the validity of the
justification.
Warning about “mixed mode operations” should
explicitly note (for the C Programming
Language) consideration of mixing signed
with unsigned types.
Entities specific to the C Programming
Language are freely intermixed with generic
programming conventions; also, most of these
conventions are really best practices, not
requirements.



Line length constraints here can impose splitting of code into
non-functionally appropriate groups. For example, a table
initialization might be longer than 240 lines in length. Add
sentence before last one.




In some cases, it would be better if assertion violations were
logged. The exceptions are when there are too many of
them, or when voter privacy would be compromised.




Make last two sentences seperate paragraphs.
Need to add sentence at beginning of this paragraph,
preceeding "In this situation…"




Who is perfroming the system build for
testing?
Use of "concurrently read" seems to require that both sides
of a ballot be read simultaneously. This is too prescriptive.
This section is essentially the same as
section 6.1.3

This section is essentially the same as
section 6.1.4

Presently the test agency is required to do
the vendor's work for them if the vendor
doesn't want to be bothered.



No mention is made of security during
development. Development phase is a
vulnerable time for security failures.


Need source code to evaluate software
design.
Need to know the developmental history and
testing done by the vendor.

Need clarification on what is required in a
"System Configuration Overview." Does this
refer to the system overview in sec. 7.2? If
so, shouldn't the language be consistent
from one section to the next?
A summary would suffice as well as an
abstract. Give them choice.



What is required in a document abstract? Is
a TDP table of contents sufficient or does
the TDP require a summary of each included
documnt?
Since some states rely soley on the federal certification,
escrow should be required, not recommended.



Technical data package should be public.



Personnel training requirements are
dependent on system configuration and
customer requirements and cannot be
accurately documented.
Breakdown is needed as to requirements per
election jurisdiction and per precinct.




Training levels and backgrounds of people
working on voting systems are critical
factors in both maintenance and security.

Training requirements should be decided by
client jurisdictions rather than vendors.
Orientation and training must be in both
system operation and security.


New versions can only be released to
customers after certification. That needs to
be specified.
Clarification is required on what
constitutes a "discrete system component."
Need to identify customers who need updates.




Sentence structure and completeness.




Versions must be uniquely named for
identification.
indicated not indicate


Again, versions need to be uniquely named.
Sentence structure



"tools" is amibuous

"tools" is amibuous



"tools" is amibuous

Need name and contact information if this is
to be of any value.
Correct sentence structure and wording.


Correct sentence structure and wording.



Provision is made in the standard for update for COTS
products releases, but there is no such provision for updating
or decertifying non-COTS voting system components if such
have been revealed to be insecure.
Should change releases include references to
internal configuration item identifiers?
Need information about supported languages
if applicable, e.g., DRE system.




The chart on page 251 indicates that provisional
voting is covered in HAVA, Sections 302a-d. The
chart also indicates that the 2002 FEC VSS and
this document do not cover the implementation of
provisional ballots.
Better summarize what is being established.




Ambiguous who the "user" is. Think the
administrator or election officials should
be the only ones who can bypass or
deactivate functions.
Again "user" is ambiguous.
Again "user" is ambiguous.




Usually there are test points and test
procedures for such equipment for
troubleshooting.
Software can't run without the hardware, and
different hardware will do different things.




Vendor should not need to put terms in their
glossary that are defined in this Standard.




Sentence structure is awkward and comma
required.




Need a better definition of what constitutes
a "software item." Is this a subsystem (such
as an election reporting program)? Or an
individual software requirement. Also,
subsection 1) requires clarification. What
is meant by "Software requirements performed
by the item?"
Need to know the error codes produced by the
system and what to do for each.

Nothing about test routines and expected
outputs.
I am not sure what HIPOs are in this context
and I can't see that such jargon adds to the
reader's understanding here. Conversely, it
is very likely voting systems will
incorporate databases and it is essential
that the entity relationship diagrams (ERD)
be included in any overview.
Documenting the "response and response time"
for each system input seems excessive.

No mention of included libraries used in the
design decisions.
What information is required in an "entity
relationhsip diagram?"
Databases don't include files. They do
include many other features that must be
described.




only applies to interfaces defined as being
for the purpose of an EDI
Nanoseconds? Spare me. The MKS SI standard
unit for time is a second.



Message encryption is part of formatting.
Incomplete sentence.



The technical data package needs to include
a threat analysis.




Evaluation requires knowledge of extent to
which system depends on “security through
obscurity”




The documentation required in 5.1.3.1 needs
to be identified in the TDP.




It is important to also include vendor-related access to
access control policy.



Repeated word "access".



Vendor can provide suggested procedures and
tools for an effective access control policy
but access control is ultimately the
responsibility of the client jurisdiction.
On page 126 under Access Control Measures, this section
would include the previously mentioned post-election access
that would be necessary to process provisional ballots.
Clarify that policies and procedures need to
remain current as well as effective over
time.

Detailed, as in obfuscation, usually isn't
helpful in technical documents. Add
requirement for clarity. Also, these
activities "are" prohibited not "should be."



Telecommunications and data transmission
security is addressed on a client by client
basis and can only be documented in general
terms.
There are certainly many other types of
attack possible on a voting system than
denial of service. Voting system should
recognize or at least log other types of
attacks.




Detailed, as in obfuscation, usually isn't
helpful in technical documents. Add
requirement for clarity.


Sounds nice but needs a bit of beefing up.




Numbering goes haywire after a. 3)
The present wording puts an unnecessary and
unjustified burden on the ITA. If the vendor
can't supply test data the system should not
go forward.
Ambiguous who the "operator" is. Think what
is meant here is the sysadmin and election
officials, not a voter.


Delivery schedules are determined on a per
customer basis. Schedules are set based on
the needs of the individual customer.
Same ambiguity as 7.8.5.b) as to "operator"
with additional problem that intervening in
the voting system operations must not
introduce any errors into balloting that has
occurred.



I don't understand what is meant by system
"acquisition" in this context. Suggest the
word be deleted.




Upgrades must be tested and they are a great
time to introduce hacks so they must be done
in a secure manner.



Recovery procedures are essential here.




Again ambiguous use of "operator" Also,
security procedures must be enforced by the
election jurisdiction.


Other systems besides DRE may transmit
election data over a network.

a) database functions and preventative
maintenance depend on the equipment
available at a jurisdiction. The tasks
detailed in subsection a) vary from customer
to customer and cannot be documented. B) The
number and skill level required to perform
tasks is dependent on client experience.

Should steps to correct software
deficiencies and configuration management
processes be included in end-user
documentation?
Parts and materials must be identified by
both size and location.
Clarify.
Add facilities for ballot and audit storage retention.




Supplies and requisite maintenance are not
included.



The factor here is distance from the
maintenance people to the voting systems
they may need to service.


The title of section A-1, “A.1 Development
of Voting Equipment”, causes confusion when
looking at the TOC or searching the document
because “Development” is elsewhere used in a
specialized sense for product development.

Correct sentence structure.


Correct usage of 'which" and editorial
comments
Correct wording.




Note that IEEE is an independent body.



Need to be clear that mechanical devices
didn't stop election fraud.



Abstract should be made consistent with
Section 1.1
In taking a look at the draft P1583 abstract, introduction, etc.,
I was struck by the absence of any information concerning
the "voluntary" nature of this (and all our) standard(s). In
particular, I think the draft standard needs some words in the
ABSTRACT and INTRODUCTION that reflect the ANSI
boilerplate: "The use of American National Standards is
completely voluntary; their existence does not in any respect
preclude anyone, whether he has approved the standard or
not, from manufacturing, marketing, purchasing, or using
products, processes, or procedures not conforming to the
standard."

To which I'd like to add something along the lines: "...
standard and is not itself a mandatory or regulatory
requirement."

While I assume the ANSI boilerplate will be included when
Much of the security section is out of scope
by this definition.


Election officials often insert the ballot
into the tabulator rather than the voter.



Other election officials besides the poll
workers usually perform these tasks.




add keywords concerning usability

Sentence structure and wording




Recommendation about candidate order on
ballot should be added.



Remove references to inappropriate entities

I am confused about the purpose of Annex C.
It needs more clarifications on how it is to
be used, either in design or testing.
Title isn't clear as to purpose of this
annex.
First, it is my understanding from Berger
that this is now Annex D and a new Annex C
has been added that I have not seen. Second,
undefined conventions, jargon, and
formatting apparently from the security
community are used throughout that make the
meaning and understanding of this annex
impossible as written. Third, there are so
many flaws in the information presented as
to make the annex worthless as currently
presented.
Choice of EAL2 is inappropriate to the
voting system setting.



No mitigation is provided to known Common
Criteria flaw regarding removal of audit
trail data that includes actions by the
voters (a known threat agent group) in order
not to violate ballot privacy.



Protection against collusionary signaling by
a voter to a vote buyer or vote intimidator
to bypass vote privacy should include
consideration of covert channels for
accomplishing the signaling.



Make changes as indicated.
This assumption is incorrect. Asumptions
regarding controlled access facilities may
not apply during storage between elections
or at numerous other times, such as between
the time of delivery to the polling place
and the opening of the polls.
The assumptions in 5.1.2.3 need to be
reflected in the PP assumptions.
This assumption is incorrect. Malicious
insiders are among the potential threat
agents. The TOE must provide means of
enforcing separation of duties to preclude
the need for trusting a single individual
not to perform a malicious act.
This assumption is incorrect. Assumptions
regarding controlled access facilities may
not apply during storage between elections.
Also, there are numerous other opportunities
for unauthorized physical access to the
voting machines, such as the time between
delivery to the polling place prior to
election day and the arrival of the polling
place election officials on election day.
Even the detailed monitoring of the polling
place by election officials when they are
busy with numerous voters may not satisfy
this assumption.
This assumption is incorrect. Asumptions
regarding controlled access facilities may
not apply during storage between elections
and at other times. The TOE must be capable
of providing some of its own protection

The threat agents identified do not track
the threat agents implied in 5.3.2.1




"Threats To Be Addressed By The Operating
Environment" is not specific about enviroment.
Change title from:

C.3.3.2 Threats To Be Addressed By The
Operating Environment

to:

"C.3.3.2 Threats To Be Addressed By The
Operating Environment (Non-IT Environment)
The threats identified in this section do
not clearly track the threats identified in
5.3.2.1.
This policy statement does not make sense in
the context of a voting system.
In general, administrative procedures should
be such that at least two people participate
in every procedure.
This policy statement does not make sense in
the context of a voting system. TOE
activity can not be monitored during periods
of storage and delivery to polling places
and storage ares. There is nobody present
to hear an audible alarm or see a visual
alarm. A combination of TOE features and
externally applied/inspected seals is used
when the TOE is being stored or delivered.
The requirements must reflect this.

The Voting System Standard is this document,
not some external reference.
This is a requirement of the TOE, not of
organizational policy
Before and after the voting, the general
public has a need and right to know all
information in the voting machine, other
that security protective passwords and keys.
However, only authorized election officials
have a need-to-perform the functions needed
to supply that information to the general
public.
This is a requirement of the TOE, not of
organizational policy
This policy statement does not make sense in
the context of a voting system.



Anonymity must be maintained even if the
voter attempts to collude to violate it
This appears to be a Non-IT Environment objective.
This objective, as written, does not make
sense in the context of a voting system.
The Voting System Standard is this document,
not some external reference.
Before and after the voting, the general
public has a need and right to know all
information in the voting machine, other
that security protective passwords and keys.
This explicitly includes all audit data,
which is part of the public record of the
election.
This security objective does not make sense
in the context of a voting system. TOE
activity can not be monitored during periods
of storage and delivery to polling places
and storage ares. There is nobody present
to hear an audible alarm or see a visual
alarm. A combination of TOE features and
externally applied/inspected seals is used
when the TOE is being stored or delivered.

This security objective does not make sense
in the context of a voting system. The
system is initially delivered to a
warehouse, where after acceptance it remains
in storage until it is needed for an
election. It is then taken from the
warehouse and delivered to a polling place,
where it is set up and used according to
procedures. It is not clearly ever
“installed” anywhere. After the election,
the system is returned to the warehouse
where it is stored until needed for the next
election and delivered to a (likely
different) polling place.
This statement makes improper use of the
term “CVR”. The CVR is the record of cast
votes. Votes should not be cast until they
have been validated by the voter.
Need to add a new objective on non-
traceability of the cast vote to the voter.


This statement does not say anything.
O.INSTALL
Most of these objectives appear to be Non-IT objectives. For
instance, which SFR meets OE.ADMIN_TRAINING?
This is an example of an IT security objective that could
be met by an SFR; FPT_PHP
Before and after the voting, the general
public has a need and right to know all
information in the voting machine, other
that security protective passwords and keys.
However, only authorized election officials
have a need-to-perform the functions needed
to supply that information to the general
public.
Before and after an election, the election
and audit data in the TOE is part of the
public record of the election.
This is an explicitly stated requirement, therefore
Section 1 needs to include the conventions that are used
in the PP; assignments, selection, iteration, refinements,
and explicitly stated requirement formats
Add to table:

FMT_SMF.1 - Specification of Management
Functions
Beside being incomplete (with places having
text to be determined (assignment or
selection), these functional requirements
are based on incomplete or incorrect
assumptions, threats, policies, and
objectives. Accordingly, they need to be
completely rewritten
Component missing from Security Requirements
This ‘refinement’ of the requirement needs to be
explained. See comments diaz - 005
All operations need to be identified; assignments,
selections, refinements. For example, a completed
assignment would be bold text within brackets and a
selection would be underlined text. to comply with
Add the following requirement
International Interpretation RI - #65




Add

Add

Add
Add


misspelled authorized


There International Interpretations that apply to several
SARs, and as such need to be annotated. I have
included a few of them.
Change from:

AGD_ADM.1 Election Official / Administrator
guidance

to:

AGD_ADM.1 Administrator guidance
Change from:

ATE_COV.1 Evidence of coverage
ATE_FUN.1 Functional testing

to:

ATE_COV.1 Evidence of coverage
ATE_FUN.1 Functional testing
ATE_IND.2 Independent testing
Change from:

AVA_SOF.1 Strength of TOE security function
evaluation
AVA_VLA.1 Vendor vulnerability analysis
ATE_IND.2 Independent testing

to:

AVA_SOF.1 Strength of TOE security function
evaluation
AVA_VLA.1 Vendor vulnerability analysis
The justification for using EAL-2, as
reflected in C.7.3 is absolutely inadequate
and incorrect. The fundamental requirement
is to protect the integrity of the voting
process from well-financed, sophisticated,
motivated attackers. There is no
requirement that this be attempted with an
EAL-2 operating system, and in fact it may
not be feasible to provide the needed
protection with such a system. Furthermore,
because the standards is primarily for US
voting systems, there is no need to remain
within the EAL structure prescribed by the
Common Criteria international mutual
recognition agreement. Modifying the
assurance requirements from those in the
standard EAL's would have only the effect of
requiring international users to separately
determine their acceptance of US
certification – a non-issue for this
standard.
Class AVA: Vulnerability assessment
Change to Administator
Class ATE: Tests add ATE_IND.2


Add the following requirement to comply with
International Interpretation RI - #3




Add heading


"The documentation shall describe the steps necessary
for secure setup, ballot generation, and opening and
closing of the polls, start-up of the TOE." does not
adequately describe this funciton.
Modify the following requirement to comply
with International Interpretation RI - #51




This is a major example of where EAL-2 is
inadequate. EAL-2 linits the vendor's
search for vulnerabilities to obvious
vulnerabilities and does not require the
search to be systematic. A minimum level
should be AVA_VLA2 or AVA_VLA3 which require
documentation of the disposition of
identified vulnerabilities and that the
search for vulnerabilities be systematic.
There are objectives, threats, assumptions, and policies
listed in Sections 3 and Sections 4 that have not been
mapped.
This policy is not defined in Section C.3.4


O.ENTITY_AUTHENTICATION objective is not
defined in Section C.3.4.
O.ELECTION objective is not defined in Section
OFFICIAL_AUTHENTICATION - Is this an objective
or a policy? It is not listed in either Section.
O.MESSAGE_AUTHENTICATION - This objective is
not defined in Section C.4.
P.AUTHENTICATED_MESSAGE - This policy is not
in Section C.3.4
P.AGREEMENT - This policy is not in Section
C.3.4
O.INSTALL - This objective is dealing with
the TOE being delivered and installed
correctly, whereas this security requirement
is to ensure the policy identified is
enforced and to restrict the ability to
modify the security attributes of that
policy to the authorized administrator.

O.INSTALL - This requirement is to ensure
only secure values are accepted for the
security attributes (of identified policies)
which does not meet this objective.

O.INSTALL - This requirement is to allow the
authorized administrator the ability to
define alternate values (of the identified
policies) which does not meet this
objective.
P.AUTHORIZED_ELECTION OFFICIALS - This
policy is not in Section C.3.4.
P.AUTHORIZATION_RECORDS    - This policy is
not in Section C.3.4.
P.TERMINATION_RECORD    - This policy is not
in Section C.3.4.
P.ENTITY_AUTHENTICATION - This policy is
not in Section C.3.4.
Assumption Rational - Missing the rationale
to support the mapping
CountersThreats - T.PHI_ACCESS - This is not
a threat in Section C.3.3.3.
Environmental Security Objectives -
OE.AGREEMENT - This environment objective is
not in Section C.4.2.
Environmental Security Objectives -
OE.RELEASE - This environment objective is
not in Section C.4.2.
Environmental Security Objectives -
OE.ELECTION OFFICIAL_TRAINING - This
environment objective is not in Section
C.4.2.
Environmental Security Objectives -
OE.ELECTION OFFICIAL.TERMINATION - This
environment objective is not in Section
C.4.2.
Add as 2nd paragraph:

This PP contains the assurance requirements from the
CC EAL2 assurance package. The EAL chosen is
based on the statement of the security environment
(assumptions, threats and organizational policy) and the
security objectives defined in this ST. The sufficiency
of the EAL chosen (EAL2) is justified based on those
aspects of the environment that have impact upon the
assurance needed in the TOE. The administrative staff
is conscientious, non-hostile and well trained
(A.MANAGE, OE.ADMIN_TRAINING). The TOE is
physically protected (OE.PHYSICAL), and properly
and securely configured (O.INSTALL). Given these
aspects, a TOE based on good commercial development
The high level design isisdescription of the
practices is sufficient. EAL 2 an appropriate level of
TOE, its subsystems, and how the security
functions are implemented. The high level
design does not include the underlying
operating system.

I am not sure I understand this statement or what it is trying to say.
I am not sure this is an appropriate
statement. And I am not sure what the last
sentence means; augmenting the assurance?
Widen table to shorten document by 3 pages.




Only sorted by current VSS 2002. Later,
most will want this sorted by this standard.
This section should also be provided with
the same information sorted by the numbers
in this standard.
The Table in Annex D refers to English units (inch/pound). Units
throughout the draft MUST be converted to metric units. Failure to
do this may cause a disapproval by the Standards Board
Annex D does not include the new items that
were not included in VSS 2002. (The ITA's
need to do this anyway -- why not have
everyone work off the same information and
not have all manufacturers and ITA's do this
separately?)
There are many incorrect cross references

List needs to be compared with FEC VSS to ensure
correctness of correlation.
No increase air discharge



Relink cross references. Currently these
show a cross reference error.
Make font same as rest of document.
Shortens document by 2 pages.
Annex E needs to be labeled as Informative or Normative.
Review the use of shall/should/may/can/will/must throughout the
document to be sure they are used in accordance with IEEE's style.

There are a lot of footnotes in the draft. Please keep in mind that
footnotes to text are informative and not normative.
The figures and tables are numbered incorrectly throughout the
draft. The figures and tables should be numbered according to
Clause. For example, figures and tables in Clause 6 should be
labeled as Figure 6-1, 6-2, Table 6-1, Table 6-2, etc.
Please label all figures and tables throughout the draft. Currently,
some tables are unnumbered.
The IEEE-SA’s style for Annexes is to list Normative Annexes first
followed by Informative Annexes. Currently, Annexes A, B, and D
are labeled as Informative and Annex C is Normative. The order of
these Annexes should be switched around.
While the draft is a huge improvement it
still requires a great deal of refinement
and rewriting. Sections 5.6 and 6.6 are to
be rewritten as well as Annex C. And I am
told that Annex C is now Annex D because
another annex has been added that I haven't
seen. Too many typos and editorial problems
remain to allow this draft to go out as a
final standard.Tables and figures are
missing or need to be redrafted. Units of
measurement are used inconsistently and
incorrectly related to other systems, e.g.,
mm and inches.MKS SI units must be used
consistently throughout with English units
in parentheses if needed or desired. There
are many precision without accuracy errors
that need to be cleaned up. The document
badly needs to be formatted and section
headings cleaned up.
Metric units should be used consistently
throughout. English units given in
parentheses.Presently it is a patchwork.
The standard should prioritize security requirements, with
"accountability" -- the ability to corrupted vote records or
totals -- at the highest priority., along with confidentiality. The
vendor should be required to define the "trusted subset" of
hardware, software, procedures, and personnel, which/who
must perform as specified to preserve accountability and
confidentiality, under the assumption that all other parts of
the system can behave adversarially.. The size of the trusted
subset must be minimized. The trusted subset must be
subjected to very rigorous review, while the aspects of the
system outside of the trusted subset can be reviewed and
The standard does not establish low for the
tested less rigorously. EAL2 is definitely tooan adequate,
testable performance standard for security.
To be trustworthy, voting equipment must
either satisfy extremely rigorous security
requirements including both design and
procedures, or there must be a trustworthy
independent method to verify the election
results.
The draft standard is not ready. Many
comments from committee members have been
dismissed without adequate consideration or
discussion.
The division into "DREs" and "paper based" does not
account for future technologies, such as electronic media
that are used somewhat like paper is now (e.g., CalTech/MIT
The document needs to be proofed for grammar,
"frogs").
punctuation and consistency. There are instances
of missing or incorrect use of words. There is an
inconsistency in the use of identifiers preceding
groups of statements (i.e. letters with periods,
letters with parenthesis, bullets and diamonds).
The bullets and diamonds should be replaced with
letters so that you can make a specific reference
to any given statement in the document. Also,
there is an inconsistency in the use of punctuation
following the statements. As a last general note,
not all the acronyms and abbreviations used in the
FIRST COLUMN – Illustration – C63 member
abbreviation, followed by sequential comment
number
General Principles
Separate the technical from the policy; do not use
technical standards to make policy tradeoffs. Favor
innovation; do not favor entrenched players or
practices. Use existing best practices and expertise; do
not re-invent the wheel or have an exclusionary
Measurement v Acceptability
The best thing a technical standard can do in an
evolving field (and to support evolution of that field) is
provide clear and accepted measurement of systems
along defined criteria so that policy decisions about the
levels of those criteria can be enforced. The standard
should not be judging what is acceptable, but rather
setting technical criteria and measurement techniques to
allow systems to be evaluated. Another reason this is
necessary is that there are tradeoffs among the different
attributes and different price points that should be
available to governments. For example, accessibility
(which cannot be absolute) and price seem necessarily a
tradeoff that cannot be evaluated technically, but rather
is inherently a policy decision. further example is the
Design v Performance
Design standards tell how to build systems,
performance standards give criteria for evaluating
systems. Describing how to build things not only stifles
innovation, but it only provides a barrier to entry, which
compounds the problem of stifling innovation.
Presenting different requirements for different
technologies is more of a design standard than a
performance standard. For example, scan and DRE
should not be differentiated in the standard. A
performance standard should clearly call out a set of
Integrity
Voter confidence in the integrity of the outcome, i.e.
tallied as cast, is of paramount importance. The more
effective and convincing a demonstration of this can be
in future electronic systems, the bigger boost to voter
confidence. The assumptions, threat model, and why the
system achieves integrity are laid out by the submitter.
These are then subjected to long established techniques
in security of white box and black box evaluation. They
also determine the level of integrity achieved. The
techniques and laboratories of the Common Criteria
should be employed. In particular a protection profile
Privacy
 Secret ballot is essential for elections in a democracy.
The privacy component may influence voter turnout.
Possibilities for improper influence damage the
perceived fairness of an election. Protection of privacy
is a security issue and should be subject to the same
types of procedures as integrity.
Usability
The issue of usability can only be evaluated by actual
use. It should include voter training and real voting
experience. Test should be defined that measure the
efficacy of the combined training and voting experience
using standardized ballots and random selection of
Reliability/Safety/Quality
The usual shake and bake, electromagnetic interference,
and quality of manufacturing processes come into play
here. It would appear that these are technology
dependent and that a lot of effort goes into
standardizing them for other types of equipment. This
would be a section that should probably only refer to
other standards and qualification processes. For




The document does not address how absentee
ballots would be counted on Election Day using a
DRE system. Unless the Board counts all
absentees in-house, each precinct would need to
have an absentee card reader that could interface
with the voting system. Since punch card voting is
not longer a valid option, consideration should be
given to an optical scan device that could interface
with a DRE system. It would be beneficial to the
Board if this consideration was covered in the
The most specific area of concern with this document is that
it does not consider how a DRE system would handle
provisional ballots. Voted provisional ballots would need to
be stored on the same memory card, but in a separate
memory file than the regular votes being cast on Election
Day. Each provisional voter's stored ballot would need to be
linked to an identification number. This number would also
appear on the required provisional voting affidavit, which
must be completed before the provisional ballot is cast. The
regular vote totals would be transmitted Election night, but
not the provisional ballot votes.

Following the election, the Board would conduct a verification
process for all provisional voters, using the information
contained on the provisional voting affidavits. The Board
would need the ability to accept or deny each individual
provisional voter's ballot during this process. The Board
would also need the ability to accept a provisional ballot in
part, if the provisional voter is registered but voted in a wrong
precinct containing different districts. The system would
need to be secure so that an individual provisional voter's
ballot could be processed, but not viewed or printed.
Approximately 20 additional comments
provided in draft with inserted comments and
track changes marked.
There are many inconsistencies in the letter
sub paragraph structure (e.g. b. vs b)) and
linkages beween them (e.g. the use of the
word and in some cases and not others).

The following recommendations are for the HAVA software
which were listed in the analysis to improve their
documentation and accuracy that I think are especially
important.
1. 102 - Sections 1,2,3 and appendix C addresses
acceptabe standards for punch cards that would no longer be
valid under HAVA.
2. 301.a.1.a Privacy and independence are not addressed.
301.a.1.b VSS has no guidelines for changing or correcting
absentee ballots.
4. 301.a.2.a HAVA definition of audit may not provide the
accurate information needed to check on voting irregularities.
             Proposed Change



Change “i.e.” to “e.g.”

delete the word additional

Change "will provide" to "provides," i.e.
"This standard [provides]..." in two places.

Revise to read "This standard [provides]
technical specifications for electronic,
mechanical, [integrity,] and human
factors..."
Add reference for MIL-STD-498, Software Test
Plan (STP) and Software Test Description
(STD)
Add reference for MIL-STD-1521.




Add reference for IEEE Std 1063-2001, “IEEE
Standard for Software User Documentation”.



Add reference for IEEE Std   829™-1998, “IEEE
Standard for Software Test   Documentation”.
Add reference for IEEE Std   1063™-2001, “IEEE
Standard for Software User   Documentation”.

Add reference for IEEE Std 1028™-1997, “IEEE
Standard for Software Reviews”.
Add reference for IEEE Std 1471™-2000, “IEEE
Recommended Practice for Architectural
Description of Software Intensive Systems”.

Add reference for IEEE Std 1016™-1998, “IEEE
Recommended Practice for Software Design
Descriptions”.
Add reference for IEEE Std 14143.1™-2000,
“IEEE Adoption of ISO/IEC 14143-1:1998
Information Technology --- Software
Measurement --- Functional Size Measurement -
-- Part 1: Definition of Concepts”.
Add reference for IEEE Std 1061™-1998, “IEEE
Standard for a Software Quality Metrics
Methodology”.
Add reference for IEEE Std 1008™-1987
(R1993), “IEEE Standard for Software Unit
Testing”.
Add reference for IEEE Std 982.1™-1988,
“IEEE Standard Dictionary of Measures to
Produce Reliable Software”.
Complete the sentence/paragraph.
Alternate formats usable by people with
disabilities may include, but are not
limited to, Braille, ASCII text, large
print, recorded audio, and electronic
formats[.] [delete- that]
sound and[;] hence[,] suitable for use as a
statement of requirements for one or more
TOEs that may be evaluated.
An implementation [of an] independent set of
security requirements for a category of IT
products, which meet specific consumer
needs.
Delete sentence. I do not believe that is
appropriate in a reference section defining
COTS.


Spell out NIAP to match the format for other
references.
Add: “RTOS    Real-Time Operating System”

Remove the text in quotes.




There are currently no provisions for the recall of federal
office holders via precinct balloting.




I would drop the last sentence.

Find the original intended definition and
add the missing text.
Change "Non-Partisan Office:" to
"Nonpartisan Office:"
Add "See Ballot Counter."

Add the phrase "….of all voter selections on
a DRE voting unit presented ……."
Make consistent with FEC definition.




Need to add definition for CCEVS. Taken care
in abbreviations.
The data that represents a voter’s choices.
Requirements need not prescribe specific
properties, but every system shall provide
its own formal ballot specification
satisfying the following:

1. The specification shall be sufficiently
detailed to allow all election auditors and
observers to determine, with perfect
consistency, whether some data does, or does
not, constitute a ballot.

2. The specification shall be publicly
available.

A designated individual or group that
desires to scrutinize the election.
The combination of voter verification (cast
as intended) and results verification
(counted as cast) that provides full
confidence that the tally accurately
reflects the electorate’s intent (counted as
intended).
The ballot box that is certified as final by
appropriate authorities. It is not damaging
for the sealed ballot box to contain
illegitimate or invalid ballots (e.g.,
provisional ballots), since these will be
detected and eliminated by the tabulation
rules. Since voters are able to detect and
prove the condition of missing ballots,
policies should be in place that specifies
accountability.
The set of elementary arithmetic and logical
operations that produce a unique tally from
any collection of ballots. Requirements need
not prescribe specific operations, but every
system shall provide its own formal
specification of tabulation rules
satisfying:

1. The specification shall allow all
election auditors and observers with access
to both the collection of ballots and the
tally data to determine, with full
confidence, whether the tally has been
properly formed according to the tabulation
rules.

2. The specification shall be publicly
available.

3. The tabulation rules shall satisfy an
“additive property”: if tallies for two
disjoint collections of ballots are each
created according to the tabulation rules,
the sum of the respective totals shall
always be identical to the total created
according to the tabulation rules for the
ballot box that is the aggregation.
An assignment to each candidate, or possible
voter response, a non-negative total. It may
also contain additional data for the purpose
of election verification.
Change to read: "A [database] or set of
files that contains geographic information
about political subdivisions and
boundaries;"
Specify voting system: "Typically as part of
the PCA, a recognized testing authority also
witnesses the building of the executable
system to ensure that the qualified
executable release is built [on the voting
system being tested] from the tested
components."
A permanent record of a voter's selections that can be
checked for accuracy visually or with the use of a trusted
device, which is preserved in case of a subsequent recount.
Either say "hardware" somewhere or give examples such as
the operating system.
…a removable memory component. It may also print a
paper copy of the CVR.



A device that transfers the vote data on a paper ballot to an
equivalent electronic representation.
Delete last sentence of definition.


… records votes, countes votes, and produces a tabulation
of the vote count using paper ballots.
Delete comma, change "which" to "that"
Eliminate the exemption.




verifiable audit trail
We suggest keeping this important part of
the FEC definition with appropriate
grammatical corrections.
The following references need to be added to
the listing:

Cryptographic Modules

FIPS 140-1: Security Requirements for
Cryptographic Modules, January 4, 1994.

FIPS 140-2: Security Requirements for
Cryptographic Modules, May 25, 2001. Change
Notices 2, 3 and 4: 12/03/2002

Cryptographic Algorithms

FIPS 197: Advanced Encryption Standard
(AES). FIPS 197 specifies the AES algorithm.

FIPS 46-3 and FIPS 81: Data Encryption
Standard (DES) and DES Modes of Operation.
FIPS 46-3 specifies the DES and Triple DES
algorithms.

FIPS 186-2 and FIPS 180-1: Digital Signature
Standard (DSS) and Secure Hash Standard
(SHS), which specify the DSA, RSA, ECDSA,
and SHA-1 algorithms

FIPS 185: Escrowed Encryption Standard
(EES), which specifies the Skipjack
algorithm
Ballot Scanner: A device used to read the data from a
marksense or voter verified ballot.




        change Definition to "Voter Verifiable Audit
        Record: A human-readable record of all of a
        voters selections presented to the voter before
        the vote is cast. Also called Voter Verifiable
        Record."


Add:
HAVA - Help America Vote Act
VSS - Voting System Standard
ANSI (p. 52) - American National Standards
Institute; CISPR - International Special
Committee on Radio Interference; CCTL -
Common Criteria Testing Laboratory; DAC (p.
152) - Discretionary Access Control; ERD -
Entity Relationship Diagram (see comment 163
below); FIPS (p. 150) - Federal Information
Processing Standards; HIPO (p. 122) - [NOTE
that I think this acronymn may have been
misused in 7.5.7.1. If correctly used then
need definition but I would strongly
recommend deletion.]; ICMP (p. 32) -
Internet Control Message Protocol; ISSA -
Information Systems Security Association;
MIL-STD - US Military Standard; NSA -
National Security Agency; SPL (p.47)
[Note:This acronym possibly misused. If
correctly used need definition.]; ST (p.
153) - Security Target; STD (p. 128) -
Software Test Description; STP (p. 128) -
Software Test Plan; TSF (p. 145) [Note: This
acronym may have been misused. If correctly
used need definition.]
Adapt a convention for the common gender
pronoun and use it consistently throughout
the standard.
Change all instances of “state” in the
political sense (as opposed to, say, a
“state machine”) to, perhaps, “election
jurisdiction”.
Delete the second sentence.




Add dates to all URLs.



Change all occurrences of “and/or” to “or”.



Change (or add to) “during system operation”
to “without physically modifying the
device”.



Add a reference in section 2 for the
standard, and use that reference here.
Strike the definition; it is not a special
type of election.


It would be better to describe this as a
voting system, as it seems to do later in
the successive paragraphs.




Change approprated to appropriate
Delete the second sentence of the
definition.

Revise the list of definitions accordingly.
Insert in the appropriate sequence: Voter
Verifiable Paper Audit Record – An audit
record having the following characteristics:
1. The record is viewed by the voter and
capable of being viewed by a recounter or
auditor in plain language on the exact same
medium on which it is stored for purposes of
the audit. Paper is the principal example of
wuch a medium.   2. There is no technology
or functionality between the voter (or
recounter or auditor) and the record
physically capable of altering the content
of the data as viewed by the voter,
recounter or auditor. An example of allowed
technology is pure optical magnification.
An example of disallowed technology is any
form of electronic interpretation of
formatted data, except for persons lacking
sufficient vision to view the record through
magnification.

Change “i.e.” to “e.g.”


This section in the document mentions provisional
ballots and may require change to deal
appropriately with provisional ballots.
Remove the common definition of “software
verification and validation” from both these
and make it a definition in its own right.
Reference IEEE Std. 1012-1998, “IEEE
Standard for Software Verification and
Validation”; perhaps use the definition from
that document. 2
Define "Voter-verified" as a primary term
and, if "Voter-verified" be retained, make
that a synonym for "Voter-verified".




The outline implied by the comment should be
expanded into a few paragraphs of text.
Change to: "Provide for a possible recount
or audit by providing a separate record of
the cast vote, which the voter has the
opportunity to review for correctness."




This clause [is it really a clause?] describes the organization
of a generic voting system.

Make it clear where this section is out of
scope but included for general information.
See comments below.
Rewrite sentence to read: This clause
[provides an overview of] the components
that comprise [an election management
system, vote recording systems covered by
these standards, and ancilliary election
reporting systems.
The diagram is unclear, specify the meaning of the arrows in
a footnote.




Change all sentences specifying something to
“shall”, e.g., in 4.2.1, “The EMS allows the
user” becomes “The EMS shall allow the
user…”.
If you want to describe other voting system
aspects not in scope of the draft for
context, they should be in the Appendix
After initial sentence add: The EMS
described below is not covered by the
current standards. However, an EMS will
normally be required and used to generate
ballots and program the voting equipment
that is defined by these standards. Not
every EMS will necessarily have all features
described in this section.
Change the verb “must” to “shall”, e.g., in
4.2.1, “The environment in which all
databases in the subsystem are maintained
must include all necessary provisions for
security and access control” becomes “The
environment in which all databases in the
subsystem are maintained shall include all
necessary provisions for security and access
control.”




We have no preference, either the Scope
should be expanded or this material should
be described elsewhere.
The Control Subsystem consists of the physical devices and
software [that] accomplish and validate the following operations.
* Recording an image of the ballot cast that identically
reflects the choices made by the voter.

* Allow for the alert of poll watchers and election judges and
officials if a voter believes that the equipment is preventing
them from casting a ballot of their choosing.
Add informational annex with illustrative
examples, as attached
· Protecting the secrecy of the vote such
that the vote may not be observed [or
overheard] during the voter’s selection of
preferences, during the casting of the
ballot, and as the voted ballot is
transmitted for recording on a storage
device, or in the recording of the CVR




This section in the document will require change
to appropriately support provisional ballots.
Change “Signifying…” to "Except in cases
where a paper ballot is deposited in a
ballot box, signifying…"




use correct section

Rewrite as: …tallying the absentee votes by allocating them
back to the voter's precinct, or by creating subtallies within
the voter's precinct. (Note: If this comment is out of
scope, then this entire subsection is out of scope and it
should be deleted.)




Add definition: This means [of
consolidation] must comply with the security
and procedural requirements that apply to
the system as a whole and to the individual
counting devices.
Change sentence to not include the specified
time period, but indicate only that a period
is required. A cross reference to 5.2.2.2
can be added.
Change “means” to “mechanism” in both
sentences.
what is rolled back Some things should not
be rolled back, such as the permanent
counter. Maybe other data such as this
should be allowed not to roll back.
Reword to: “The printed report shall contain
all that is in the device audit log. The
device audit log shall contain all exception
conditions encountered since the earlier of
either the installation of the ballot
configuration (election file) or
commencement of testing for the election to
which the said audit log applies.
Change to: “A voting system proides a means
for obtaining a printed report of the votes
counted on each counting device”




Add the phrase "vendor-independent" into "must provide a
means" -- to read "must provide a vendor-independent
means" in each of these sections.
Strike this requirement. As long as there
are the appropriate paper tapes and
electronic audit trails, the consolidation
is only an efficiency issue, not an
integrity issue.
Change sentence to not include the specified
time period, but indicate only that a period
is required. A cross reference to 5.2.2.2
can be added.
Change to: The printed report shall have
the ability to contain all information
generated by one or more system audit logs.




If you want to describe other voting system
aspects not in scope of the draft for
context, they should be in the Appendix
add "if applicable" to the end of the
sentence.




The audit log shall also contain, for each
of these items, a checksum with strong error
detection properties, e.g., an MD-5 hash
code. The working group should discuss
minimum specifications and, possibly,
prevention of spoofing.
Add sentence: Nor shall the system allow
alteration or modification of the contents
of the alternate file in any manner that
affects election results.




Add : “Such access shall not alter recorded
election data. To ensure this, a checksum
of election data shall be recorded in the
audit log upon closing of the polls and a
timestamp of all changes to the election
data shall be recoded in the audit log.”
Data in any alternate files must be 100% consistent with all
information maintained in the primary file.




Eliminate the bullet assuming that the
concerns are addressed by prohibitions
elsewhere in the standard.

Move “development libraries, device drivers,
operating systems, and monitors” to bullet
3.
* Documentation describing how an update is to be certified
and performed, should there be a declared or discovered
defect in the voting system, software, hardware, or firmware,
or any COTS products used in or in the development of the
system that could compromise its operation as an election
device. should to shall: The voting system
Change
[shall] include the following documentation:

Replace “Diskette, tapes, or compact disks
containing copies” with “Copies on permanent
machine readable media”.
Add a “makefile or build script” after
“utilities”; add “and build” after “read”.

Change to: “CASE tool output or other
documentation of all data and program flows,
interactions of threads, uses of shared
data, and timing analyses of all software
components, in compliance with IEEE Std
1016™-1998, “IEEE Recommended Practice for
Software Design Descriptions”, and IEEE Std
1471™-2000, “IEEE Recommended Practice for
Architectural Description of Software
Intensive Systems”.
Require compliance with section 4.3.5
(“Documentation Requirements”) of IEEE Std
1228-1994, “IEEE Standard for Software
Safety Plans”.
Require compliance with section 4.32
(“Software Documentation and Source
Listings”) of IEEE Std 982.1™-1988, “IEEE
Standard Dictionary of Measures to Produce
Reliable Software”.
Require compliance with sections 2
(“Software Safety Design Analysis”) and 3
(“Software Safety Code Analysis”) of IEEE
Std 1228-1994, “IEEE Standard for Software
Safety Plans”.
After “identification” add “with a checksum
with strong error detection properties,
e.g., an MD-5 hash code”; the working group
should discuss minimum specifications.

After “identification” add “ and a copy on
permanent machine readable media”.


The bullets for “operator’s manual” and
“user manuals” should either be replaced by
or annotated with a reference to IEEE Std
1063-2001 and a requirement that the
documentation comply with IEEE Std 1063-2001
.
Insert a statement in an appropriate place
that “The system shall not automatically
shut down or time out during any official
part of a voting-related procedure, such as
filling out a ballot by a voter, except in
accordance with the requirements of voting
law and procedure.”
Add item e as follows: “ Randomization used
for privacy protection must be based on
random noise or random events and not solely
on pseudo-random algorithms. Such
algorithms are in fact deterministic and
provide repeatable values when started from
the same initial conditions or “seed
parameters”. Repeatable sequence values can
facilitate linking of votes to individual
voters if the sequence of voters using a
machine is known.”
Add a section stating “The operating system,
the language processors, and/or the software
shall be designed to prevent excessive or
improper input from allowing the input of
data or executable code by a malicious
perpetrator.


Start new paragraph: [As] COTS products
require updates due to a detected security
breach or vulnerability [the] voting system
vendor must provide a method to assess the
impact of COTS updates on the voting system,
as well as a method for providing notice and
distribution of updates to purchasers[,
testing facilities, and election officials
and boards]. Where COTS products are known
to be inherently risky ([e.g.,] memory leaks
in the C++ language), vendors must
adequately describe the control methods they
have employed to ensure these risks have
been mitigated.
Eliminate (or move to where it’s in context)
“or by providing information on how to
securely configure a particular IT product
within a system.”

Change vendors” to “COTS vendors” or “voting
equipment vendors”.



Paragraph break with the sentence beginning
"COTS products require updates…"
delete second space before "voting system"


Delete. Until further assessment of
Challenged/Provisional relationship to
voter.
add " or minimized (ex., use bounds checking
sw to analyze and identify memory leaks)"
Add a requirement for a voter verifiable
paper ballot.


Replace sentence with the following:
"Underlying products, such as operating
systems, database systems, firewalls,
network devices, web browsers, smart cards,
biometric devices, general purpose
application components, libraries, and
hardware platforms, that are crucial to the
correct and secure operation of the entire
system must be thoroughly tested. This
includes COTS systems. In addition, there
must be a line by line code review of ALL
software that interacts with the voting
system in any fashion. This is required
because of the potential risk of malicious
code."
All of these activities must be fully documented and retained
for all certified systems.



Third bullet change to:
"To ensure the integrity and independent auditability of the
ballots as cast by the voters."




Notwithstanding the fact that system
certifiers can rely upon the prior
validations of the individual components of
the system [ ] provided they are properly
installed and configured [with the latest
security patches], there must still be an
evaluation of the integrated system to make
certain that security holes have not been
left or created during the integration
process.
Mandate compliance with IEEE Std 1228-1994,
“IEEE Standard for Software Safety Plans”

Change “operating systems” to “RTOS s (Real-
Time Operating Systems)”.
Change “COTS product may” to “COTS products
shall”. Mandate compliance with section
4.3.11 (“Previously developed or purchased
software”) of IEEE Std 1228-1994, “IEEE
Standard for Software Safety Plans”.
Change “IT systems are procured and
constructed to meet specific requirements
and typically use existing” to “The system
may”.


Mandate that testing preclude any security
breach or vulnerability; mandate compliance
with section 4.3.11 (“Previously developed
or purchased software”) of IEEE Std 1228-
1994, “IEEE Standard for Software Safety
Plans”. Mandate COTS be subject to the
specifications of IEEE Std 1008™-1987
(R1993), “IEEE Standard for Software Unit
Testing”. Add reference to IEEE Std 982.1™-
1988, “IEEE Standard Dictionary of Measures
to Produce Reliable Software”.
Bring into conformance with Annex D (“V&V of
reusable software“) of IEEE Std 1012-1998,
“IEEE Standard for Software Verification and
Validation”, e.g., “Reusable software (in
part or whole) includes software from
software libraries, custom software
developed for other applications, legacy
software, or commercial-off-the-shelf (COTS)
software. The V&V tasks of Table 1 are
applied to reusable software just as they
are applied to newly developed software.
However, the inputs for these tasks may not
be available for reusable software, reducing
visibility into the software products and
processes.“
Eliminate “(ex. memory leaks in the C++
language)”

Eliminate “ex. memory leaks”.




More appropriate would be "security
vulnerabilities in Microsoft products".
Develop a set of meaningful requirements.

Strike the words “General Purpose Computing
Equipment”.
Strike the words “Any components developed
by a voting jurisdiction”


Develop a set of meaningful requirements.

add "Certification should be based on both
the technology and procedural aspects
comprising a system as defined in the
vendor's documentation (see Sec 5.1.3.1)."
Develop a set of meaningful requirements.




Strike the bullets “Data communications
security” and “Risk, response and recovery”.


This subsection [?] lists generic threats to which a voting
system may be subject. It is, of course, not possible to
enumerate all threats, but this establishes a lower bound on
the threats that must be defended against.
Find how each threat is addressed and cross-
reference that.
3. The systems may be unattended for periods of time when
they could be at greater risk. 4. The need for anonymity of
voter ballot reduces or entirely removes many traditional
forms of auditing commonly used for other electronic
systems (such as ATMs in banks). to activate Trojan
E-4. A voter or election official is able
horse or other malicious code that has been previously
installed, in order to affect or manipulate ballot contents or
4. totals.
voteThe persons attempting to compromise the
election process could be insiders with full
knowledge of the election system including,
but not limited to, political operatives,
vendor personnel, polling place workers, or
election administrators.
For elections, the principal asset is
governmental power. That power is
transferred by the results of counting voted
secret ballots. Hence, integrity of the
voted ballot is critical through the entire
process from capturing the voter's intent,
casting it into the ballot box, counting it
to produce the election results, and finally
retaining it to resolve disputes.

The principal vulnerabilities to the voted
secret ballot are (1) undetected compromise
of election integrity, (2) compromise of
ballot secrecy, and (3) denial of voting
service. [need to reorder existing 5.1.2.3
threats underneath this taxonomy]
Add: "4. The systems may be unattended or in idle storage
for periods of time."

Rewirte sentence: 2. There are people who
are motivated [and have the training and
ability] to compromise the election process.

Someone with legitimate or illegitimate access to the system
source modifies the voting system software directly, or
modifies the programming environment to modify the
software. The modification could be malicious code that
affects voting directly, or it could enable the subsequent
installation of malicious code. B into "Software Development,
Perhaps we can merge A and
Testing, and Distribution"
A third party is able to activate a Trojan
Horse remotely via RF, IR, network,
telephone, or other remote control.
A voter, poll worker, technician, or election official …
Add item E-5: A voter or election official
is able to surreptitiously connect an
external device to a voting machine and
tamper with the machine or its data using
functiionality resident on the external
device. An example of such an attack would
be connection of a handheld computer to a
voting machine through a device that
simulates the physical connection of a
smartcard used by the voting machine for a
purpose such as voter authorization.
Revise per comment
Election Verification: These standards
address election verification requirements
to ensure that all ballots are counted-as-
intended. Any change to election data
throughout the voting process shall be
detected by the voting system.
Insert a section in 5.1.3 entitled
Hardware/Software Security. The section
should state: “Voting equipment should be
exclusively dedicated to voting. However,
if shared equipment is used, all persistent
storage devices and media not dedicated to
voting should be physically removed or
disconnected from the equipment prior to its
use for any activity related to voting, with
the exception of the read-only firmware that
interfaces the hardware to the operating
system. All non-persistent storage,
including all input/output channels, must be
cleared before and after the voting activity
by writing at least three cycles of the
sequence of a character, its complement, and
a random character into all addressable
locations. The hardware, software, and
documentation provided must facilitate
conversion of the system between non-voting
and voting configurations. For example, all
hard drives must be removable so voting-only
drives can be substituted for non-voting
drives, suitable software must be provided
to clear non-persistent storage and
input/output channels, and users must be
instructed to safeguard voting-only devices
Insert a new section in 5.1.3 titled “Voter
Verfiiable Paper Audit Trail”. The section
should state “There is controversy about
the suitability of all-electronic systems to
provide protection against combinations of
malicious activity, human error, and/or
equipment failure that could cause a ballot
to be recorded that differs from the intent
of the voter. Also, Section 301(a)(2)(B) of
HAVA can be interpreted as requiring a voter
verifiable paper audit trail. Accordingly,
it is recommended that all DRE systems
include a voter-verifiable, paper audit
trail satisfying the requirements for such
features defined in other sections of this
standard.”
Add to the section created under comment SK-
4 above: A voter verified paper audit trail
is mandatory for any system in which any of
the following conditions is found: 1.
Either the system software or any COTS used
as either a system component or development
tool, including compilers, libraries, and
other tools, is too complex to clearly and
thoroughly evaluate at the source code level
to ensure absence of backdoors and other
malicious code or means of introducing
malicious code. 2. All other security,
accuracy, integrity, and availability
requirements are not satisfied clearly,
easily, and without any question or
requirement for interpretation. 3. - There
are any reports or significant suspicions
that similar technology may have failed to
record all ballots exactly as cast. 4. -
There is any question whatever about the
ability of all using jurisdictiions to
easily and completely satisfy all
assumptions regarding supervision of
machines and relevant personnel at all times
machines are in use, regarding fully secure
storage of machines between elections, and
regarding other procedures intended to
Defensive countermeasures are designed to
thwart attacks that exploit the principal
vulnerabilities in attempts to compromise
the assets. Countermeasures can be designed
into the data protocol, the
software/hardware implementation, and/or the
election administrative procedures.
Countermeasures typically fall into three
categories: protection, detection, and
deterrence.
Add the following sentence: "Because there
is no full proof way to protect against all
of the threats identified in Section 5.1.2.3
via software and testing alone, the voting
system must include a voter verifiable paper
ballot or its equivalent."
Wireless connectivity, both in the development and
deployment of election systems poses significant security
risks, such that it is necessary to require that the use of
wireless communications devices be prohibited in any stage
of election equipment construction, development and use.

There should be an audit record provided in the audit trail for
all access, along with a reporting mechanism for access
violations.
Four levels of subheadings should be
sufficient. Also, current formatting makes
it difficult to keep chapters, sections,
subsections, and sub-subsections straight.
Then sub-subsections appeared to have
disappeared. That makes it very difficult to
track changes. Suggest limiting to four
levels and formatting correctly before
resubmitting for review.




Develop a set of strong requirements.




Add a description of required security
measures for general access.
Add new item "g. Capability to restrict use
of common words, e.g. password, secret, and
common dictionary words for passwords and to
enforce strong password policies by the
voting jurisdiction need to be provided by
the vendor."
Add: “and shall log each and every access
with a timestamp and all data specified in
these features”
Item e should be removed, or perhaps
substituted with direction that components
incorporating unencrypted keys must be
physically secured.


"must log all activities associated with
ballot processing and restrict access based
on the type of request being asked for"
Add reason: f. Role-based and discretionary
access control [so that superuser accounts,
e.g., root, sysadmin, sa, ora, etc.,
accounts can be deactivated.]
Add h. All default superuser accounts, root,
sa, ora, sysadmin, etc., in all operating
systems and software installed in the voting
system shall be disabled and individual
accounts with such superuser privileges as
are essential and necessary set up by the
vendor. The voting jurisdiction shall be
encouraged to disable the vender superuser
accounts and establish its own.

Add minimal requirements for key length and
algorithmic robustness.

for voter verification, ensure that the votes on the paper
record can be concealed from others during and after the
voting process.
… vor voter verification, ensure that paper copies of
cancelled ballots do not reveal the voter's selections, by
providing means to destroy or conceal the ballot.
 the order and times of casting the votes can be obscured
either during the process of storing or retrieving them.
Delete or clarify this item.



Define uncommon acronyms at first use, and
perhaps at subsequent uses separated by more
than a single page.
Either add in parenthesis or substitute
"paper copy of voter selections"or change
previous sections.
Delete "if a paper copy of voter selections
is printed for voter verification" and
replace "deposit the detached paper copy"
with "deposit the voter verifiable paper
ballot"
Delete "if a paper copy of voter selections
is printed for voter verification" and
replace "print text on the paper copy" with
"print text on the voter verifiable paper
ballot"
Replace "any paper copy of voter selections
that may be printed for voter verification"
with "the voter verifiable paper ballot"


Replace "any paper copies of voter
selections that may be printed for voter
verification" with "any voter verifiable
paper ballot"
Replace "and the Voter Verifiable Audit
Record if provided" with "and the voter
verifiable paper ballot". Also, replace the
last occurrence of "Voter Verifiable Audit
Record" with "Voter Verifiable Paper
Ballot".
This section must allow either for the
voting station or the voter to deposit the
printed record.




…from being associated with a voter or the order that the
ballots were cast, while also ensuring ballot integrity.
Delete language.




c. Ensure that ballot image data does not
contain any fields [or codes, including but
not limited to timestamps,] that identify
the sequence that the ballots were cast or
in any way allow for voter identification;
"were cast, including provisional codes
associated with a CVR".
“In systems providing voter interaction in
multiple languages, the CVR and all copies
thereof, and voter verifiable audit records,
shall be recorded in which the voter has
chosen to vote.”




Delete "for voter verification" from each of the sections
since any paper record of the CVR should be handled in
this manner




Replace with "In systems providing voter interaction in
multiple languages, the CVR and all copies thereof, and
the Audit Record, shall be free of indications of the
language selected by the voter."
Remove the last sentence.




Drop the last sentence




Add the requirement that the firmware should
be open source.
Develop requirements for security access to
the voting devices after successful
completion of election day testing.




….embedded in the system but prohibited from being altered
during election operation.


End sentence at "completion of election day testing."




Voter verified ballots can be used to form part of the election
system's audit trail. These ballots can be retained for
possible later use in spot-checks or mandatory recounts, or
they can be optically/electronically scanned or hand-counted
to form or confirm the official tally of election results. Some
variations of the voter verified balloting system may permit
the voter to later "look up" or review an encrypted version of
their ballot that does not reveal its contents, but
mathematically can confirm that the cast votes were properly
entered into the official totals. A voter verified ballot can also
contain additional information that can be used by the
election officials or vote tallying system to confirm that ballots
have not been substituted, duplicately entered into the ballot
Eliminate “If software is resident information
box or vote totals, or altered in any way, but this in the
system as firmware”.
Change “may be verified” to “is reported”.
Add requirement that a checksum must be
produced and verified with some central
agency, preferably using a challenge so that
the person starting the device can not know
the expected result nor produce it on
her/his own.
Require physical locks with redundant keys
(so that no individual can gain access)
protecting the innards of the device so that
any unauthorized access will result in
obvious physical damage.
Explicitly specify minimum strength.
Change “may” to “shall” for the bootstrap
and monitor.



Deleted item.




Strike the second half of the sentence.

Remove this restriction unless there is
sufficient justification.

Indent and number the 3 points.

Add the requirement that all COTS used in
any voting system must be open source.

Enabling these audits should be required
only for systems supporting them.


Sentence starting "It also ensures the existence of an audit
record…" must be changed to ensure that ballots are not
identified to the voter casting them. As well, there should be
NO WAY to alter election data, other than to record ballots.
All existing ballot data must be retained intact in the event of
any termination or haltingincluding voter initiated deletions,
…largest expected data, of election software processes.
with maximum….
Add requirement: “The voting device shall
operate as a dedicated piece of hardware and
its hardware and software shall be tested
together.” Add reference to IEEE Std 1471™-
2000, “IEEE Recommended Practice for
Architectural Description of Software
Intensive Systems”.
Add requirement: “The voting device shall be
an embedded system consisting of dedicated
hardware and software tested together, and
complying with IEEE Std 1471™-2000, ‘IEEE
Recommended Practice for Architectural
Description of Software Intensive Systems’.”


Add requirement: “Any and all communications
between the voting device and the outside
world shall be initiated by the voting
device; furthermore, no data received from
the outside world shall be retained by the
voting device longer than is necessary to
parse the said data and drive the protocol
state machine.”
Eliminate paragraph.

Delete this item.

Add the requirement for voter verifiable
paper ballots.

…denial of service, and time and logic bombs. All
keypresses of any single or combination of voters and/or
administrators shall be precluded from adtivating any
software or firmware process other than those directly
pertaining to the election aspect being used. Data flow
Eliminate the used to sentence, controls.
analysis should be first validate thesethe preamble.




All [single or combination of] keypresses of
any single or combination of voters and/or
administrators shall be precluded from
activating any software or firmware
process[, e.g., an "Easter Egg" like
feature,] other than those directly
pertaining to the election aspect being
used.
Add requirement: “The voting device shall be
an embedded system consisting of dedicated
hardware and software tested together, and
complying with IEEE Std 1471™-2000, ‘IEEE
Recommended Practice for Architectural
Description of Software Intensive Systems’.”


Change to: “Upon commencement of use for an
election, the system shall perform a sanity
test of the CPU, test of the code image
against the error detection data therein, a
test of all RAM, of the real-time clock and
timers and the consistency thereof, and a
test of every other peripheral device;
failure of any of these tests shall be
logged and shall prevent use of the device
for voting; these tests shall be run
immediately prior to entering normal running
mode for voting. Also, when otherwise idle,
the system should repeatedly perform tests
of the code image and other data in memory
that can not be modified, of RAM that is not
in use, and of whatever other entities for
which tests may be devised.

Change “public network” to “communicate with
the outside world in any manner including,
but not limited to, a PSTN, VPN, or
Internet”
Move the words “in human readable format” to
the introductory paragraph of item “a”.
Change “data” to “date”.
Specify UT (GMT) and specify allowing
setting of local time zone for display
purposes; require, however, that UT be used
for all logging.
Disallow altering of the system clock;
mandate that a defective system clock be
replaced. This is necessary to prevent
tapering with what is reckoned as during
polling hours. Also, recommend use of
checking the clock against broadcast radio
signals such as provided by NIST.
[Minimum r]equirements for the content of
audit records are described in Section
[4.5.4] of the Standards.
eliminate or replace second reference in
same sentence
Committee needs to decide if TDP is
proprietary or not.




Change 4 to 4.5.4   Add the additional
reference?




Delete the first two sentence of section
5.1.3.5.1, because they are false. Election
audit trails in no way verify the
correctness of the reported election
results!




Add to the end of the paragraph: “Access
control of audit data is on a “need to
perform” basis to ensure that election
officials can make the data public at the
appropriate time according to election
procedures. At the time prescribed by law
and procedure, the general public has a
“need to know” the data. However, the
integrity of the data must be protected at
all times.”
Reduce the scope of the introductory text in
5.1.3.5.1.
Replace "essential" with "necessary but not
sufficient"




Either remove the statement in parenthesis
or substitute the statement " Automatic
ballot generation is only permitted under
the conditions defined in section
5.6.7.2.1."




Develop extremely advanced AI technology or
time travel capabilities to fetch it from
the future.
Eliminate or drastically restructure this
requirement.


We need to modify the wording to permit test modes as long
as they cannot be exercized during the voting process or
delete the parenthesis so that the language in the other
sections governs.
Replace with "The ballot interpretation logic shall test
and record the correct installation of ballot styles or
formats on voting devices for the voting precincts at the
polling location and that the ballot logic produces a
correct count for each candidate and issue on the ballot
(NOTE: The system shall only automatically generate
voted ballots when in a test mode);"
Change the wording to "Identification of the
voting location for which the voting system
was programmed."



replace with identifying the election
location that is being voted by the machine

I believe these items are intended to be
verified by a human operator. The
requirement should be simply for the
information to be output.
delete

eliminate "human readable message" or add
code can reference it in some documentation
optionally
There shall be a secure time/date stamp protocol used as
well as a human readable format.


… the greatest extent possible (in accordance with accepted
industry practices).




Reword: The system shall be capable of
producing a backup copy of any electronic
audit records…
change to relative time clock not
registering date
CVR must me excluded from review when system
is activated for elections.
Add: “Any election data transmitted from the
voting machine to a remote location, being
subject to many sundry methods of tampering,
shall not be used in any final, official
tally, but rather may only be used for a
preliminary tally; all election results
shall be constructed from data transferred
directly, without use of any network or
other open system, from a physical device
that resided in the voting device during the
election to a tallying device at the
jurisdiction’s central elections office.”

Decide and specify what is meant by "Local
Area Network."


Replace “datagram” with “message” wherever
it appears.




Additionally, only [currently certified]
devices and applications will be allowed to
interface with the voting system hardware or
software.
Can we dispense with the overspecification
of specifics? How about "Voting systems
which transmit data shall ensure the
integrity of all transmitted data"?
Only use of wired transmission should be allowed in this (and
any) section of the standards document.


Protocols to enforce these conditions must be non-
proprietary and capable of demonstrating correctness,
accuracy and integrity.
Change “datagram” to “packet” or “message”.

Replace with "Be configured so that only datagrams
authorized and required by the voting system appear on
the physical network medium and that datagrams from
the voting system are not transmitted to non-voting
systems."
Replace with "Manage encryption keys to ensure that
the keys are not compromised and that the keys are
changed on a periodic basis."
Reword as follows "Manage encryption keys
to…."
Specify some official category used by NIST
or specify a functional description of the
cryptographic strength required.
Related to comment 12, above. Physical
isolation is a better solution than
encryption 100% of the time.


replace the wording with more generic
references; it is not a given that a PKI
structure must be used
Reword as follows "Configure the local
network so that…."
Eliminate the section.

Add: “Any and all threads used for
interacting with the voter and recording
data internally to the device shall have a
higher priority than any other task on the
system, excepting perhaps a monitoring, self-
check task of nominal bandwidth, so as to
prevent any external communications from
disrupting the main functionality of the
device.”
Add: “All modules buffering or processing
data communicated from an external source
shall use a pre-determined amount of RAM so
that no babbling or malicious communication
with the voting device can degrade the
voting related functionality.”
Eliminate “and software” from the first
paragraph and eliminate item “a”.
Cabling and external attached connectors.


Eliminate the section. Add a requirement:
“All communications between the voting
device and the outside world shall be
initiated by the voting device; the voting
device shall refuse to buffer or parse any
communication not requested.”
This should include appropriate and timely reporting of any
observed breaches or breach attempts during the election
setup, actual election, and post election and canvass.




Delete it
Change “may” to “shall”.
Eliminate or severely modify section
5.1.3.4.2.
Clarify the text.
replace the wording with more generic
references; it is not a given that a PKI
structure must be used
Additionally, only [currently certified]
devices and applications will be allowed to
interface with the voting system hardware or
software.
Communication interfaces to the public
telephone network or any other network
outside of the physical boundaries of the
polling place should be disabled in real
election mode.
Election verification includes voter
verification and results verification. Voter
verification shall provide mechanisms to
ensure that each ballot is captured and cast
as intended by the voter. Results
verification shall produce an irrefutable
tally to ensure that anyone can verify all
ballots were counted-as-cast.
Voter verification ensures that the voter's
ballot accurately captures the voter's
intent and is contained in the ballot box
for counting. Since only the voter knows
his/her intent, only the voter can be
responsible for voter verification.
The voter shall have the option to access
human readable information that securely and
uniquely connects the voter's intended
choices to a ballot, according to the
tabulation rules.
1. The voter shall have the option to verify
that the ballot cast by the voter is in the
sealed ballot box.

2. The voter shall have the option to
determine that the voter's ballot, as
captured in the sealed ballot box, preserves
The same connection to the voter's intended
choice as was indicated by the human
readable information presented to the voter
during the act of voting. The information
granted to the voter shall provide
irrefutable proof of any discrepancy.


Results verification shall produce an
irrefutable tally from the sealed ballot box
such that any election auditor or observer
can determine, with full confidence, that
all ballots were counted-as-cast. Measures
shall be in place to ensure that votes
cannot be added, removed, or modified.
Election policies should provide precise,
publicly accepted procedures for creating
(“sealing”) the sealed ballot box at the
proper time.
1. The sealed ballot box shall be made
available for scrutiny to any election
auditor or observer.

2. An election auditor or observer shall be
able to determine, with full confidence, for
each ballot in the sealed ballot box,
whether it is “legitimate” in the sense that
it was cast by a unique, eligible voter.

3. An election auditor or observer shall be
able to determine, with full confidence,
that no ballot in the sealed ballot box has
been undetectably deleted or changed.

An election auditor or observer shall be
able to reproduce the election results, with
full confidence, per the tabulation rules.

Wireless connectivity, both in the development and
deployment of election systems poses significant
security risks, such that it is necessary to require
additional security measures to specifically mitigate the
risks posed by the use of wireless communications
devices during all obvious if it is non-obvious.
Only state the stages of the election process.



…be able to produce a vendor-independent, non-proprietary,
human-readable report of all votes cast.

DRE systems that provide voter verified ballots may also
maintain an electronic copy or tally of results. Should there
be any differences noted, the voter verified ballot set should
be considered to supercede any electronic copy of the cast
ballots.voter-verified paper copy is made it
If a
shall be deemed the "official" or "correct"
copy.
voting devices and or memory modules
Replace with "As an additional means of ensuring
accuracy in DRE systems, voting devices shall record
and retain redundant copies of the original CVR. This
is a requirement whether or not a paper copy of voter
selections is printed."
eliminate this



The annonymity and security of new machines
must be as good as the security in the
processes they are replacing: ie people
overseeing each others handling of paper
ballots
eliminate: for voter veriraication

eliminate: for voter veriraication

eliminate: for voter veriraication

If this is a matter of statistical analysis
of the testing results, it should be stated
as such.

There must be a zero-error rate, so as to not to affect the
outcome of any election. If a zero-error rate is not able to be
achieved, jurisdictions must be informed as to the actual
error rate, such that if an election falls within this range
between candidates, a run-off election must be required.
A mechanism for differentiating between deliberate
undervotes and votes lost by the system must be employed,
such that the error rate can be ascertained without doubt.
2) Independently from voting data storage,
recording voter selections of candidates and
contests into CVR storage [and on to
optional paper copies if so equipped.]



Explicitly address accuracy requirements for
human input devices.


The proposed new text:

a. For paper-based systems:
1) Scanning paper ballots to detect
selections for individual candidates and
contests;
2) Conversion of selections detected on
paper ballots into digital data;

This section in the document mentions provisional
ballots and may require change to deal
appropriately with provisional ballots.
Need some way to determine the DRE memeory
is error free. Unless DRE memory is in test
mode for 22 months prior to determine this
factor.
Replace with: "System memory devices used to retain control
programs and data shall have demonstrated error-free data
retention for a minimum period of two years. This
requirement shall apply to any memory module addressable
as a single memory device, even if it internally uses
redundant storage elements. This requirement on memory
devices is separate from the requirement of 5.2.1.2 and
5.2.3.2 (c) that specific types of data be stored in multiple
memory locations; each such location would be subject to
the requirement of this section."




replace with language saying the ITA deems
the memory capable of retaining data for 22
mos
Providing spec. data from the manf. proving
this capability would constitue compliance.
At the end of the last sentence insert “or
print-outs on paper”
The use of magnetic media, or media that can be readily
altered should be avoided, or additional mechanisms and
controls put in place to ensure that election and ballot data is
not changed or destroyed.




Delete item



We suggest the following change:
In addition to the common standards, DRE
systems that record votes on the voting
station shall:
a. Maintain a record of each ballot cast as
a CVR using a process and storage location
that differs
from the main vote detection,
interpretation, processing, and reporting
path;

Add at end: "This requirement applies to retrieval of Cast
Vote Records even if the system also provides voter verified
ballots, although in this case, if there is any difference in
results, the voter verified ballot set shall supersede any tallies
generated from the electronically recorded data."
Add:
"d. Provide output of CVRs, audit data
records, VVARs and related election
information in a manner that allows
detection of any subsequent alteration or
degradation of their data, even if stored on
a removable medium."




        Replace with "These are requirements whether
        or not a paper copy of voter selections is
        printed."
...existing immediately prior to (and not the cause of) the
error or failure….
Change “scenario consist” to “scenario
consists”.
Change “consisting” to “comprised”.




Allow one machine per polling place to
reboot once during an election. Assume a
reboot takes 2 minutes.
This number should be corrected as per the discussion in the
July meeting.
Discuss in the working meeting; suggest 109
or 1010seconds.
Increase the required MTBF to 1500 hours,
implying a 1% probability of failure during
an election and a 2% probability of failure
during equipment setup and readiness
testing.
Incorporate an average 1 year storage time
into the scenario and add a sentence after
the MTBF requirement sentence specifying a
probability of operation of 99% after one
year of storage (equivalent to a 1% per year
failure rate in storage).
Although a more quantitative basis for
assessing maintainability, such as the mean
[time] to repair the system is desirable,
the qualification of a system is conducted
before it is approved for sale and thus
before a broader base of maintenance
experience can be obtained.
Discuss in the working meeting.



Prohibit modification, including for repair,
of the device from the time the ballot is
configured through the retrieval of election
data.
Need to add wording to reflect the comment here in the
appropriate place.
add wording "for a specific equipment type"


Remove 'DRE'




delete this sentence and add the following
to the previous sentence. "as the DRE
presents the ballot to the voter."
replace dots by alpha numeric numbering
Move parenthesis as follows: "...bias that
would (either intentionally or
unintentionally) encourage or impose…"
remove it


remove it

remove the table. By adding the distance
variable compliance to the standard is
unobtainable. In addition, the font height
is already defined in section 5.3.10.2-1




Delete “DRE” in the first sentence. Where a specific
requirement is applicable only to DRE, call that out in the
requirement or section heading; there may be none.
Redraft and present all units in centimeters
and inches.


Delete “DRE” in the first sentence. Either
provide a table indicating applicability of
principles to systems by type (DRE, non-DRE)
or provide an indication of applicability in
each statement.




delete speech input


* Recountability - the election totals should be readily and
independently verifiable if required by law, procedure, or
litigation.
Privacy should not prevent the ability of the election officials
to determine and ensure, between voter uses, that the
election equipment is not being tampered with (or used, by a
Add voter to or more than one ballot, for example).
singlebefore cast after (3) another list item:
"The votes on an individual machine shall be
stored in such a way that it is not possible
to differentiate votes by whether
accessibility features were used."
replace with the comment to allow connection
of their assistive devices through some
common connection (is referred to later)

Add informational annex with illustrative
examples, as attached




delete last sentence




remove item
Please add the following:
When audio content or instructions are
provided, allow the voter to select a
language in accordance with languages
available according to the Voting Rights Act
of 1965, as amended. Also, voters shall be
permitted to verifiy the ballot in the
language in which the vote was cast.
delete this point



Should be removed - don't be prescriptive on
the implementation



Add informational annex with illustrative
examples, as attached



make no refernce to fond size but change to
Font shall besuffisiant and remove text
between brackets

Add: 5. Where a voting system provides a voter verified
ballot in a visual form (e.g. hardcopy) a system providing an
audio output for voters shall
a. Read ballot data for the audio rendition of the voter
verified ballot from a data path common with the main output
of the Cast Vote Audit Record and different from that used to
render the information in Section 5.3.4, paragraph 6.
b. Provide means for the voter to terminate audio
presentation of the voter verified at any point, including prior
to presentation. However, the system need not provide
means to reject the ballot prior to presentation of at least
Change sentence to read "Alternatively, the
text size can be made adjustable.
Change the wording to read "Braille can be
provided where practical, "
clarify


remove the word option




clarify

clarify

clarify

renumber this article
Eliminate "wireless coupling" from this section -- must be
hard-wired.

delete this point

delete speech input


5.3.10.5 Accessibility for Voters with
Limited Movement[,] Coordination[, and
Prosthetic Devices]
delet 3



Eliminate reference through allowing
assitive devices to be connected or the unit
to be repositioned to allow better access?

Change to 5.3-6 through 5.3-7.
Add figure number.
delete: that will allow voters who use
wheelchairs to approach the voting station
in the same orientation as voters who do not
use a wheelchair. Insert the drawing from
fec 2002 where the wheelchair voter
approaches the voting station paralllel
 Insert the drawing from fec 2002 where the
wheelchair voter approaches the voting
station paralllel

h. Where possible, voter should be able to use the privacy
screen to conceal their vote choices in a similar fashion to
non-disabled voters.
Use cm and in as units consistently.



a. Where clear floor space only allows
forward approach to an object, the maximum
high forward reach shall be [60 cm (24
inches)]. The [maximum] low forward reach
shall be [30 cm (12 inches)].



b. Where forward reach extends over an
obstruction with knee space below, the
maximum level forward reach shall be [60 cm]
(24 inches). When the obstruction is less
than [50 cm] (20 inches) deep, the maximum
high forward reach [shall not exceed 20 cm]
(4 inches) [past the obstruction]. When
the obstruction projects [50 to 60 cm (20 to
24 inches)], the maximum high forward reach
shall [not exceed the depth of the
obstruction].
c. The position of any operable control
shall be determined with respect to a
vertical plane [that is between 107 and 122
cm (42 to 48 in) above the floor] centered
on the operable control, and at the maximum
[ ] the [control shall be within [60 cm (24
in) in front of the voter].
d. Where any operable control is [26 cm] (10
inches) or less behind the reference plane,
the height shall be 137 cm (54 inches)
maximum and [60 cm (24 inches)] minimum
above the floor.
e. Where any operable control is more than
25 cm (10 inches) and not more than 61 cm
(24 inches) behind the reference plane, the
height shall be 117 cm (46 inches) maximum
and 60 cm (24 inches) minimum above the
floor.
g. Operable controls located on horizontal
surfaces [in front of the voter] shall have
a tilt of between 10 and 20 degrees to make
them easier to see and access from a sitting
position
shall have a tilt of more than 10 and less
than 90 degrees to make them easier to see
and access from a sitting position.
change the wording to the sentence to:
Voting equipment shall provide features that
ensure voter independence while the voter is
voting.

Align 9 and 10 with earlier entries.
Consider the potential differences of use of
a right target vs. a left target.




Correct indentations.




remove table and 1 dot referring to
character size


define the types of coulor blindness to be
addressed
Tallying should be tested using all language options to
ensure votes are registered to the appropriately selected
candidate.
* Use of colors should not impose external bias on vote
selection.

The table should either contain 2 columns,
indicating minimum and preferred, or if
applicable, there should be a minimum,
maximum and preferred column. (The column
headers should abvioulsy line up as well.)
Simplify to one distance, minimum font and
minimum zoom font values
User adjustibility in a polling place is not
a good idea. Requirements should be able to
be met with multiple ballot styles for the
voter to choose at the beginning of a voting
session.
remove the requirement and address this
issue in the contrast and luminance
requirement.

narrow the ratio range to 5 to 1 to 7 to 1


remove bolding from sentence


specifically define what well-known graphics
are (i.e. a red octagon indicates stop or
warning).
remove the sentence

change wording to "To avoid confusion it is
recommended that no more than six colors be
used.


remove 8th bullet
Change the sequence to 4 is third, 5 is
forth and 3 is fifth.




he following solution is offered:

“Once the ballot is submitted, the voter
shall be prevented from making any further
modifications to the ballot or shall be
prohibited from casting another ballot.”
delete when possible and start with The
system shall…


change text to:          If a voter chooses
to delete data, the system shall provide
immediate feedback that the data has in fact
been deleted and the voter must be able to
reenter his or her choices


delete and a error message should be
presented
For paper-based voting systems, such as mark-sense or
voter-verified, there must be a well-identified procedure for
the voter to use to "spoil" their ballot and obtain a new one.
This must be within the constraints of laws pertaining to
number of times and reasons allowed for spoilage. Ballots
"spoiled" either physically or electronically must not be
entered into be avote tally for cast ballot audit trail. thus
There must the method or ensuring that the data
deleted has not been entered into the vote totals.

Informing the voter that their votes were properly registered
may consist of the preparation of a paper ballot containing a
summary of the ballot choices the voter selected, that can be
presented to the voter for verification.
For paper-based voting systems, such as mark-sense or
voter-verified, there must be a well-identified procedure for
the voter to use to "spoil" a ballot and obtain a new one. This
must be within the constraints of laws pertaining to number of
times and reasons allowed for spoilage. Ballots "spoiled"
either physically or electronically must not be entered into the
remove item
vote tally or cast ballot audit trail.


use the word "selections"
remove portion of sentence

As indicated

Change title to "Voter Input/Control Devices
and Feedback"    Further, make an opening
statement that clarifies this point.



Specify the preferred embodiment or if
neither, identify such.

delete


5. Input devices and controls shall be
designed to [minimize or] eliminate [ ]
accidental actuation. As warranted,
instructions shall be provided indicating
their proper manner of operation.
If SPL is correct then need definition and
add it to section 3.
Change separation to 3 cm (1.2 inches) and
minimum height and width of 2 cm (0.8
inches)
· If there is a "home position," the
capability for an automatic return to that
point should be provided [that is
intuitively obvious to the voter.]
revise language to reflect comment


add wording such as "It is recognized that a
touch screen inherently averages the touch
selection. For example, when 2 points are
selected on a touch screen and the driver
averaged the distance between the touches
and activates the nearest area.
decrease separation to .1 in




Provide no standard here, allow poll workers
to manage this time.
delete last sentence



States that impose a time-limit for voting must be informed
about the length of time that may be necessary for all voters
(including disabled) to cast a ballot using the system. If this
amount of time exceeds the state regulation, then an
exemption must be applied.

Too prescriptive and again a system feature




clarify

clarify

add wording to beginning of sentence "When
applicable…"
While the system is processing a request
(e.g., proceeding to display a new page), no
further button presses shall be recorded or
stored, i.e., type-ahead capability shall
not be provided.
Change the number in parenthesis to 1
second. Put the parenthesis at the end of
the sentence.




We recommend using the same language as the
VSS:

g. The system should provide feedback to
user inputs in less than a second, but if
processing takes longer, feedback should be
provided that the system is processing the
voter's input.
change to 1.5 seconds or less


replace with > 10 seconds, the system should
provide feedback to alert a voter that
activity is still being performed until it
is completed
combine items
The system shall check user inputs for
acceptability, e.g., check for inputs that
seem to be in error (such as putting [an]
Arabic number in a name field) and alert
voters to the error.
We recommend dropping paragraph 2.
last sentence should read When a attempt to
overvote is made, the syystem should not
accept the choice and alert voters to the
condition.
actions can be reversible or restarted
through a ballot cancellation process
This was a poor example, remove it.


clarify




Allow red to indicate warnings.
Restate the sentence to say help can be
provided independently from the voter
interface itself.



The environmental requirements for voting
systems include [secure] shelter, space,
furnishings and fixtures, supplied energy,
environmental control, and external
telecommunications services.
Either require COTS equipment to comply to
the same standards as all other voting
equipment or remove the paragraph
altogether.
All precinct count systems shall be designed
for storage and operation in [an] enclosed
[secure] facility ordinarily used as a
[secure, locked] warehouse or polling place,
with prominent instructions as to any
special storage [or security] requirements.
[For example, voting systems kept in
unlocked storage containers will require a
higher level of secure storage facility than
those voting systems with a securely locked
storage container.]




Describe the requirements directly.
a. Systems shall operate with the electrical
supply ordinarily found in polling places
(120[VAC/60Hz]);
[c.] The backup power capability is not
required to provide lighting of the voting
area [or voting system].
…should retain the contents of all memories intact, until the
end of data collection or recovery efforts.

systems shall be capable of operating on
back up power, such that no voting data is
lost or corrupted. When power is inturrupted
or cuized the system shall retain…..

last sentenc should read: The back up power
is required to provide lighting of the
voting area
Change to read (44° to 104° Fahrenheit)




Fix typo and use FEC 2002 VSS requirements.
Maintain what the current standard calls out
or grandfather equipment that already meets
2002 standards.




Leave standard at 50F-95F or grandfatehr in
equipment that already meets 2002 standards.




Describe the requirements directly.
See comment 254 and notes in previous review
about adding shock and vibration standards.
Add shock and vibration standards.

Add standards for operation at elevation and
for non-operating air shipment.




Resolve difference, if any, with FEC 2002
VSS logo requirements
b) Ensure that components provided by
external suppliers are free from damage or
defect that could make them unsatisfactory
[or hazardous when used] for their intended
purpose.
add item "e" before: Electric power
……level".
Maintain what the current standard calls out
or grandfather equipment that already meets
2002 standards.
a) 2 kV AC & DC [in] external power lines;
Define these two requirements in a clear,
concise fashion.
Check values and specify more realistic
values.

c) +.5 kV DC line to line [at distances]
>10m [between lines];
d) +.5 kV DC line to earth [at distances]
>10 m [from the voting system]; and
e) +1 kV I/O [signal]/control [at distances]
>30 m [from the voting system].


Superscript 26 at end of sentence.
When memory devices are used to store votes
and which voters or poll workers will handle
as part of normal use, these memory devices
shall be tested both [while] installed and
separately from the host equipment.

Make footnote 26 superscripted.
make 26 high case
26. Some jurisdictions which often
experience high levels of [electrostatic
discharge] due to typically low humidity or
high altitude may wish to increase the level
to ±25 kV air discharge. Such conditions are
commonly found at elevations above 2 km
(6,500 feet) or in high desert regions.

Leave the standard at + 15 kV and drop the +
25 kV "option".


replace with comments




Allow FCC Part 15 Class A
from "10 GHz" to "2.5 GHZ"

Renumber to be consistent.
Should be a) and b)
Restate these specifications in clear,
concise terms that even I can understand.
Should be 150 KHZ acc to IEC 61000-4-6.
Note: FEC 2002 in 3.2.2.11 Conducted RF
immunity does not state any freuquecy range.

should be I/O



Define the unit for magnetic fields.
Specify that the measurement shall be RMS.
Re-write of section (in process).




Vince Lipsio to rewrite section 5.6 and
submit for review. Comments on specific text
in this section follows.
Add section specifying the monitoring of
stacks at run-time. Example wording
(paraphrased from a real SRS approved by
the FDA): “Each stack’s high water mark
shall be monitored at least once a second;
60% usage of any stack shall be logged as a
software anomaly and 80% usage of any stack
shall result in a fatal software error .”
Add section specifying minimal requirements
for the handling of hardware and software
exceptions.
Delete “Unmodified third-party software is not subject to code
examination; however,” and replace it with “All third party software
shall be subject to source code an d other examination to preclude the
presence of trap doors, hard-coded passwords, vulnerabilities and
other non-deliberate errors, deliberate errors allowing the introduction
of malicious code, and malicious code of any kind, especially
malicious code intended to trigger upon use of the software in voting
systems.”
Eliminate reference to single-entry, single
exit procedures. Eliminate prohibition of
infinite loops. Add sections do deal with
shared data and semaphore requirements. Add
section to deal with race conditions, dead
tasks, tasks being permanently excluded from
running due to erroneous pre-emption or
waits, etc.
Eliminate prohibition of infinite loops
(because these are the typical
implementation of a task); eliminate
reference to single-entry, single exit
procedures (or substitute something that
makes sense in the context of a multi-
threaded system).
Add section to specify safe use of shared
data and requirements for semaphores.


Add section to deal with race conditions,
dead tasks, task deadlocking, tasks being
permanently excluded from running due to
erroneous preemption or waits, and other
such inherent problems of multi-tasking in a
real-time system.
Reference (indirectly here in the scope,
directly throughout the section) at least
the following: IEEE Std 1228-1994, IEEE Std
829™-1998, IEEE Std 1028™-1997, IEEE Std
1471™-2000, IEEE Std 1016™-1998, IEEE Std
14143.1™-2000, IEEE Std 1061™-1998, IEEE Std
1061™-1998, IEEE Std 1008™-1987 (R1993), and
IEEE Std 1228-1994.
Add the phrase “and embedded in the device
so as to require physical modification to
the device to alter” to the definition of
firmware.


Eliminate all such subjective references or
move them to an annex of recommended
practices.
COTS products, especially software libraries, are a
vulnerable attack point and must be subject to risks
assessment prior to use in voting products. Configuration
management should include vendor updates and alerts when
flaws are detected that could compromise election
operations or cast ballot data integrity. Object code modules
should be provided such that compiled versions of programs
Delete this clause
can be compared.




Ensure compliance with section 4.3.11
(“Previously developed or purchased
software”) of IEEE Std 1228-1994, “IEEE
Standard for Software Safety Plans”.


A more complete description of “formal
tests” should be given.
Eliminate the phrase “Unmodified third-party
software is not subject to code examination”




Require a single, invariant code image for
the system, thusly forcing all these
concerns to be addressed when the software
is built, before testing and V&V, and
require a checksum of the validated and
released binary image to be recorded and
checked upon initialization of the system.

add comments




The intention of the Voting Systems
Standards Source Inspection Process for
Independent Test Authorities (ITAs) is to
prevent deviant or malicious code from being
introduced into the voting process; to
safeguard from external threats being able
to effect unintended changes to voting
processes or data corruption to occur as a
natural part of voting processes. The ITA
is obligated to identify any such threats
and the submitting vendor is expected to
resolve these occurences before approval.
In order to review this code properly, an
ITA must be able to follow the code clearly
to ascertain whether any of these defects
have occurred - this has been addressed as
readability and maintainability guidelines
in the standards. When an ITA cannot
understand the code that has been submitted
for review sufficiently to determine if any
dangers exist, their request for changes and
improvements for readability should be
enumerated using items defined in the
guidelines. The ITA reviewing the source
has the lattitude to request changes using
these guidelines only when they feel the
Add: “and all configuration items allowed by
the system shall be validated. Furthermore,
each configuration item shall be range
checked when fetched so that hardware
malfunction or tampering can not allow an
untested configuration to be used.
Provide independent reports for each ballot casting device as
well as consolidated one.

* Ensure and confirm that extracted ior duplicated nformation
is identical to that on the original cast ballot storage medium.




Change “designed” to “implemented” or
“coded”.
Delete. Should have its own IEEE Manual and
refernce that manual.

Since assembly language is not considered a
high level language, Delete the inclusion
of Assembly language within this section.
Alternatively, if an operating system
software may be designed in assembly
language such operating system software is
required to meet all the same provisions or
requirements any other operating system
software is subject to.
This restriction should be removed as long
as a modern compiler is used that permits
the source code to written symbolically and
it follows all the rules required of the
source code written in higher level
languages.




Move sections to an addendum

Require all tools, including compilers and
interpreters, to be validated and verified
in the same manner as application software.


Add requirement that all executable code
shall either reside on ROM or shall be write-
protected while the device is operational;
if the latter, the code’s checksum shall be
generated after disabling the write-enable
signal to the memory containing the code
image.
Prohibit dynamically allocated memory or, at
least, force graceful system reset if
dynamic memory exhausted.



Add bullets to check all cases in a
switch/case statement, and to check all
subscripts when referencing an array, and
range-check all data when writing or
fetching.
Add: “else, the application software shall
explicitly check to ensure against such
situations”.

Change “imbedded” to “embedded”
Change “header” to “readMe file”.




Nothing   presently; I'll enumerate these and
convert   them into recommended best practices
at some   future time.
Delete.   Common in structured programming.

All modules thant construct a function
should be tested together, including library
modules. Actual modules should be used for
all test versions.
Eliminate the section, or, better yet,
reverse its sense.



Mandate that testing preclude any security
breach or vulnerability; mandate compliance
with section 4.3.11 (“Previously developed
or purchased software”) of IEEE Std 1228-
1994, “IEEE Standard for Software Safety
Plans”. Mandate COTS be subject to the
specifications of IEEE Std 1008™-1987
(R1993), “IEEE Standard for Software Unit
Testing”. Add reference to IEEE Std 982.1™-
1988, “IEEE Standard Dictionary of Measures
to Produce Reliable Software”.
Bring into conformance with Annex D (“V&V of
reusable software“) of IEEE Std 1012-1998,
“IEEE Standard for Software Verification and
Validation”, e.g., “Reusable software (in
part or whole) includes software from
software libraries, custom software
developed for other applications, legacy
software, or commercial-off-the-shelf (COTS)
software. The V&V tasks of Table 1 are
applied to reusable software just as they
are applied to newly developed software.
However, the inputs for these tasks may not
be available for reusable software, reducing
visibility into the software products and
processes.“
Change everything that can be tested against
to a “shall” sentence with objective
criteria. Move everything else to an
addendum of recommended practices.
Remove this clause.




Either eliminate the requirement or inspect
for compliance.


In the second sentence, after “security
requirements defined in” insert “Section
5.1.3.1 and”.
In the second sentence, replace the comma
after “security patches” with “and”.
Replace “and must be tested” by “. In
complying with the requirement of 5.1.3.1,
the vendor must document how the COTS has
been defended against the threats identified
in 5.1.2.3 (A-1), (A-3), (B-1) and (B-2),
such as by testing”.
COTS to be evaluated shall include
compilers, libraries, and any other software
tools used in system development and capable
of introducing backdoors or other malicious
code.
This section has several problems. The
module usage should be changed to subrouting
or function, remove the strict requirement
of only one exit per subroutine or function.
Change so the most recent version of COTS is
not required.
Eliminate “Headers are optional for modules
of fewer than ten executable lines”.




While should be added to Do-While. Recursion should be
allowed only if detailed description of need added.
Concurrent process flow should be allowed if detailed
description of need added. Assignments within branch tests
should be avoided (ex. use of if (a=b) should not be used,
rather, do the assignment outside of the test).
Delete this clause. Different languages
have different constraints which perform
this same function. Such limitation in the
standard, tends to restrict or lock out
newer programming techniques, in favor of
older techniques.             Is the intent here to
issue a new standard fot older technologies?
The specification should be revised to state
that GoTos should be disallowed except where
required by the programming language.




Delete. Names should be more than one
position and be discriptive, but a name that
changes by one position does not change the
functionality .



Change “its executable” to “all”.




Add requirement to include in audit log: a)
The system shall generate, store, and report
to the election officials and, where
appropriated, the voter all error messages
as they occur [and all such messages shall
be written to the audit log];
Add definition: c) All error messages
requiring intervention by an operator,
precinct official or voter shall be
displayed or printed unambiguously in
[English and the language selected if the
error affects the voter or their vote], or
by means of other suitable visual indicators
without compromising voter privacy;

replace with idea printed error messages
could be codified and referenced for
performance/execution constraints
Add after first sentence: The system shall
display and report critical status messages
using unambiguous indicators or English
language text. [If voter interaction to
correct the error is required the system
will also display the error message in the
language selected by the voter.]
The system need not display non-critical
status messages at the time of occurrence
and may be stored in memory [and the audit
log] to be recovered after ballot processing
has been completed.
c) Register and accumulate votes in a secure and
[permanent] location;




Change “50 states” to “sundry
jurisdictions”.




Add cumulative voting
r) No vote selected s) Blank ballot cast. * All of these
variations must be implemented in such fashion that it is
readily discernable that: a) votes are registered to the
appropriately selected candidate and b) tallies reflect the
algorithm properly (as in N of M or IRV), considering under-
and over-votes correctly.

This section in the document mentions provisional
ballots and may require change to deal
appropriately with provisional ballots.




Change “set to zero” to “recorded in the
audit log”.
device registers ballots shall provide a
public and a protective counter. The
protective counter shall give the total
number ofballots or cvr for DRE registered
in elections and tests on each device
must be set to zero before any ballots or
cvr are registered
All systems shall provide a means of
installing ballots and [certified] programs
on each piece of polling place equipment...
election data is organized to link only
allowable ballotstyles to poll place
specific locations
f) Segregate test data from actual voting
data, either procedurally or by
hardware/software features and [ensure] that
reported results cannot combine actual
voting data and test data.
Renumber to be consistent.


Renumbered: g) These elements shall be
capable of being tested separately, and
shall be proven [and certified] to be
reliable verification tools prior to their
use; and
Add bullet requiring a checksum must be
emitted and verified with some central
agency, preferably using a challenge so that
the person starting the device can not know
the expected result nor produce it on
her/his own.
refer to comments




Change to: “Automatic disabling of any
device until self-test has successfully been
completed.”
Remove the word "punching or"

Add “and all attempts of such actuation,
whether or not successful, shall be entered,
with a timestamp, into the device’s audit
log”.


election data is organized to link only
allowable ballotstyles to poll place
specific locations
add "within the same voting session"
Delete section.
j) Prevent or detect the attempted feed of
overlapping ballots, and if detected halt
the reading of the ballot and provide a
message to [election officials, the
operator, and the audit log] identifying the
condition;
delete punch card
remove "punch or "

delete or appropriate election official
…without correction, appropriately tallying all validly voted…




i) For electronic image displays [or paper
verified ballots], prompt the voter to
confirm the voter's choices before casting
the ballot, signifying to the voter that
casting the ballot is irrevocable and
directing the voter to confirm the voter’s
intention to cast the ballot;
o) Provide a capability to retrieve CVRs in
a form readable by humans (in accordance
with the requirements as specified in the
DRE System requirements subsections of
Section 5.2[)] without compromising voter
privacy;



delete

All systems must provide a means to close
the polling place, provide capabilities to
accumulate and [report] results [ ] for the
jurisdiction, and to print audit trail[ ]
reports.
If the system provides the capability to
transmit results [to another facility],
additional requirements apply.


refer to comments
Add the following: "Produce a Voter
Verifiable Paper Ballot;"
k) Ensure that extracted or duplicated
information is identical to that on the
original Cast Vote Record storage medium
[for DRE machines].
Make the first sentence read “ ... design
of the system, potential exposure to risk,
and the threats identified in Section
5.1.2.3.”




Add design review and source code review for security and
confidentiality.
Delete first sentence up to the first comma.


Delete "… from types of attacks known at the time the
system is submitted for qualification."
Specify something about the attacks
Add: “Any election data transmitted from the
voting machine to a remote location, being
subject to many sundry methods of tampering,
shall not be used in any final, official
tally, but rather may only be used for a
preliminary tally; all election results
shall be constructed from data transferred
directly, without use of any network or
other open system, from a physical device
that resided in the voting device during the
election to a tallying device at the
jurisdiction’s central elections office.”

Add the text of the comment to the
paragraph.

Insert a new section 6.1.1 titled “System
Design” and stating “The design
documentation explaining how the voting
system is designed to counter and defend
against the threats of Section 5.1.2.3 shall
be analyzed. Tests shall be conducted to
verify that the system adequately defends
against these threats. These shall include
conducted or simulated attacks based on
hypothesized scenarios based on the threats
and on analysis of vendor-identified
vulnerabilities.
Add Section 6.1.5: If the system does not
include a voter-verifiable paper audit
trail, the suitability of the system to
operate without such a function shall be
evaluated. Such evaluation shall include
review of evaluation results for software,
procedures, reported problems with similar
technology, reliability, and accuracy
evaluations to identify of conditions that
mandate the inclusion of a voter verifiable
paper audit trail.
Modify the text in 6.1.2 to refer
specifically to public networks.


Delete the “and” before “maintenance” and
between “trail” and the period, insert “,and
for system protection during storage between
elections.”
Are jurisdictions able to use uncertified
systems? If not, these two items should be
removed.

Change the phrase to "be subject to
reeaxamination."

Fix the text.



Replace words "be subject to reexamination" in both
sentences with "be required to be reexamined."



The details specified for the chip should include high level
descriptions that were used in the design, such as RTL, high-
level language functional models, gate-level netlists, and
layouts, as well as the verification environment, such as
simulation testbenches.
Delete the reference to EAL-2

The generic PP needs to be converted into a
specific PP (or more, if necessary to
address different kinds of voting systems,
such as DRE versus non-DRE) covering all
classes of voting systems.
1. A design review board shall be convened
to review the design of a voting system to
determine compliance with the election
verification standards (Section 5.1.3). The
board shall consist of independent experts
in the fields of election systems, election
administration, data security, computer
security, and other fields as deemed
appropriate by the duly designated oversight
body (e.g., Election Administration
Committee, Secretary of State).

2. The board's review shall to be based on
publicly available information.

Confer with group on addition of material on Data Integrity.




* The testing should involve a randomized data set, not just
the same ballot position or subset of ballot positions. *
Testing should confirm that there is nothing in the setup,
shutdown, or restart procedures that can cause a loss or
alteration of data.
This section specifies testing and error rates that may be
discovered using automated procedures. Error rates that
result from actual user testing can be considerably higher.
Threshholds for user error rates must also be established on
the basis of accepted industry usability standards.

[Wald's (?)] Probability Ratio Sequential
Test [see ???] using [a] binomial
distribution is recommended. In the case of
ballot position error rate, the calculation
for a specific device (and the processing
function that relies on that device) is
based on:
Define the parameters.


Use MTBF two places in this paragraph.


The probability ratio for this test is
derived from [an] exponential [ ]
distribution [see ???].
iii. Appropriate feedback and alert must be provided to voter
and election officials in the event of error that affects
availability or usability of voting device.


Needs better definition.
Re-number 6.3. The first few sections in
6.3 will have to be letters under 6.3 so
6.3.1 does not get used in the intro
discussion for the testing section.
* Recountably - the election totals should be readily and
independently verifiable if required by law, procedure, or
litigation.
Add table to evaluate password protection.



Add table to evaluate roles.



Require inspection of items 2, 3, 4, 6, 7,
and 10.
Working group should examine list for errors and omissions.

Non-applicable standards should be noted as
such, with a rationale provided [in the
public test report] for their exclusion in
non-obvious cases.

added ANSI document to standards document




added definitions for testing types

Revise requirements as in following
comments.Label tables as Table 6.3.3-1
etc.or some similar convention.




Add: 5. I




Add requirement for expert evaluation to
items 2, 3, and 4.
Add requirement to test voter privacy to
item 2.
Add requirement for usability testing to
items 2, 3, 4, and 5.
Suggest: Ensure that ballots cast through
personal assistive devices are
indistinguishable from other ballots and
cannot be traced to the voter.
Require an inspection for items 1 and 3.

Require an expert evaluation for items 1 and
3.

add check to Expert column




Require an inspection for items 1, 2, and 4.




Require usability testing for items 1, 2,
and 3.

Require expert evaluation for items 1 and 2.



Require usability testing for item 1.

Require expert evaluation and usability
testing for item 1. Other testing would also
be a good idea.


Require an inspection for items 1 through 4.



Require usability testing for items 1
through 4.


Require usability testing for item 1.


Require both testing and usability testing
of means used to activate the ballot.



Fix footnote numbering.
Remove the reference.




add check to Expert column




Require an expert evaluation for item 7.

Require usability testing for items 2, 3, 4,
5, 6, 7, 8, and 9.

change wording from "One of the …" to "The
color …"

clarify




add check to Expert column




Require an inspection for items 1, 2, 4, 6,
7, 8 and 9.
Require an expert evaluation for items 1, 5,
and 7.




Require testing of items 1, 3, and 4.
Require usability testing for items 2, 3, 4,
5, 6, 7, 8, 9, and 10.




Add disclaimer [(if applicable) to items 11,
12, and 13.

add check to Expert column




add check to Expert column




add check to Expert column




add check to Expert column




add check to Expert column




add check to Expert column




Require inspection of items 1 through 13.



Require an expert evaluation for items 2 and
8.

Require testing of items 1, 3, 5, and 8.
Require usability testing for items 1
through 8.
Require an inspection for items 1, 2, 4, 5,
and 6.

Require testing for item 4

Require usability testing of items 1, 3, 4,
5, and 6.

Alert [voter] to tasks [that] must be
completed within a time limit
Alert [voter] to problems and possible
resolutions
Require inspections for items 1 through 7.


Require an expert evaluation for item 7.


Require usability testing for items 1
through 7.



add check to testing column


add check to Expert column




Require an inspection for items 1 through 5.


Require usability testing for items 1
through 5.


If this section actually differs from
6.3.3.8 then provide unique title. Otherwise
combine the two sections into one table.

remove check from expert column and add
checks to Inspection and Testing



remove check from expert column and add
checks to Inspection and Testing
remove check from expert column and add
check to Inspection



Require an inspection in steps 1, 3, 4, and
5 (or 6,8, 9, and 10 if tables are combined)


Require usability testing for items 1
through 6 (or 6 through 11 if tables are
combined).
The testing authority shall review the
modification(s) to determine what, if any,
tests must be run to confirm a unit’s
continued compliance. If an engineering
evaluation of the change(s) is (are) not
clear whether a retest is required, the test
shall be performed [again].
Needs clearer definition.
Clarification.



The use of test fixtures or ancillary devices to facilitate
volume hardware qualification testing is [permitted].


There must also be an additional acceptance metric based
on real voter data input. This could be related to a usability
metric, or created by asking humans to replicate a set of
ballot choices. Results must then be compared to the
equipment fault rate in order to determine acceptance
threshholds.




Delete last three paragraphs of 6.4.3.




Add shock and acceleration tests here and to
section 5.4.6.
Delete first sentence of second paragraph.




Change paragraph to “COTS systems or
components must be documented by their
suppliers to have been tested to at least
the same rigor as required of voting devices
as specified hereinbelow; else, the said
COTS components shall be tested in a like
manner to any other component.”




Require environmental testing of COTS
hardware.

New para.: When preparation for storage is
required, the equipment shall be prepared
using any protective enclosures or internal
restraints that the vendor specifies for
storage.
Add new paragraph: During the environmental
testing sequence the system cannot fail more
than three times for any combination of
reasons. If the system fails any test twice
in a row it shall be rejected.

Add figures 514.3-2 and 514.3-3 and renumber
them consistent with text of these
standards.
Use degree C rather than degrees F. Use en
character for minus sign for negative
numbers rather than hyphen.

Use SI units consistently.

Allow the chamber temperature to stabilize.
Maintain this temperature for a [minimum] of
4 hours after stabilization.
See proposed change for #2.
Identify the redundant wording and eliminate
it.
Re-write of section (in process).




Lipsio to rewrite and submit for review and
comments. Defer acceptance of this section
until that is done.



Delete this clause.             The computer or
machine should be able to read and execute
the code, not a human.
Current computer science theory recognizes that no amount
of source code review and functional testing is capable of
confirming that a system is free from bugs, defects, or
nefarious code that could compromise election operations.
Therefore, it is imperative that systems include sufficient
features necessary to ensure that ballots cast are recorded
and tabulated as per their voters intentions.


Remove all exemptions for COTS product review from this
standard on the grounds that such pose a serious security
flaw. COTS products shall be presented in their entirety for
open review in the same way that vendor software is
examined.




Specify that the COTS exclusion only applies
to system components outside the trusted
subset.
Specify that inspection can be done to ATTEMPT to find
back doors, etc.
Rephrase so that it is clear that upon
modification of any component, the entire
system must undergo regression testing.

Eliminate the sections; ensure compliance
with section 4.3.11 (“Previously developed
or purchased software”) of IEEE Std 1228-
1994, “IEEE Standard for Software Safety
Plans”.
Require compliance with IEEE Std. 1012-1998,
“IEEE Standard for Software Verification and
Validation”.
Add requirement that the system conform to a
“Required Reliability Rating” of “Very high
“ (assuming the absence of a paper audit
trail ) as defined in section 4.33.2 of a
IEEE Std 982.1™-1988, “IEEE Standard
Dictionary of Measures to Produce Reliable
Software”.
Add requirement that the system conform to
IEEE Std 829™-1998, “IEEE Standard for
Software Test Documentation”.
Eliminate the exemption of COTS software
from the testing requirement.



Eliminate the exemption.




Require compliance with IEEE Std 1028™-1997,
“IEEE Standard for Software Reviews”.
Source code should be reviewed to ascertain the existence
or availability of malicious code, trap doors, Easter eggs, or
other program features that could be used to compromise or
exploit the system and/or the ballot data.



Cut the replacement rule, figures, flow
charts, etc.


Combine into section 5.6.2.4; eliminate this
section.

Delete. Not necessary for persons with
proramming knowledge. Programs and
flowcharts may look different, but have the
same results.




Either define “obvious” in an objectively
verifiable manner, or eliminate the phrase.
Make this a table.
Combine into section 5.6.2.6; eliminate this
section.

These old languages limits programming
ability. Newer languages are more
efficient,effective, and give greater
performance.
Structed programminig is required. No GO
statements allowed. Program should make
calls and return to main module.

Delete this item.
Modify the sentence to state " The vendor
shall justify to the satisfaction of the
testing authority any module lengths…."

After “mixed mode operations” add “,
including mixing of signed and unsigned data
in C or C++ or other languages where this is
tantamount to mixing modes”
Convert the language-independent items that
are testable into “shall” statements and
move the remaining items to an addendum. I
suggest an addendum that contains guidelines
for using the C Programming Language since,
assembly language aside, that is the least
fool-proof language in common use.

Modules should be constructed as to be grouped according
to functionality. The vendor shall justify any module
lengths….




Say that assertion violations should be
added to the audit log. There should be
limits placed on the number of times a given
assertion is logged to avoid exceeding
system resources, and information should not
be logged if it can compromise
confidentiality.
New para.: All functional testing shall be
performed with executable code that has
either been compiled from the reviewed source
code or has been verified to match this
compiled code. New para.: Code that is
modified as a result of testing, or for any
other reason, shall be resubmitted for
verification and qualification testing.]
Where any module affects the operation of another (such as
a module that checks data limits from another module), there
must be in-context examination.




Delete. Does not state who is doing the
system build. Vendor or Tester?
Say that the ballot must be scanned so that data from
multiple pages or both sides of the ballot are kept together in
the cast vote record.
This section should reference section 6.1.3
and the wording in section 6.1.3 should be
merged between the 2 sections.
This section should reference section 6.1.4
and the wording in section 6.1.4 should be
merged between the 2 sections.
Rewrite last sentence to read: [If the
vendor's developmental test data is
incomplete, the test agency [may require
additional developmental testing by the
vendor before conducting qualification
testing.]
Add security requirement. ¨ Vendor practices
for managing the configuration [and
security] of the system during development
and for modifications to the system
throughout its life cycle.
Reword: d. Software design[, source code,]
and specifications;
Either new sub-paragraph n. or insert after
f.: Vendor developmental history and
testing.




The TDP shall include a detailed table of
contents for the required documents, an
abstract [or summary] of each document and a
listing of each of the informational
sections and appendices presented.




All contents of the technical data package shall be submitted
in electronic formats that can be read and displayed by
widely available software applications.
Therefore, it should be required that all technical
documentation presented for certification and acceptance
testing be placed in escrow such that it is accessible
throughout all continued use of the voting system by
purchasing municipalities.package shall be
The technical data
available in its electronic format to any
U.S. citizen who requests it, at nominal
cost.




The vendor shall specify the number of
personnel and skill level required to
perform each of the following functions [for
their voting system with a breakdown as to
requirements per election jurisdiction and
per precinct]:
Personnel recommendations will also include
notes specifying that all individuals with
access to the voting system undergo a
background check and be U.S. citizens.


The vendor shall specify requirements for
the orientation and training [in both system
operation and security] of the following
personnel:
5th diamond: Releasing new versions of the
system to customers [after the new versions
are certified by the testing authority];


* Identification and notification of customers who may require
updates due to system changes that could affect proper
operations.




Requirements for configuration management
apply [to all voting systems subject to the
Standards] regardless of the specific
technologies employed . These system
components include [but are not limited to]:

c. [Uniquely] name versions.

The Quality Assurance Program shall, at a
minimum, address the topics [indicated]
below.
c. [Uniquely] name versions.
The vendor shall provide a description of
the procedures and related conventions for [
] maintaining information about
configuration management [ ].
a. Specific [hardware and software] used,
current version, and operating environment;
b. Physical location of the [hardware and
software], including designation of computer
directories[, databases,] and files;

c. Procedures and training materials for
using the [voting system].
c. [Name and contact information of the]
individual who conducted the test;
a. [Parts] and materials to be used in
voting systems and components [are suitable]
for the intended application.
b. [Special] tests, if needed, [are
designed] to evaluate the part or material
under conditions accurately simulating the
actual operating environment;
System changes that have resulted from identification of
insecure voting system components must be propagated to
all systems currently deployed. (This might be more
appropriate in the configuration management section, or a
different section under maintenance.)

a. The performance characteristics of each
operating mode and function in terms of
expected and maximum speed, throughput
capacity, start-up time, maximum volume
(maximum number of voting positions[,]
maximum number of ballot styles supported[,
and languages supported if relevant, e.g.,
DRE system), and processing frequency;


This section in the document mentions provisional
ballots and may require change to deal
appropriately with provisional ballots.
The vendor shall declare the scope of the
system’s functional capabilities, thereby
[summarizing and] establishing the
performance, design, test, manufacture, and
acceptance context for the system.
c. Required capabilities that may be
bypassed or deactivated during installation
or operation by [election officials] shall
be clearly indicated;
d. Additional capabilities that function
only when activated during installation or
operation by [election officials] shall be
clearly indicated; and
e. Additional capabilities that normally are
active but may be bypassed or deactivated
during installation or operation by
[election officials] shall be clearly
indicated.
Add e.: Test points and procedures.


The vendor shall describe the function or
functions that are performed by the
[hardware and contained] software programs
that comprise the system, including software
used to support the telecommunications
capabilities of the system, if applicable.

a. Glossary: A listing and brief definition
of all software module names and variable
names, with reference to their locations in
the software structure. Abbreviations,
acronyms, and terms should be included, if
they are [not included in these Standards or
are] uncommon in data processing and
software development or are used in an
unorthodox semantic;
c. Program Analysis: The results of
software configuration [and] algorithm
analysis[,] and selection, timing studies,
and hardware interface studies that are
reflected in the final software design and
coding.




Add d) A tabulation of all error codes and
messages produced together with the required
fix for each.
Add h) Test routines and expected outputs.

Delete HIPOs and add ERD: This overview
shall include such items as flowcharts,
[entity relationship diagrams (ERD), data
flow diagrams, and other graphical
techniques that facilitate understanding of
the programming specifications.




Add: a. Module and unit design decisions, if
any, such as algorithms [or libraries] used;
c. Identification and description of all
database entities and how they are
implemented physically (e.g., tables, disk
partitions, segment layout and distribution
of tables and indexes across drives, disk
mirroring, triggers, indexes, roles,
auditing, stored procedures, documented
logical and physical ERD, and associated
system scripts, e.g., Korn shell);
refer to comments

4) [MKS SI] units of measurement (such as
meters, kilograms, [seconds) and other units
potentially associated with the vendor's
voting system, e.g. currency such as
dollars];
2) Message formatting [and encryption];
Information submitted by the vendor is [to]
be used by test authorities to assist in
developing and executing the system
qualification test plan.
The security specification shall including a
document analyzing each of the specific
threats of section 5.1.2.3, including a
description of the defenses against those
threats, the consequences of failure of the
defense, and the available options for
recovering from such failure.
The specification shall also include a list
of all data that must be kept secret to
ensure the security of the system, and a
list of the roles of those holding these
secrets (e.g., should poll workers have a
password fo altering election data at the
precinct).
Insert a new section 7.6.1 titled “System
Design” and stating “The vendor shall
provide the documentation identified in
5.1.3.1 explaining how the system is
designed to counter and protect against the
threats identified in 5.1.2.3.
Circumstances where vendors or their agents have access to
the equipment must also be detailed in the access control
policy and procedures documentation.

The vendor also shall define and provide a
detailed description of the methods used to
preclude unauthorized access to the [ ]
control capabilities of the system itself.
b. 2) Policies and processes used by the
vendor to ensure that such protection is
updated to remain [current and] effective
over time;
b. 6) A [clear, i.e. sans unessential
jargon, and detailed description of all
activities that [are] prohibited during
system setup and during the timeframe for
voting operations, including both the hours
when polls are open and when polls are
closed.




b 4) A detailed description of the system
capabilities and procedures to be employed
by the jurisdiction to diagnose [or at least
log] the occurrence of [ ] denial of service
[or similar] attack[s], to use an alternate
method of voting, to determine when it is
appropriate to resume voting over the
network, and to consolidate votes cast using
the alternate method;
The vendor shall provide a clear, i.e. sans
unessential jargon, and detailed description
of the following additional procedures
required for use by the purchasing
jurisdiction:
This documentation shall be prepared such
that these requirements can be [effectively]
integrated by [an election] jurisdiction
into local administrative and operating
procedures.
Resequence in proper order.
Delete the second sentence. Generating test
data for the vendor should not be an ITA
responsibility.

b) Provides procedures that clearly enable
the [system administrator and election
officials] to assess the correct flow of
system functions (as evidenced by system-
generated status and information messages);
c) Provides procedures that clearly enable
the [system administrator or election
officials] to intervene the system
operations to recover from an abnormal
system state [without introducing errors in
the existing ballot counts or inadvertently
or deliberately destroying ballot records or
images;
a) Defines the procedures required to
support system [ ] installation[ ] and
readiness testing (these procedures may be
provided by reference, if they are contained
either in the system hardware
specifications, or in other vendor
documentation provided to the testing
authority and to system users);
b) Describe procedures for providing
technical support, system maintenance and
correction of defects, and for incorporating
[and testing] hardware upgrades and new
software releases [in a secure manner].

c) Detailed Examples: Detailed scenarios
[and remedial procedures] that outline
correct system responses to faulty operator
input. Alternative procedures may be
specified depending on the system state;
d) Manufacturer's Recommended Security
Procedures: This appendix shall contain the
security procedures that are to be executed
by the system [administrator and enforced by
the election jurisdiction].
d) How transmission of [election] data over
a network are performed (DRE systems[ and
others] where applicable);




b) Size [and location];

c) Organizational affiliation (i.e.,
[election] jurisdiction, vendor) of
qualified maintenance personnel.
d) Recommendation regarding secure storage and retention
of ballot materials (whether paper or electronic) and other
audit records.


a) Recommended number and locations of spare
devices[, ] components[, or supplies] to be
kept on hand for repair [or maintenance]
purposes during periods of system operation;

b) Recommended number and locations
[(distance)] of qualified maintenance
personnel who need to be available to
support repair calls during system
operation;
Rename the section to, perhaps, “History of
Voting Equipment in the USA”




Such ballots [were] then counted by hand,
and a tally of votes [ ] delivered to the
election authority.
A growth in the number of elective offices
and public issues, in the numbers of
political parties and candidates offered,
and in the number of voters led to the
development of [ ] mechanical voting
device[s that] would automatically tally the
votes – thereby, it was thought, eliminating
human error or chicanery while speeding the
counting process. The mechanical device[s]
require[d], however, that choices be arrayed
on a fairly large panel and that voters pull
down levers corresponding to their choices.
The levers trigger individual counters
mounted in the sealed back panel of the
device that are, at the end of the day, read
by election officials.
IBM punchcard technology gave rise to two
types of punchcard voting systems. The
Votomatic device requires voters to insert
an IBM punch card behind the spine of a
booklet that lists the choices and, with
each turn of a page, advance the voter
across the columns of the card. Voters
indicate their choices by inserting a stylus
in a corresponding hole in the spine of the
booklet[ that] dislodges a prescored chad.
After the card is removed from the booklet,
an electronic card reader can then rapidly
read the resulting punchcard ballot. The
[card] readers feed into a computerized vote
tally system. The [alternative] Datavote
device lists the voter’s choices on the
punchcard itself (usually on both sides of
multiple cards). The voter inserts the card
in what amounts to a keypunch device that,
when a lever is depressed next to the
voter’s selection, cuts a hole along the
side of the card. Again, the card is
[mechanically] read and the votes
[electronically] tallied. [ Reading the]
punchcards can occur either in a central
location to which ballots are delivered, at
the polling place after the close of polls,
The IEEE began its effort [as an independent
body] in 2001, giving project authorization
for IEEE P1583 at the June 2001 meeting of
the Standards Board.
These mechanical lever machines[, which by
no means eliminated election chicanery,]
began to be replaced in the mid 20th Century
by a variety of contemporary technologies.

Substitute current version of section 1.1
for Abstract.
... standard and is not itself a mandatory
or regulatory requirement.
Do you want me to take a crack at rewriting
the abstract? Also, keywords are incomplete,
e.g., DRE, direct recording equipment,
tabulator.
It includes the equipment used to display
and cast a ballot and for precinct ballot
tabulation where the voter [or an election
official] inserts a voted paper based
ballot.
Change to read: This includes preparation of
the equipment for election use, accuracy
testing of the prepared equipment,
transporting of the equipment to and from
the polling site, storage of the equipment
between elections and use of the equipment
by [election officials] in supporting its
use in the election functions.
add: universal access,

Therefore, the information presented below
provides guidance for the presentation of
ballots[ ]. These principles should be used
in conjunction with the general information
presentation principles in Section 5.3.3 to
make ballots more effective from a usability
point of view [for computer displays of
ballots]. It is recognized that State laws,
in many cases, govern ballot design.

Add bullet: Candidates should be listed in
some random order, not simply alphabetically
or by party affiliation, e.g, Republicans
should not always be listed first.

Example: FTP_ITC.1 -- "healthcare" should be
changed to "voting system"
A purpose statement. Is this a requirement
and how do we test?

Change title to read: Generic [Security]
Protection Profile For [Voting Systems]
Define all conventions prior to their usage.
Rewrite and reformat the annex based on
those definitions and formatting used in the
rest of the Standard. Remove or substantially
reduce the jargon that make this annex read
like the mumbo-jumbo of some cult. Submit
both the new Annex C and the revised Annex D
(currently Annex C) to the P1583 security
working group for independent review.
Requires discussion by entire working group
to determine proper EAL level, likely EAL4
with components from EAL5 and EAL6. Rewrite
Annex to match EAL level determined by
working group. See attached notes.
C.7.3.1 Since the audit trail must
necessarily be turned off during voting in
order to preserve ballot privacy, additional
security assurances must be provided in
order to maintain ballot data integrity.
Such assurances could include the addition
of an voter verifiable audit record to the
voting system.
Analyses required for assurance should
include covert channel analysis to the
extent covert channels can be used for
signaling to bypass vote privacy
requirements. This can be accomplished by
adding a portion or a modified version of a
covert channel analysis component to the PP
assurance requirements.
Table 4 Security Objectives and Functional
Requirements Mapping

FAU_GEN.1 Audit data generation
This component outlines what data must be
included in audit records and what events
must be audited. This component traces back
to and aids in meeting the following
objective: O.AUDIT_RECORD that supports
P.ACCOUNTABILITY and P.AUDIT_CONTROL
policies.

FAU_GEN.2 Election Official identity
association
This component ensures that events recorded
in the audit trail are not associated with
individual Voter identities. This component
traces back to and aids in meeting the
following objectives: O.AUDIT_RECORD,
O.ENTITY_AUTHENTICATION, O.ROLES, and
O.ELECTION OFFICIAL_AUTHENTICATION that
support P.VOTER_ANONYMITY policy

FAU_SAA.1 Potential Violation Analysis
This component monitors the TOE and sends an
alarm when a security violation has
occurred. This component traces back to and
aids in meeting the following objective:
O.EVENT that supports P.ALERT policy.

FAU_SAR.1 Audit Review
Revise the assumption




Revise the assumptions accordingly.

Revise the assumption




Revise the assumption




Revise the assumption




The (potential) threat agents implied in
5.3.2.1 include (malicious) vendors, their
employees, suppliers, and the suppliers'
employees; perpetrators who can access the
delivery process; providers of services in
the delivery process; election officials
(non-polling place), election officials
(polling place), voting system technicians,
intruders into voting machine storage
facilities, voters, and eavesdroppers
(listening to compromising electromagnetic
emanations or inter-equipment
communications).
Replace with "Threats To Be Addressed By The
Operating Environment (Non-IT Environment)"
Eliminate the exemption.




Provide a table showing how the threats in
C.3.3.3 track the threats in 5.3.2.1

Revise the statement taking other comments
on this section into account.
Revise the statement accordingly.


Revise the statement accordingly.




Revise the statement accordingly.

Revise the statement accordingly.

Revise the statement accordingly.




Revise the statement accordingly.

Either delete the statement or revise it to
make sense. For example, no session with a
voter should automatically terminate, but a
session with an administrator might
reasonably do so.
Append the text of the comment to the
statement.

Clarify or delete the objective.

Revise the statement.
Delete everything after “natural language
format.”




Revise the statement accordingly.




Revise the statement accordingly.




Revise the statement accordingly.



Add an objective to the effect that the TOE
must preserve vote anonymity with or without
collusion of the voter to bypass it.

Clarify or delete the objective.
Delete



Revise the objective accordingly.
Delete “and election data.”




Rewrite in accordance with voting system
application and security requirements as
reflected in the remainder of the draft
standard.



FMT_SMF.1 Specification of Management Functions




Add the following Sections:

C.5.1.5.3 Specification of Management
Functions (FMT_SMF)
C.5.1.5.3.1 FMT_SMF.1 Specification of
Management Functions
C.5.1.5.3.1.1 FMT_SMF.1.1
The TSF shall be capable of performing the
following security management functions:
[assignment: list of security management
functions to be provide by the TSF].

C.5.1.5.3 Specification of Management Functions
(FMT_SMF)
C.5.1.5.3.1 FMT_SMF.1 Specification of Management
Functions
C.5.1.5.3.1.1 FMT_SMF.1.1
The TSF shall be capable of performing the following
security management functions: [assignment: list of
security management functions to be provide by the
The TSF shall provide authorized election officials with
the capability to verify the integrity of stored TSF
executable code.
Provide assurance requirements appropriate
to the requirements, threats, and
objectives, after they have been revised and
become properly stated.




Delete "ATE_IND.2 Independent testing"
AGD_ADM.1 Administrator guidance AGD_USR.1
User guidance
ATE_COV.1 Evidence of coverage
ATE_FUN.1 Functional testing                Add
"ATE_IND.2 Independent testing"
Add the following Sections:

C.5.2.1.2 International Interpretation #3
The configuration list shall uniquely
identify all configuration items that
comprise the TOE.


C.5.2.1.2 International Interpretation #3             The
configuration list shall uniquely identify all
configuration items that comprise the TOE.
Change to "The installation, generation and start-up
documentation shall describe all the steps necessary for
secure installation, generation, and start-up of the TOE.
"
Change from:

C.5.2.2.2.1.1 ADO_IGS.1.1C
The documentation shall describe the steps
necessary for secure setup, ballot
generation, and opening and closing of the
polls, start-up of the TOE.

to:

C.5.2.2.2.1.1 ADO_IGS.1.1C
The installation, generation and start-up
documentation shall describe all the steps
necessary for secure installation,
generation, and start-up of the TOE.
Upgrade the assurance requirement in
accordance with this comment and the
previous comment (SK-35).
Make tables 7.5" wide and column widths:
Column 1 - .6
Column 2 - 3.7
Coumn 3 - .7
Column 4 - 2.5
Add another Annex that sorts Annex D by
P1583 sections.




Big project, but go through and add the
specifications that are new with this
standard.



Verify and correct any cross referencing
errors,
This must be done prior to the next draft -- insufficient time to
provide content in this comment.
leave out Increased air discharge from 15 to
25 kV and replcae by NC




Change font to Time Roman 10 pt.




Vince Lipsio to rewrite sections 5.6 and 6.6
and submit for review. Rewrite Annex C. Fix
problems listed below, use metric units
consistently, and correct typos. Then submit
a v 5.1 for final ballot.




See comments below on particular sections.
This will require global changes in the document. It is not
amenable to line-by-line editing.




Add these principles explicitly at the beginning
and amend generally to conform.
Modify draft as appropriate.




Make all statements about how a system should
be constructed advisory. Add a set of
performance criteria at the beginning.




All text related to integrity should be replaced
by a Protection Profile and language referencing
the Common Criteria, including the choice of
testing labs.




All text related to privacy should be replaced
by a Protection Profile and language referencing
the Common Criteria, including the choice of
testing labs.
All text related to usability should be replaced
by a definition of tests according to the
comment.



All text related to reliability, safety, and quality should
be replaced by references to the
appropriate standards, such as those listed in
the comment.




Need statement abut System Performance.
For 2002 Technology Standards and Machine
Performance,   the processor of a voting
machine should be greater than a 386. A
Pentium level would be more appropriate.
The Cook County Board is required by Illinois
State law to provide provisional voting on Election
Day. Even though other states without provisional
voting may use this document, it would be
beneficial if provisional voting was addressed.
The following references do not cover all the
sections in the document that would be affected
by DRE provisional voting, however, they are
believed to be the most important sections that
would need to be amended.

Sections in the document that mention provisional
ballots and would require coordinated change are
identified by comments Gough-008 to Gough-
(sections 4.4.1, 4.5, 4.5.5, 5.1.3.2.5.d, 5.1.3.5.6.a,
5.2.1.2, 5.2.2.2, 5.5.3.2.a, 5.2.7.b, 5.6.3, 5.6.5.1,
5.6.5.2, 5.6.6, 5.6.8, 5.6.8.2.b, 5.6.8.3.3, 5.6.9,
5.6.9.2, 7.6.2). Additional sections that may deal
with provisional balloting and may require change


These inconistencies should be corrected.
     Proposed Resolution
(by the Chair on each comment submitted)
NC - See Gough-003. Recommend a task group be
appointed to develop a wording proposal for this issue.
RGHBH001
NC - See Gough-003. Recommend a task group be
appointed to develop a wording proposal for this issue.
RGHKC003




RGHKC002




RGHBH002
RGHKR001




RGHBH003
RGHKR002
RGHBH004
RGHKR003




RGHBH005
RGHKR004
RGHBH006
RGHKR005




RGHKR006
RGHBH007
RGHKR007




RGHKR008




RGHKR009


RGHKR010



RGHBH008
RGHBH010


RGHKR011




RGHBH009




RGHKR012
RGHKR013




RGHKR014
RGHKR015




RGHBH011




RGHKR016
RGHBH012
RGHKR017
RGHKR018
NC - See Gough-003. Recommend a task group be
appointed to develop a wording proposal for this issue.




RGHBH013


RGHMD001
RGHBH014




RGHSL041




RGHSL040


RGHSL039

RGHSL044
RGHBH019




RGHKC014
RGHSL001
RGHKC015
RGHSL002


RGHSL005




RGHSL003

RGHSL004

RGHSL006




RGHBH020
RGHSL042
RGHBH015
RGHSL007




RGHSL007


RGHSL043


RGHSL045


RGHSL046

RGHSL008




RGHSL047
RGHSL009
RGHSL010


RGHSL011
RGHSL012




RGHKC011
RGHBH016


RGHSL014




RGHSL013




RGHKC012
RGHSL015

RGHSL016

RGHSL017
RGHBH017




RGHSL018
RGHBH018




RGHSL019




RGHMD010
RGHMD003




RGHMD002




RGHMD004
RGHMD009




RGHBH021
RGHBH022




RGHBH023
RGHKC016
RGHBH024
NC - See Gough-003. Recommend a task group be
appointed to develop a wording proposal for this issue.
RGHBH025




RGHBH026




RGHBH027


RGHBH028
RGHBH029
RGHKR019




RGHKR020




RGHKR021
RGHMD008
RGHSL020




RGHSL021
RGHSL022
RGHSL023




RGHSL024


RGHSL025




RGHSL026
RGHSL027




RGHSL028




RGHSL029




RGHSL030




RGHSL031




RGHSL032
RGHSL033


RGHSL034




RGHSL035




RGHSL036
RGHSL037




RGHMD005
RGHMD006




C - Agreed that real human data entry is required to
supplement automated testing. The first paragraph has been
changed as follows: The use of test fixtures or ancillary
devices to facilitate volume hardware qualification testing is
encouraged. These

Further comment - Change is incorrect -- should read
...."human interaction will uncover anomalies"... not will NOT
uncover anomalies!!!
RGHMD007




NC - This is the testing section and it is not appropriate to
include standards requirements here. Although there is no
degree of testing or source code inspection can 100%
assure perfect system operation, testing and inspection
should be as complete as

Further comment - Since software testing in inherently
insufficient, there should be a comment here regarding the
flawed nature of source code review. The first sentence
NC - Only unmodified COTS is exempted. This is a drastic
change that permeates the spec and cannot be considered
at this time. Also, if required, a vendor cannot control COTS
source availability which would also limit vendor choices in
system design.

Further comment - Unmodified COTS is not exempt from
serious security flaws, as evidenced in the continual update
patches that must be downloaded for Microsoft operating
systems, for example. The exemption for COTS products
NC - Review for malicious code is a standards requirement
issue and not a definition in the testing area. It should be
covered in section 5.6.

Further comment - Source code review is the appropriate
place to check for malicious code. This must be covered
here as well as in Section 5.6 (which refers to section 6.6
throughout).
C - The comment contains a valid objection. The wording
has been changed as follows: i. Excluding code generated
by commercial code generators, is written in small and easily
identifiable modules that are constructed to be grouped
according to functional.

Further comment - How is code generated by commercial
code generators identified? The remainder of the correction
is acceptable, but the commercial code generation clause is
NC - The suggested wording seems to refer to low level
modules within a functional modules/subsystem. This
section is discussing new or changed functional modules
interaction with the other system functional modules or
subsystem components.

Further comment - The comment pertains to any new or
changed functional modules that interact with other modules
or components, and still needs to be addressed in terms of




RGHMN002




RGHMN003
NC - Out of scope. Add to list for consideration in future
documents.

Further comment - Comment stands, escrow should be
required, not recommended.




RGHMN012




RGHMN013




RGHMN014
NC - Outside scope of the TDP

Further comment - Maintenance must include configuration
management information propagated to the purchasers
where changes could affect security or reliability of the voting
system. Comment should be reconsidered by working
group.
RGHMN015




NC - See Gough-003. Recommend a task group be
appointed to develop a wording proposal for this issue.
RGHMN004




RGHMN005
RGHMN006




RGHBH030




NC - Covered in 5.1.3.2.4

Further comment - 5.1.3.2.4 does not also mention specific
times when certain access shall be disallowed. Comment
should be reconsidered by working group.




RGHMN007
RGHMN008




RGHMN009
RGHMN010



RGHMN011
NC - Outside scope of this standard

Further comment - Section 7.9 pertains to system
maintenance and therefore requires material on ballot and
audit storage retention and maintenance. Comment should
be reconsidered by working group.
NC - specific language to make change not provided. Agree
that detailed comparison is needed and any volunteer to help
is appreciated.
AC

Action for editing committee.


Action for editing committee.

Action for editing committee.



Action for editing committee.

Action for editing committee.




NC - Comments proposals will be reviewed
when submitted.


Action for editing committee.
NC - Return to commentor for specific
wording suggestion for the draft.




NC - Return to commentor for specific
wording suggestion for the draft.




NC - Return to commentor for specific
wording suggestion for the draft.


NC - Return to commentor for specific
wording suggestion for the draft.


Action for editing committee




NC - Not a comment. Editorial error in entry.
NC - Return to commentor for specific
wording suggestion for the draft.
NC - Return to commentor for specific
wording suggestion for the draft.




NC - Return to commentor for specific
wording suggestion for the draft.




NC - See Annex C "Protection Profile"




NC - See Annex C "Protection Profile"
NC - See Sections 5.3 & 6.3.




NC - See relevant sections for reliability,
safety and quality. Comment is already
reflected in existing text.




NC - Return to commentor for specific
wording suggestion for the draft.




NC - Needs proposed wording change.
NC - Recommend a task group be appointed to develop a
wording proposal for this issue.




Insert into comment list
Action for editing committee.
Reference Information   Reference Information
C - Definition chaned as follows:                                 The final sentence was not added in
                                                                  V5.0. Need to correct as you said you
93.80. Recall Issues (with Options): The A process that           would.
allows voters to remove their elected representatives from
office prior to the expiration of their terms of office. Often,
the recall involves not only the questio
NC - Used in this context the existing definition should stay      Resolution is incorrect. Either the word
the same. Ballot Scanners used in the polling place can only       "marksense" should be removed from
read the ballot style and layout that the voter uses to cast the   the definition or the words "or voter
ballot. There is no way for a scanner to differentiate a ballot    verifiable" should be added as per
ver                                                                comment, in order that scanners may be
                                                                   used to confirm all forms of paper
                                                                   ballots, not just those that are
NC - Section 4 does not give requirements but rather         The diagram is unclear, inconsistent and
describes the voting system generically.                     confusing. Need to correct as per my
                                                             comment and proposed change.
In standards the text is normative and illustrations and
diagrams are illustrative and secondary to the text. While
diagrams should accurately illustrate the t
NC - Out of scope for this document   The document addresses tallying of
                                      absentee votes incorrectly and allows for
                                      the creation of "ghost precincts." Since
                                      the document has gone out of scope in
                                      this section, the bullet should be
                                      corrected to address the tallying
                                      appropriately or the entire se
NC - Both out of scope and not possible until P1622   Since assessment of these reports is
completes its work.                                   critical to the security controls applied to
                                                      the voting system, it is inappropriate for
                                                      vendor-proprietary reports to be used.
NC - This section is descriptive and not giving specifications. Since the paragraph specifies that the
                                                                integrity of the primary file must not be
                                                                affected, it is appropriate to add the
                                                                caveat that data in any alternate files
                                                                must be consistent with that in the
                                                                primary file.




NC - Out of scope. This is determined by the relevent            The lack of any specification regarding
election authority, such as NASED or the individual state        updates and configuration management
officials.                                                       is a serious security flaw that must be
                                                                 addressed by the standard.
NC- Not appropriate place   Verification materials must be retained in
                            order to provide evidence of appropriate
                            certification testing. Comment should be
                            moved to appropriate section.
3. NC - goes to policy 4. Add "3. 3. The need for    "impacts" is the wrong word choice for
anonymity of voter ballot impacts many traditional   the new phrase and is misleading.
                                                     "precludes" (or reduces or removes) is
forms of auditing commonly used for other            appropriate.
electronic systems (such as ATMs in banks). "
C-Added as recommended                               the word "totals." was omitted from the
                                                     end of the sentence in the draft.
C- Wireless can be deployed securely - changed          It is not agreed that wireless can be
to add "                                                deployed securely. This must be taken
                                                        up by the entire working group.
Wireless connectivity, both in the development
and deployment of election systems poses
significant security risks, such that it is necessary
require additional security measures to specifically
NC - wording is not correct                             Access violations and violation attempts
                                                        are an important part of the security
                                                        process and must be noted in the audit
                                                        trail.
C- add "while maintaining ballot integrity"   Phrase was not added in draft v5.0
NC - Specifications are not the place to add requirements.     This comment is now requested to be
New specifications should be proposed as additions to          placed at 5.1.3.4.1 d)
Section 5.

NC- if operation of an override is identifyable then           Override feature could be exploited.
tampering can be identified. If an override is                 Comment must be addressed by
                                                               working group.
needed to correct a DOS attack on the device (ie
take a voting location out of service by activating
tamper Add to document as should last paragraph in
C - VW - protection, then itsecond tobe identifyable.           The wording added in 5.1.3.5.1 is overtly
section 5.1.3.5.1                                               vague and is intended to permit vendors
                                                                to provide internal audit trails that can
WFW - Adding the VoteHere approach as generically               not be independently recounted or
described, may or may not be acceptable as another method verified. This issue needs to be
of voter verification. We are getting into areas that I believe discussed in the working group.
rightful
NC - specific language to make change not            OS and DB audits are insufficient to
provided. Current language is adequate.              provide the ensurances identified in this
                                                     section. The paragraph is inadequate
                                                     and incorrect and must be reworked.


NC - a "voter initiated deletion" would not add to   A voter initiated deletion may still create
the storage requirements in a significant way.       an additional record, depending upon
                                                     implementation. Sufficient deletions
                                                     could impact data requirements
                                                     adversely.
C- Added"All keypresses of any single or            Add "time and logic bombs" Also need
combination of voters and/or administrators shall   to address data flow analysis to validate
                                                    these controls.
be precluded from adtivating any software or
firmware process other than those directly
pertaining to the election aspect being used."
NC - The time/date stamp is part of the Audit log    Comment was misunderstood. Secure
and is required to be secured from loss by current   time/date stamping protocol must be
                                                     used in addition to human-readable
language                                             format.
C as provided                                        replace "practical" with "possible" as
                                                     provided
NC - wireless connections can be deployed with   Comment must be discussed by working
solid security. These issues are dealt with in   group. Wireless poses significant
                                                 security risks.
other sections.
NC goes to design                                Correctness, accuracy and integrity are
                                                 performance.
NC - if used should be identified                     Antennas should not be used since they
                                                      pose a significant security risk.




C Added new para d. "d. The vendor should          The jurisdiction must also report
provide appropriate and timely reporting to the    breaches if they were the observers.
appropriate jurisdiction of any observed breaches
or breach attempts during the election setup,
actual election, and post election and canvass for
all systems of th
NC goes to design                                        Goes to auditability, reliability and
                                                         recountability which are all performance.

NC - The responsibility for determining which record     The resolution indicates a lack of
superceeds the other is for EAC/States and case law to   understanding of the concept of voter
decide.                                                  verified balloting and needs to be
                                                         resolved by the working group.
NC 0 Not achievable by any known means. Error Since this is a performance standard,
rates a                                       the error rate must be disclosed so that
                                                    jurisdictions can know when an election
                                                    must be recalled on the basis of
                                                    insufficient data to determine the winner.
NC Requires voter to declare intent to undervote   The equipment must be able to
and that is a voting jurisdiction issue not        ascertain the difference between a
                                                   deliberate undervote and a system error
equipment issue.                                   that caused vote lossage.
NC goes to design                                        Unalterability of media for critical data
                                                         should be a performance requirement.




NC - The responsibility for determining which record     The resolution indicates a lack of
superceeds the other is for EAC/States and case law to   understanding of the concept of voter
decide.                                                  verified balloting and needs to be
                                                         resolved by the working group.
C added text"event causing the "            Comment was misunderstood,
                                            resolution is incorrect.




???????????? Someone was researching that   Need to insert correct number.
number.
NC wording not provided   5.2.7 a. 3) Precinct based scanners
                          should be available to produce reliable
                          and accurate feedback to voters using
                          mark-sense ballots as to their
                          correctness in preparation (including
                          alerts to undervotes and overvotes).
NC - out of scope for this section on voter interfaces. Other   Recountability is a requirement of
users of voting systems will be addressed in a later revision   election law and must be addressed.
of the standard.
NC - out of scope for this section on voter interfaces. Other   Need for poll workers to ensure that the
users of voting systems will be addressed in a later revision   voting device is not tampered with
of the standard.                                                should be addressed here in a footnote,
                                                                or elsewhere in the standard.
NC - Requirement is not clear and it is not clear that it could   Comment needs to be addressed by
be implemented as stated.                                         working group.
NC - T-Coil wireless coupling is well established and very     Wireless coupling may be appropriate
important for a segment of the population with hearing loss.   for non-voting applications but may void
                                                               privacy requirements here. This should
                                                               be discussed.




NC - Need for privacy already a general requirement.           Privacy screen may need to be different
                                                               for disabled voters. This should be
                                                               separately addressed.
NC - out of scope for this section on voter interfaces. Other   Insufficient testing for language options
users of voting systems will be addressed in a later revision   has been noted in equipment failures in
of the standard.                                                actual use. This must be addressed.
NC - Addressed in item D042 above                               Comment was misunderstood. Colors
                                                                like "green" could be misconstrued to
                                                                mean the "green party" so this should be
                                                                addressed.
NC - Already addressed in 5.3.8, Item 5                           Ballot spoilage needs to be clarified,
                                                                  since it may not be possible to "reverse"
                                                                  a paper-ballot selection, hence
                                                                  "spoilage" would need to be invoked.
                                                                  5.3.8 5) does not sufficiently address
                                                                  this issue.
NC - out of scope for this section on voter interfaces. Other     This also must be addressed with the
users of voting systems will be addressed in a later revision     spoilage issue above.
of the standard.
C - Add the comment that "If voter verification via a paper       The change does not appear in draft
record of the ballot cast is provided it shall inform the voter   v5.0
that their votes were properly registered and contain a
summary of the ballot choices the voter selected, that the
voter can veri the footnote is changed to read:
C - The end of                                                    The change does not appear in draft
                                                                  v5.0
The design should make it clear where and how to vote, how
to change a vote which has not been cast and the system
shall provide feedback that the vote was accepted by the
system. The guidance in this sec
C - Changed in incoporate time-limit indication. Exemption   The time-limit indication may be
item not included as it is up to local policy.               confusing or troubling to some voters. If
                                                             the machine "times-out" though, this
                                                             could be very upsetting. What I meant
                                                             by the comment was that the amount of
                                                             time a voter might need to prepare a
                                                             ballot may exceed the regula
NC - Suggested wording too vague. I agree some time           Use time similar to other sections -- 24
specification should be added to cover units that use battery months?
backup RAM for non-volatile memory. However, need to
define a concrete time period. (24 hours?, 2 weeks?)
NC - This is covered in section 5.1.1 as shown by the          COTS products themselves should be
following excerpt: "COTS products require updates due to a     subject to thorough evaluation, not just
detected security breach or vulnerability. The voting system   their updates. COTS provide a
vendor must provide a method to assess the impact of           significant security risk. This must be
COTS updates on the votin                                      addressed by the working group.
NC - The individual unit report is report is already covered by 5.6.10 missing from pdf document???
item C).

C - Section added as follows: k) Ensure that extracted or         5.6.10 missing from pdf document???
duplicated nformation is identical to that on the original Cast
Vote Record storage medium.
NC - While there is agreement that including language for   Corrections should be as follows: a) ...If-
the points identified would be an improvement, specific     Then-Else, While, Do-While, ... f)
language changes that would be directly usable have not     Concurrent process flow should be
been suggested.                                             allowed only if detailed description of
                                                            need has been documented. g)
                                                            Assignments within branch tests should
                                                            be avoided (ex. use of if(a=b) sh
C - suggested r) is included in l) tabulation of overvotes and   Need to add: The TDP must reflect the
undervotes. The following section has been added: r)             implementation of these voting
Casting of a totally blank ballot (if permitted)                 variations in such fashion that it is
                                                                 readily discernable that a) all votes are
                                                                 registered to the appropriately selected
                                                                 candidate(s) and b) tallies reflect the
                                                                 algorithm properly (as in
C - Wording changed as follows: f) Allow the voter to vote a Change was not reflected in draft v5.0
new ballot in the event of a detected exception or, if the
equipment and state law permit, allow the voter to submit the
ballot ‘as is’ without correction appropriately tabulating all
validly v
c -as indicated   The draft is incorrect, and does not
                  reflect the changes as specified. "shall
                  be reexamination" in two places should
                  be replaced with "be required to be
                  reexamined"
C- Revised wording: Industry recognized third party class             Resolution makes no sense. This
libraries (e.g., Microsoft Foundational Class (MFC) C++               comment needs to be addressed. There
libraries)                                                            is still NO material in this section on
                                                                      Data Integrity and this is a serious
C- Revised wording: Software that operates on ballot                  omission. makes no sense with respect
                                                                      Resolution
printers, vote counting and verification devices, memory              to comment. Comment needs to be
devices used for results storage and/or ballot activation and other   addressed to reflect appropriate testing
hardware typically installed at precinct, early voting or in-person   procedures.
absentee
NC - The error rates in this section do not include human    Resolution is incorrect. For paper
error where votes are inadvertently not cast as intended. It tabulators, a predefined set of ballots
refers to machine introduced errors. It also does not assume are typically read. For DRE-style
an automated process. For paper tabulaors, real ballots are machines, it is my understanding that
read.                                                        testing is typically NOT performed on the
                                                             ballot casting device, only the ballot
                                                             recording device. In any e




NC - The suggested wording relates to the functional                  Resolution makes no sense. There is no
requirement of availability (or failure) indication not to the        section 5.6.9.1.3d in the V5.0 draft.
section subject, meeting availability standards. The concern          5.6.8.3.1d pertains only to the "failure of
is already covered in the statement of 5.6.9.1.3d "d) A means         the main power supply" and not to the
of identifying                                                        actual unavailability of the system due to
                                                                      other equipment malfunctions.
No change - Recountabilitiy is out of scope for this section. It Recountability still needs to be
addresses testing for Section 5.3 which addresses voter          addressed, since it is a component of
interfaces and not those of other users. These will be           election law and impacts usability.
addressed in a future revision.




NC - See response to corresponding comment on 5.3.10.2.        Comment needs to be addressed by
                                                               working group.
  IEEE P1583
BALLOT COMMENT SUBMISSION FORM

          Date: 9-30-03 P1583 Draft 5.0 August 2003
                Document:
           Commenter Clause/ Subclause        Paragraph           Type of
          and Number                       Figure/ Table         comment
                                                                 (General/
 #                                                               Technical/
                                                                  Editorial)
     1    Lipsio-5D            1.1                         3 E

     2    selker-1002          1.1                         3 E

     3 Corry-006               1.2                          E


     4 Corry-007               1.2       2nd sentence                T



     5    Lipsio-5E             2                           E


     6    Lipsio-5F             2                           E

     7    Lipsio-60             2                           E


     8    Lipsio-61             2                           E

     9    Lipsio-62             2                           E


     10   Lipsio-63             2                           E

     11   Lipsio-64             2                           E


     12   Lipsio-65             2                           E

     13   Lipsio-66             2                           E



     14   Lipsio-67             2                           E


     15   Lipsio-68             2                           E
16   Lipsio-69            2                         E


17   Lipsio-6A            2                         E


18   Lipsio-6B            2                         E


19 bronaugh - 001         3   Definition 8              E
20 Corry-008              3                     8       E




21 Corry-010              3                    48       E


22 Corry-012              3                    73       E



23   df2                  3   No. 58                    E

24   Lipsio-6C            3                         E

25   MercuriD50 - 006     3   definition #93            E
     (formerly mercuri-
     011)


26   HD-003               3                    75       G

27   HD-004               3                    98       G

28   HD-001               3                     8       G

29   HD-002               3                    59       G
30   PPLX - 002               3 Section 3.          G
                                Definition for
                                Directr
                                Recording
                                Electronic




31   wfw - 002            3             59       General

32   MercuriD50 - 070     3    Definitions #21      T
     (formerly mercuri-
     162)
33   Adler-016   3   T




34   Adler-017   3   T

35   Adler-018   3   T




36   Adler-019   3   T
37   Adler-020   3                       T




38   Adler-021   3                       T



39 Corry-009     3                44     T



40 Corry-011     3   66, last sentence   T




41   Dill-10     3   Def #98             T


42   Dill-11     3   Def #30             T

43   Dill-28     3   Def # 42            T




44   Dill-6      3   Def # 17            T

45   Dill-7      3   Def #26             T
46   Dill-8        3    Def #64            T

47   Dill-9        3    Def #73            T
48   selker-1003   3                  98   t




49 VCW-01          3    def 98             T




50   PPLX-004          3 Section 3.        T
                         Paper Based
                         Voting system.
                         Definition #
                         64.
51   PPLX-005   3   Closed and Open   T
                    Primaries:
                    Definitions 25
                    and 61.
52 vwilliams - 27   3    References        T




53   PPLX-003           3 Section 3      T and E
                          Firmware.
                          Definition #
                          49.
54   PPLX-006             3 Section 3.            T and E
                            Recall Issues
                            (with Options).
                            Definition 82




55   Dill-5        3       Def #4


56 berger - 001    3.1                               E



57 Corry-013       3.1                               E




58   Lipsio-55    3.101                       G


59   Lipsio-58    3.15    3.15 and elsewhereG



60   Lipsio-59    3.37    3.37 and elsewhereG
61   Lipsio-5A         3.42     3.42 and elsewhereG



62   Lipsio-6E         3.49                           E




63   Lipsio-6F         3.64                           E

64   Lipsio-70          3.7                           E

65   Lipsio-71         3.72                           E



66   PPLX-007                   4 Section 4           G
                                  System
                                  Description
67   PPLX-012                 4.6 Documentation       G
                                  Section 4.6




68   PPLX-037     2.6.4.1        5.6.4.1 Error
                                 Messages             G
69
     Sklein-051         3.0      Item 26              T


70
     Sklein-052         3.0      Item 38              T
71
     Sklein-055         3.0        All                 T




72   Lipsio-76      3.54, 3.93                         E

73   Lipsio-77     3.94 and 3.95                       E




74
     Sklein-049         4          All                 T




75   Aragon - 06       4,4,1       last bullet point   T




76   Dill-12            4.0        first sentence      E



77 Corry-014            4.0        Entire section      T
78 Corry-015              4.0      First sentence        T




79   MercuriD50 - 007     4.1      diagram               T
     (formerly mercuri-
     014)




80   Lipsio-5B                                       draft
                          4.2 4.2 and throughout the G




81   RGH 002              4.2                            G


82 Corry-016              4.2                            T
83   Lipsio-5C                      4                        G
                                4.2.1 .2.1 and throughout the draft




84   PPLX-008           4.2.1             Sections 4.2.1         G
                                          through 4.2.4

85 Corry-017                     4.3      First sentence         E

86 MercuriD50 - 008             4.4.1     bulleted item        General
   (formerly mercuri-
   023)
87 MercuriD50 - 009             4.4.1     bulleted item        General
   (formerly mercuri-
   025)
88 Aragon - 07                  4.4.1    last bullet point       T




89 Corry-018                    4.4.1     15th bullet point      T
90   PPLX-009             4.4.1           Precinct                  T
                                          Voting.
                                          Section 4.4.1.




91   df3                          4.4.2                             E

92   MercuriD50 - 010             4.4.3   Entire subsection       General
     (formerly mercuri-
     026)




93 Corry-019                      4.5.1                         E
                                          5th para., 2nd sentence




94   df4                          4.5.1   Paragraph 6               E



95   Lipsio-78                    4.5.1   para. 5                   E




96   RGH 003                      4.5.1   7th paragraph             E



97   Lipsio-0D                    4.5.1   para. 4             T
98    PPLX-010          4.5.1               Polling Place             T
                                            Reports.
                                            Section 4.5.1




99    MercuriD50 - 011 4.5.1, 4.5.2, 4.5.3, First sentence in       General
      (formerly mercuri-      4.5.4         each section
      027)


100   RGH 004                   4.5.1.      5th paragraph             E




101   df5                       4.5.2       Paragraph 3               E



102   PPLX-011          4.5.2               Precinct                  G
                                            Reports
                                            Section 4.5.2




103   RGH 005                   4.5.3                                 G


104   HD-005                    4.5.4       1st bullet                T




105   Lipsio-0E                 4.5.4       Bullets 1 - 5       T
106   MercuriD50 - 012     4.5.5   Add to end of section         T
      (formerly mercuri-
      030)


107 Corry-020              4.5.5                         T
                                   2nd para., last sentence



108   Lipsio-0F            4.5.5   Para. 1                 T




109   Lipsio-73             4.6    Bullet 5.10                   E



110   Lipsio-74             4.6    Bullet 5.6                    E


111   MercuriD50 - 013      4.6    Add bullet at end           General
      (formerly mercuri-
      034)



112 Corry-021               4.6    First sentence                T


113   Lipsio-01             4.6    Bullet 5.2              T


114   Lipsio-02             4.6    Bullet 5.2              T


115   Lipsio-03             4.6    Bullet 5.3              T




116   Lipsio-04             4.6    Bullet 5.3              T
117   Lipsio-05        4.6       Bullet 5.3           T




118   Lipsio-06        4.6       Bullet 5.3           T




119   Lipsio-07        4.6       Bullet 5.4           T




120   Lipsio-08        4.6       Bullet 5.6           T



121   Lipsio-09        4.6                            T




122   Lipsio-79       5.1.1      Para. 6                     E




123   Lipsio-7A       5.1.1      Para. 7                     E




124   RGH 006         5.1.1      last paragraph              E

125   Simons - 003    5.1.2      the entire section          G

126   Dill-13        5.1.2.1     Third bullet                E
127   Simons - 004   5.1.2.1     the entire section          G

128   Simons - 005   5.1.2.2     the entire section          G




129   Dill-14        5.1.2.3     Add introductory explanation E



130   Simons - 007   5.1.3.2.1   the entire section          G
131   Simons - 008        5.1.3.2.2     the entire section     G

132   Dill-22             5.1.3.2.5     item e                 E



133   RGH 012             5.1.3.2.5     item a                 E


134   df6                 5.1.3.4.2     Paragraphs 3-4         E

135 Lipsio-7D             5.1.3.4.6     Item “b”              E
136 Corry-031             5.1.3.5.1                           E
                                        3rd para., 2nd sentence


137   RGH 016             5.1.3.5.1     4th pp                 E

138   RGH 017        5.1.3.5.1 & 5.1.3.5.3 entire clause       E

139   Lipsio-7F           5.1.3.6.4                            E

140   Lipsio-82           5.1.3.6.8                            E
141   Lipsio-85             5.2.5                              E

142 Lipsio-86              5.2.5                               E
143 berger - 005            5.3         Para 1                 E

144 berger - 006            5.3         4th bullet             E
145 berger - 009            5.3         para 2                 E
146 berger - 010            5.3                                E

147 berger - 007              5.3          5 places            E
                       also 5.4.4, 5.4.6,
                        5.6.2, 5.6.2.6,
                     5.6.7.2.1, 5.6.8.1.2,
                     5.6.8.1.3, and many
148 Jhulshof-020         other places
                           5.3.10.3                        2   E
149 df8                    5.3.10.6        1                   E
150 df9                    5.3.10.6        5.3-7               E
151 vc1-11                   5.3.3         9 & 10              E
152 wfw -006                5.3.3.7         Last paragraph     E
                                             and sentence.



153   RGH 060             5.3.4-8                              E
154   Aragon - 02          5.3.5                 5             E

155 Corry-042              5.3.8                           2   E




156   Jhulshof-028         5.5.1        last sentence          E
157 Jhulshof-029      5.5.4      second sentence    E
158 Corry-066         5.5.4      First sentence     E
159 Corry-068         5.5.4      Last sentence      E




160 df11              5.5.4                         E
161 Jhulshof-032      5.5.7      3 and 4 sentence   E
162 Corry-069         5.5.7      f) and g)          E
163 Jhulshof-031      5.5.7      g)                 T
164 berger - 011     5.6.1.2     last para          E

165   wfw - 011      5.6.8.1.2         a)           E

166   Jhulshof-035   5.6.8.3.2   b                  E
167   wfw -012       5.6.8.3.2         b)           E

168   MercuriD50 -    6.3.1      Entire             T
      079 (new)                  subsection
169 Corry-195         7.11.1     2nd paragraph      E




170 Corry-201        7.11.12     Last sentence      E


171 Corry-197         7.11.7     1st sentence       E



172 Corry-202         7.12.2     a. 1st sentence    E


173 Corry-203         7.12.2     b.                 E



174 Corry-155          7.3       First sentence     E




175 Corry-168         7.5.10     a. Glossary        E
176 Corry-169     7.5.10   c. Program analysis    E




177   Lipsio-94     A      title of section A-1   E




178 Corry-205      A.1                           E
                           1st para., 2nd sentence


179 Corry-206      A.1     2nd para.              E
180 Corry-208          A.1      4th paragraph           E




181 Corry-209          A.1      Last paragraph          E



182 Corry-207          A.1      3rd para.               T




183   MercuriD50 -   Abstract   page ii                 E
      004 (new)
184 Peterson-1       Abstract   Section 1 or abstract   G
185   selker-1001    Abstract    Pg. ii               t
                                 Keywords

186 Corry-002        Abstract                         T



187 Corry-003        Abstract                     T
                                 1st para., 2nd sentence




188 Corry-004        Abstract                          T
                                 2nd para., last sentence




189 JL - 008            D                             E


190 berger - 002        D        Tables 1 & 2         E




191   Jhulshof-038      D        Ann. D page 213      T
                                 VSS clause
                                 3.2.2.8.
                                 last colunm
192 berger - 012     D.2.2.7.2   a, b.7, b.8          E

193 berger - 003        E        Table 1              E

194 JL - 006            E                             E
195 JL - 001           Gen                            E


196 JL - 002           Gen                            E

197 JL - 003           Gen                            E



198 JL - 004           Gen                            E

199 JL - 007           Gen                            E
200 Corry-001    Gen   Document   G




201   Dill – 1   Gen              G




202 Dill-001     Gen              G




203 Dill-002     Gen              G



204   Dill-2     Gen              G



205 Corry-005    Gen   Document   G


206   RGH 001    Gen              G
207 Gough-001    Gen   G




208   DC - 005   Gen       G/T




209   DC - 006   Gen       G/T




210   DC - 007   Gen       G/T




211 DC - 001     Gen       G/T
212   DC - 002      Gen   G/T




213   DC - 003      Gen   G/T




214   DC - 004      Gen   G/T




215   Alice - 015   Gen    T
216 Gough-002          Gen




                             T
217   HD-024           Gen




218   GHM - 007   General        G
                 Comment



Change “i.e.” (“id est” = “that    is”) to
“e.g.” (“exempli gratia” = “for    example”)
we do not have to presupose the    nature of
the changes for internet voting
This standard does provide, not    will provide
tech specs.

No mention of system integrity.



MIL-STD-1521 is referenced, first in section
6.1.3, bullet “a”.

MIL-STD-498 is referenced, first in section
7.7.1
IEEE Std. 610.12-1990 is referenced, first
in section 3.33. Add reference for IEEE
Std. 610.12-1990.
Need reference for IEEE Std 1063-2001.

Reference IEEE Std 1228-1994 Add reference
for IEEE Std 1228-1994, “IEEE Standard for
Software Safety Plans”.
Reference IEEE Std 829™-1998

Reference IEEE Std 1063™-2001


Reference IEEE Std 1028™-1997

Reference IEEE Std 1471™-2000



Reference IEEE Std 1016™-1998


Reference IEEE Std 14143.1™-2000
Reference IEEE Std 1061™-1998


Reference IEEE Std 1008™-1987 (R1993)


Reference IEEE Std 982.1™-1988


Last line is incomplete.
Incomplete sentence.




Punctuation missing at end of sentence.


Incomplete sentence.



NIAP

Add definition for RTOS (which, presumably,
is the sort of OS a voting device uses).
Election laws may change regarding federal recalls, although
recall can be performed using impeachment proceedings,.




Public Counter is similar to ballot counter.
Cross reference them.
The definition doesn't identify that this is
DRE related.
The definition has apparently been
truncated. "formats that"
Non-partisan should not be hyphenated
This definition is different from the
definition given in the FEC Voting System
Standards adopted in 2002.   That definition
is:

“A Direct Record Electronic (DRE) Voting
System records votes by means of a ballot
display provided with mechanical or electro-
optical components that can be activated by
the voter; that processes data by means of a
computer program; and that records voting
data and ballot images in memory components.
It produces a tabulation of the voting data
stored in a removable memory component and
as printed copy. The system may also provide
a means for transmitting individual ballots
or vote totals to a central location for
consolidating and reporting results from
precincts at the central location.

The proposed definition is:

“A voting system that records votes by means
of a ballot display provided with mechanical
or electro-optical components that can be
actuated by the voter; that processes the
data by means of a computer program; and
that records voting data and cast vote
records in internal and/or external memory
components. It produces a tabulation of the
votingis notstored inanywhere thatmemory
CCEVS data defined a removable I can
find. What is it?
Ballot Scanner definition does not incorporate voter verified
products.
Add definition for "Ballot"




Add definition for "Election Auditor" and,
equivalently, "Election Observer"
Add definition for "Election Verification"




Add definition for "Sealed Ballot Box"
Add definition for "Tabulation Rules"




Add definition for "Tally"



It is a database that is being referred to
here, not a data file.


Does not specify where build is to occur.




Definition is too narrow


Can components be software? All the examples are
hardware.
Is a device that prints a paper ballot for reading by another
device a DRE? Perhaps we should say "no" and call it paper-
based. I'm also concerned about the "removable memory
component". Later, there is discussion of DREs that transfer
votes over a network. bar codes, OCR, etc. in this definition
I would like to include

Explanation about exemption is unnecessary, and may
become inconsistent if we add change requirements on
COTS
Is this meant to exclude voting systems based on computer-
printed paper ballots?
Nitpic
voter verified audit record is a user
interface statement indicating that there is
an ability for voters to recognise and judge
an audit. Either this should be established
as a completeable voter responsibility or we
should leave out the word voter

      Definition "Voter Verifiable Audit Record: A
      human-readable printed record of all of a voters
      selections presented to the voter before the vote
      is cast. Also called Voter Verifiable Record."
      is specifically limited to one design, paper, and
      needs to focus on performance.
We urge against defining this important term
in any other documents, more particularly in
a document that has been superseded.     The
problem is repeated in the definition: “A
voting system referred to in the 1990
Standards as a Punchcard and
Marksense (P&M) Voting System that records
[emphasis added].” Paper based voting
systems do not have to be punchcard or
Marksense.   We have the following
suggestion to improve the definition:
“Voting System that records votes, counts
votes, and produces a tabulation of the vote
count, using one or more ballot cards or
sheets of paper or a written list of
choices.” We suggest adding the italicized
words.

Furthermore, we note that the FEC 2002 VSS
has this to say about Paper Based Systems:
“Additionally, a paper based system may, or
records votes using other approaches whereby
the voter’s selections are indicated by
marks made on a paper ballot by an
electronic input device based on input from
the voter, as long as such an input device
does not independently record, store, or
tabulate the voters selections...”
These definitions rely too heavily on a
specific implementation and the term
“affiliation.”   While the definition of
Open Primary doesn’t seem to present a
problem, the definition of Closed Primary
does: “Closed Primary: A primary election
in which voters receive a ballot listing
only those candidates running for office in
the political party with which the voters
are affiliated”. One can imagine a DRE
machine that presents a first question as to
the party’s primary upon which the voter
chooses to vote. Following such a decision,
even those made within the privacy of the
voting booth, the voter is given a choice of
only one party’s candidates. While this
could be considered an Open Primary, it
falls within the proposed definition of
Closed Primary.

Finally, there is no definition for what
some call a “Blanket Primary”. In this
case, a voter may vote for candidates of one
party in some races and for another parties
candidates in another race. While there
have been court cases about the legitimacy
of such a type of Primary, there is no harm
describing it. Whether or not it is legal
is a totally different question.
Addition to References

Several NIST security guidlines are
available and should be referenced in this
standard.




We recommend that this definition be
improved. Not only is the definition
problematic, it causes significant problems
later when the Draft Standard calls for
source code for firmware. Much firmware
found in today’s electronic voting systems
will not have any source code since that
firmware (usually found in device
controllers, BIOS and other discrete
hardware units) is usually provided by third
parties who do not make source code
available.

The definition itself is problematic. For
example, is software stored on a CD-ROM
firmware? Is software stored on a hard
drive that is write-protected firmware? Is
software stored on a flash memory that can
only be modified by removal from the voting
machine firmware?
The problem here is mixing two different
ideas. The first is recall. The second is
Options. The danger of mixing is borne out
by recent court decisions in California
which allow a recall, but do not allow any
restriction on Options. We suggest two
different definitions: Recall and Recall
Options. Does a system support Recall
(according to Definition 82) if it allows
voters voting No on the recall to vote for a
successor? Recent law seems to suggest that
such a possiblity should not even exist.
Separating these shields the definition from
specific legal findings.
Accessibility --by this definition, no equipment can be
accessible.

Add VSS & HAVA.



Add acronyms used in this draft but
presently undefined. Page where referenced
is given in parentheses.




Use of “their” as a singular pronoun of
common gender is slang; also, the draft
often, probably usually, uses “his/her”.
“States” excludes U.S. Territories, does not
allow for the possibility of more local
control, and makes the standard gratuitously
U.S.A.-specific.
URLs, being subject to continuous change,
need to have an associated date when used as
a reference. Note that this was agreed upon
at the meeting in January, 2002.
“and/or” is inappropriate in formal English
where “or” denotes an inclusive or and
exclusive or is expressed by added “but not
both”.
The distinguishing feature of “firmware” is
not its inalterability during operation
(which can be accomplished with RAM by
various methods such as disabling the write-
enable signal), but its inalterability
without electrical or mechanical means.
“The 1990 standards” are undefined.

The word “Agency” is used in its own
definition.
This definition does not apply to all
states; in Florida, for example, there is a
“Presidential Preferential” which has
nothing to do with choosing delegates.
This section and its figure describes a DRE
system. But it is equally applicable to a
paper based system.
“This section Software and firmware
documentation, information, and materials,
including the following:

the release software, firmware, utilities,
hardware, and instructions required to
install, operate and test the voting
system.”

While this is a good idea, this requirement
is utterly impractical. There is “firmware”
in device controllers and the Operating
System BIOS. There is likely to be firmware
in places not even suspected by any vendor.
Such copies of the “source” is simply not
possible.


The word “appropriated” is a typo
Unmodified COTS are not exempt from
evaluation to preclude the threats
identified in 5.3.2.1 (A).
The definition of “datagram” should be
dropped and the definition of whatever word
is substituted for datagram added to allow
for both TCP and UDP protocols (or, more
generally, for connection and connectionless
protocols) to be covered by statements
relative to communications in the standard.
Add definition of “Voter-Verifiable Paper
Audit Record”




Change “i.e.” (“id est” = “that is”) to
“e.g.” (“exempli gratia” = “for example”)
Both these sections define “software
verification and validation”.




This section should provide information
regarding the likely sequence of events in
using the system. This sequence of events
includes storage between elections, pre-
election activities, delivery to a polling
place, use during an election, return to a
central facility, post-election activities,
and return to storage. Anticipated
conditions at these sites should be
described.
Insufficient antecedent support for
"separate method" (separate from what? the
Cast Vote Record?). Clarify that the audit
trail starts from the voter's decision
(voter intent being the legal standard in
many jurisdictions), and is a "separate
method" from that point forward rather than
merely an alternative output mode for the
Cast Vote Record.
Sentence mentions "components" which are not hardware,
and inappropriately restricts scope to DREs. Optical scan
systems are presumably included in the discussion, and the
Much of this section is not of scope of
Election Management System is outpart of the DRE. the
current standards.
System description goes beyond an EMS used
with DRE.




Is it true that the single-arrow flows are enforced as
unidirectional? Also, do the dotted-lines indicate the fact that
there are times when communications are prohibited or
restricted?




Most pf the document is not written as a
standard; according to “IEEE Standards
Companion”
(http://standards.ieee.org/guides/companion/
part1.html#how ), ‘One of the major
difficulties in splitting up the work (the
divide-and-conquer school of thought) is
that there is often an inconsistency of
tone in the document as a result. One way to
avoid this problem is to remember to use
standards verbs (shall, should, and may) as
the primary means of conveying the tone of
your document. Standards primarily use
"shall," recommended practices primarily use
"should," and guides primarily use "may."
Remember, however, that this is not an
exclusive definition. Standards can use
"may" every once in a while, just as guides
can use "shall." Indeed, this kind of use is
almost inevitable. What needs to be
attained is an overall consistency of tone.
If a guide uses "shall" almost all the time,
with a few "mays" sprinkled in, is it
really a guide? The overall tone is
mandatory, and that can cause a problem. So
consistency in the use of verbs, and the use
of proper standards verbs, can help to
We need to eliminate the Election Management
System information, it is not in scope for
the standard
The EMS described is out of scope of current
standards.
“IEEE Standards Companion”
(http://standards.ieee.org/guides/companion/
part1.html#how ), ‘General violation of One
other aspect of standards writing that is
very confusing for working groups is the use
of the word "must." Traditionally, "must" is
frowned upon in standards writing because
its mandatory nature can be confused with
"shall." In other words, when you say a user
must do something, are you mandating this?
Or are you saying it's an inevitable result
of the situation they are in? Remember,
"must" is not a defined standards verb in
standards organizations. Therefore, the
mandatory nature of a statement with "must"
in a standard could be called into question
in a court of law, and there would be no
existing practice or rules to back up its
meaning (keep in mind what was discussed
earlier, the quasi-legal nature of standards
and the need for a clear understanding of a
standard's intent). For this reason, "must"
should be avoided unless it is being used in
a descriptive fashion (if it is raining, the
sky must be gray). Stick to the defined
standards verbs for the sake of clarity
between you and the users of your standard.’
Describe material that should be excluded
according to the Document’s Scope.

That defines, which expands. Correct
grammar.
Add to bulleted item #13


Add new bulleted item to bottom of list


Same general comment as Aragon - 04.
Address known misconceptions about
interaction between this "separate record"
and the requirement for accessibility under
HAVA and 5.3.10.
No mention of audio secrecy.
. If you take this section and follow the
bullets listed under it, you arrive at:
“The precinct voting stations present the
ballot to the voter and provide capabilities
for:

has been cast after the vote is stored
successfully”
A paper based system does not have a “voting
station” that signifies that the vote has
been stored successfully.”

1st sentence "described in section 4.1" -
not described in section 4.1.
The reporting of absentee votes as "separate precincts" has
created considerable confusion in end-of-day tallying. These
are known as "ghost precincts" and it makes it difficult, if not
impossible (in some court cases) to ascertain which precinct
the ballots should be attributed to. All votes should be
attributed to the precinct in which the voter was authorized to
"This sent to is undefined.
vote, notmeans" separate precincts.




Specifies 22 months for data retention. A
spec is not appropriate in section 4, but
should be in section 5 which it is --
5.2.2.2, 5.2.2.3, and 5.1.3.5.6.h
The second sentence is difficult to parse
because in “This means must”, it is not
clear at first that “means” is a noun
referring to the means in the first
sentence.
"rolling back" - needs to be more specific



It would be better if the device audit log
were mandated to log exceptions at all times
to have a baseline for showing that the
device behaves similarly in actual use as in
test use, and to record activity outside of
actual voting use as evidence against
tampering in the event of a recount or
suspicion of tampering or malfunction.
“A voting system provides a means for
obtaining a printed report of the votes
counted on each voting device.” This
section is too prescriptive. One can easily
imagine a DRE environment in which all
voting stations are networked and for which
there is no need for a report from each
voting device. Similarly, one can imagine a
paper based system in which there are no
voting records stored on the voting device.

Reporting sybsystems should provide vendor-independent
means for producing polling place, precinct, consolidated and
audit log reports.


change "more than one voting device" - this
is a marketable feature, NOT a standard



Specifies 22 months for data retention. A
spec is not appropriate in section 4, but
should be in section 5 which it is --
5.2.2.2, 5.2.2.3, and 5.1.3.5.6.h
The section says, “The printed report shall
contain all information generated by the
system audit log.” There may be more than
one log. Also, the printing of a log report
should be optional. A good log should be
cumulative and printing it every time may
result in reams of paper.
Consolidated reports is not in scope of this
draft

Requires logging of OS and version, hardware
and peripherals. This is not always
feasible in a custom hardware environment
and might not even have an OS. Peripherals
may be temporarily attached as part of the
election phase and it's use and not it's
presence should be included in the audit
log. (e.g. The attachement of an external
printer or modem would not be logged but
rather the printing of a report or the
transmission of results.
Identification without proof of authenticity
or integrity is meaningless.
Add need for consistency of alternate files with primary files.




Does not prohibit alteration or modification
of the alternate data file. This is a gaping
security hole in the current Diebold GEMS
software.
Changes to election results can not be
allowed not alter these results.




None of this should needed; the presence of
this is evidence that this standard does not
specify a secure or mission-critical system.

Tools and components are mixed together;
however, components were specified above in
bullet 3.
It needs to be specified how updates to software are going to
be supplied and performed.




Documentation is a requirement, not an
option.

Should give general term, not specific
examples.

Need a copy of the makefile or equivalent,
with directions, in order for this to be
useful.
Flowcharts are only a specific type of
software documentation, and a mostly archaic
method.




Specification is insufficient.
Specification is insufficient.




Specification is insufficient.




Identification without proof of authenticity
or integrity is meaningless.



Need a copy of tools in order for this to be
useful; the system can not be rebuilt
without the tools and specific versions of
tools may become unavailable.
Bring into conformance with IEEE Std 1063-
2001, “IEEE Standard for Software User
Documentation”



“Using validated products can significantly
reduce the cost ... by providing information
on how to securely configure a particular IT
product within a system” makes no sense.

It is unclear if “vendors” means “COTS
vendors” or “voting equipment vendors” in
“vendors must adequately describe the
control methods they have employed to ensure
these risks have been mitigated.”
There is a change of gears just past the
middle of the paragraph.
The section talks about requirements, but
there are none!
What does "to include vendor or contractor facilities" mean ?
This section talks about requirements, but
there are none!
While these elements may not be under the
direct control of the vendor, it is still
necessary to include security requirements.
Otherwise, the entire system could be
compromised.
What is the scope? How is it intended to be used?



Instead of allowing the vendor to specify
the features and capabilities of the access
control mechanisms, a set of strong vendor
requirements should be included in this
document.
There are no minimal general access
requirements given.
Item is unclear. What does "multiple-language presentation"
mean? If it means voter can selection English + their native
language, it may be ineffective at concealing voter identity,
The acronym "CVR" e used
as well as lengthening thisballot without
definition. A definition can be found 9
pages further on in 5.2.1.2.
Paragraphs 3-4 The 3 points should be
indented and numbered 1-3.
“Time and data” is a typo.
Correct sentence structure and section
reference.

System Operating Manual referred to
redundantly
This text is redundant, duplicating a small
part of text found previously in 5.1.3.5.1.
This is redundant to verbiage in section
5.1.1.
This is a requirement.
Grammatical error

Repeated word “consist … of consisting”
Remove 'DRE' in first sentence to be
consistent with document scope.
Change 'catch errors' to 'identify errors'.
Replace 'such' with 'these'
Add to acronyms list - HFE - Human Factors
Engineering
Change 'standards' to 'requirements'.




point 2 is missing
Figures 5.3-7 through 5.3-9 are wrong.
Figure number missing for 5.3-7
Indents are inconsistent
The sentence lacks specificity. Does this
mean the luminace and contrast requirements
can be met thru multiple ballot choices or
machine adjustibility. Previous paragraphs
state that "if display is user adjustible."
the use of the word data is incorrect.
In "eliminate the accidental actuation",
remove "the".
Correct typographical error.




item "e" is missing
26 is low case
Footnote 26 needs to be superscripted.
Incomplete sentence.




"air discharge26 and…"
 f) and g) wrong
Should be numbered a) and b)
20 is wrong
Add space in the last paragraph between
"5.1of".
Standard should not include any reference to
punch cards
hava does not allow punch card
Standard should not include any reference to
punch cards
Entire subsection needs to be reviewed for correctness.

Sentence structure and completeness.




indicated not indicate


Sentence structure



Correct sentence structure and wording.


Correct sentence structure and wording.



Better summarize what is being established.




Vendor should not need to put terms in their
glossary that are defined in this Standard.
Sentence structure is awkward and comma
required.




The title of section A-1, “A.1 Development
of Voting Equipment”, causes confusion when
looking at the TOC or searching the document
because “Development” is elsewhere used in a
specialized sense for product development.

Correct sentence structure.


Correct usage of 'which" and editorial
comments
Correct wording.




Note that IEEE is an independent body.



Need to be clear that mechanical devices
didn't stop election fraud.



Abstract should be made consistent with
Section 1.1
In taking a look at the draft P1583 abstract, introduction, etc.,
I was struck by the absence of any information concerning
the "voluntary" nature of this (and all our) standard(s). In
particular, I think the draft standard needs some words in the
ABSTRACT and INTRODUCTION that reflect the ANSI
boilerplate: "The use of American National Standards is
completely voluntary; their existence does not in any respect
preclude anyone, whether he has approved the standard or
not, from manufacturing, marketing, purchasing, or using
products, processes, or procedures not conforming to the
standard."

To which I'd like to add something along the lines: "...
standard and is not itself a mandatory or regulatory
requirement."

While I assume the ANSI boilerplate will be included when
add keywords concerning usability


Much of the security section is out of scope
by this definition.


Election officials often insert the ballot
into the tabulator rather than the voter.



Other election officials besides the poll
workers usually perform these tasks.




The Table in Annex D refers to English units (inch/pound). Units
throughout the draft MUST be converted to metric units. Failure to
do this may cause a disapproval by the Standards Board
Widen table to shorten document by 3 pages.




No increase air discharge



Relink cross references. Currently these
show a cross reference error.
Make font same as rest of document.
Shortens document by 2 pages.
Annex E needs to be labeled as Informative or Normative.
Review the use of shall/should/may/can/will/must throughout the
document to be sure they are used in accordance with IEEE's style.
There are a lot of footnotes in the draft. Please keep in mind that
footnotes to text are informative and not normative.
The figures and tables are numbered incorrectly throughout the
draft. The figures and tables should be numbered according to
Clause. For example, figures and tables in Clause 6 should be
labeled as Figure 6-1, 6-2, Table 6-1, Table 6-2, etc.
Please label all figures and tables throughout the draft. Currently,
some tables are unnumbered.
The IEEE-SA’s style for Annexes is to list Normative Annexes first
followed by Informative Annexes. Currently, Annexes A, B, and D
are labeled as Informative and Annex C is Normative. The order of
these Annexes should be switched around.
While the draft is a huge improvement it
still requires a great deal of refinement
and rewriting. Sections 5.6 and 6.6 are to
be rewritten as well as Annex C. And I am
told that Annex C is now Annex D because
another annex has been added that I haven't
seen. Too many typos and editorial problems
remain to allow this draft to go out as a
final standard.Tables and figures are
missing or need to be redrafted. Units of
measurement are used inconsistently and
incorrectly related to other systems, e.g.,
mm and inches.MKS SI units must be used
consistently throughout with English units
in parentheses if needed or desired. There
are many precision without accuracy errors
that need to be cleaned up. The document
badly needs to be formatted and section
headings cleaned up.
The standard should prioritize security requirements, with
"accountability" -- the ability to corrupted vote records or
totals -- at the highest priority., along with confidentiality. The
vendor should be required to define the "trusted subset" of
hardware, software, procedures, and personnel, which/who
must perform as specified to preserve accountability and
confidentiality, under the assumption that all other parts of
the system can behave adversarially.. The size of the trusted
subset must be minimized. The trusted subset must be
subjected to very rigorous review, while the aspects of the
system outside of the trusted subset can be reviewed and
The standard does not establish low for the
tested less rigorously. EAL2 is definitely tooan adequate,
testable performance standard for security.
To be trustworthy, voting equipment must
either satisfy extremely rigorous security
requirements including both design and
procedures, or there must be a trustworthy
independent method to verify the election
results.
The draft standard is not ready. Many
comments from committee members have been
dismissed without adequate consideration or
discussion.
The division into "DREs" and "paper based" does not
account for future technologies, such as electronic media
that are used somewhat like paper is now (e.g., CalTech/MIT
"frogs"). units should be used consistently
Metric
throughout. English units given in
parentheses.Presently it is a patchwork.
FIRST COLUMN – Illustration – C63 member
abbreviation, followed by sequential comment
number
The document needs to be proofed for grammar,
punctuation and consistency. There are instances
of missing or incorrect use of words. There is an
inconsistency in the use of identifiers preceding
groups of statements (i.e. letters with periods,
letters with parenthesis, bullets and diamonds).
The bullets and diamonds should be replaced with
letters so that you can make a specific reference
to any given statement in the document. Also,
there is an inconsistency in the use of punctuation
following the statements. As a last general note,
not all the acronyms and abbreviations used in the
Privacy
 Secret ballot is essential for elections in a democracy.
The privacy component may influence voter turnout.
Possibilities for improper influence damage the
perceived fairness of an election. Protection of privacy
is a security issue and should be subject to the same
types of procedures as integrity.
Usability
The issue of usability can only be evaluated by actual
use. It should include voter training and real voting
experience. Test should be defined that measure the
efficacy of the combined training and voting experience
using standardized ballots and random selection of
Reliability/Safety/Quality
The usual shake and bake, electromagnetic interference,
and quality of manufacturing processes come into play
here. It would appear that these are technology
dependent and that a lot of effort goes into
standardizing them for other types of equipment. This
would be a section that should probably only refer to
other standards and qualification processes. For
General Principles
Separate the technical from the policy; do not use
technical standards to make policy tradeoffs. Favor
innovation; do not favor entrenched players or
practices. Use existing best practices and expertise; do
not re-invent the wheel or have an exclusionary
Measurement v Acceptability
The best thing a technical standard can do in an
evolving field (and to support evolution of that field) is
provide clear and accepted measurement of systems
along defined criteria so that policy decisions about the
levels of those criteria can be enforced. The standard
should not be judging what is acceptable, but rather
setting technical criteria and measurement techniques to
allow systems to be evaluated. Another reason this is
necessary is that there are tradeoffs among the different
attributes and different price points that should be
available to governments. For example, accessibility
(which cannot be absolute) and price seem necessarily a
tradeoff that cannot be evaluated technically, but rather
is inherently a policy decision. further example is the
Design v Performance
Design standards tell how to build systems,
performance standards give criteria for evaluating
systems. Describing how to build things not only stifles
innovation, but it only provides a barrier to entry, which
compounds the problem of stifling innovation.
Presenting different requirements for different
technologies is more of a design standard than a
performance standard. For example, scan and DRE
should not be differentiated in the standard. A
performance standard should clearly call out a set of
Integrity
Voter confidence in the integrity of the outcome, i.e.
tallied as cast, is of paramount importance. The more
effective and convincing a demonstration of this can be
in future electronic systems, the bigger boost to voter
confidence. The assumptions, threat model, and why the
system achieves integrity are laid out by the submitter.
These are then subjected to long established techniques
in security of white box and black box evaluation. They
also determine the level of integrity achieved. The
techniques and laboratories of the Common Criteria
should be employed. In particular a protection profile
The document does not address how absentee ballots would
be counted on Election Day using a DRE system. Unless the
Board counts all absentees in-house, each precinct would
need to have an absentee card reader that could interface
with the voting system. Since punch card voting is not longer
a valid option, consideration should be given to an optical
scan device that could interface with a DRE system. It would
be beneficial to the Board if this consideration was covered in
There are many inconsistencies in the letter
sub paragraph structure (e.g. b. vs b)) and
linkages beween them (e.g. the use of the
word and in some cases and not others).

The following recommendations are for the HAVA software
which were listed in the analysis to improve their
documentation and accuracy that I think are especially
important.
1. 102 - Sections 1,2,3 and appendix C addresses
acceptabe standards for punch cards that would no longer be
valid under HAVA.
2. 301.a.1.a Privacy and independence are not addressed.
301.a.1.b VSS has no guidelines for changing or correcting
absentee ballots.
4. 301.a.2.a HAVA definition of audit may not provide the
accurate information needed to check on voting irregularities.
             Proposed Change



Change “i.e.” to “e.g.”

delete the word additional

Change "will provide" to "provides," i.e.
"This standard [provides]..." in two places.

Revise to read "This standard [provides]
technical specifications for electronic,
mechanical, [integrity,] and human
factors..."
Add reference for MIL-STD-498, Software Test
Plan (STP) and Software Test Description
(STD)
Add reference for MIL-STD-1521.




Add reference for IEEE Std 1063-2001, “IEEE
Standard for Software User Documentation”.



Add reference for IEEE Std   829™-1998, “IEEE
Standard for Software Test   Documentation”.
Add reference for IEEE Std   1063™-2001, “IEEE
Standard for Software User   Documentation”.

Add reference for IEEE Std 1028™-1997, “IEEE
Standard for Software Reviews”.
Add reference for IEEE Std 1471™-2000, “IEEE
Recommended Practice for Architectural
Description of Software Intensive Systems”.

Add reference for IEEE Std 1016™-1998, “IEEE
Recommended Practice for Software Design
Descriptions”.
Add reference for IEEE Std 14143.1™-2000,
“IEEE Adoption of ISO/IEC 14143-1:1998
Information Technology --- Software
Measurement --- Functional Size Measurement -
-- Part 1: Definition of Concepts”.
Add reference for IEEE Std 1061™-1998, “IEEE
Standard for a Software Quality Metrics
Methodology”.
Add reference for IEEE Std 1008™-1987
(R1993), “IEEE Standard for Software Unit
Testing”.
Add reference for IEEE Std 982.1™-1988,
“IEEE Standard Dictionary of Measures to
Produce Reliable Software”.
Complete the sentence/paragraph.
Alternate formats usable by people with
disabilities may include, but are not
limited to, Braille, ASCII text, large
print, recorded audio, and electronic
formats[.] [delete- that]
sound and[;] hence[,] suitable for use as a
statement of requirements for one or more
TOEs that may be evaluated.
An implementation [of an] independent set of
security requirements for a category of IT
products, which meet specific consumer
needs.
Spell out NIAP to match the format for other
references.
Add: “RTOS    Real-Time Operating System”

There are currently no provisions for the recall of federal
office holders via precinct balloting.




Add "See Ballot Counter."

Add the phrase "….of all voter selections on
a DRE voting unit presented ……."
Find the original intended definition and
add the missing text.
Change "Non-Partisan Office:" to
"Nonpartisan Office:"
Make consistent with FEC definition.




Need to add definition for CCEVS. Taken care
in abbreviations.
Ballot Scanner: A device used to read the data from a
marksense or voter verified ballot.
The data that represents a voter’s choices.
Requirements need not prescribe specific
properties, but every system shall provide
its own formal ballot specification
satisfying the following:

1. The specification shall be sufficiently
detailed to allow all election auditors and
observers to determine, with perfect
consistency, whether some data does, or does
not, constitute a ballot.

2. The specification shall be publicly
available.

A designated individual or group that
desires to scrutinize the election.
The combination of voter verification (cast
as intended) and results verification
(counted as cast) that provides full
confidence that the tally accurately
reflects the electorate’s intent (counted as
intended).
The ballot box that is certified as final by
appropriate authorities. It is not damaging
for the sealed ballot box to contain
illegitimate or invalid ballots (e.g.,
provisional ballots), since these will be
detected and eliminated by the tabulation
rules. Since voters are able to detect and
prove the condition of missing ballots,
policies should be in place that specifies
accountability.
The set of elementary arithmetic and logical
operations that produce a unique tally from
any collection of ballots. Requirements need
not prescribe specific operations, but every
system shall provide its own formal
specification of tabulation rules
satisfying:

1. The specification shall allow all
election auditors and observers with access
to both the collection of ballots and the
tally data to determine, with full
confidence, whether the tally has been
properly formed according to the tabulation
rules.

2. The specification shall be publicly
available.

3. The tabulation rules shall satisfy an
“additive property”: if tallies for two
disjoint collections of ballots are each
created according to the tabulation rules,
the sum of the respective totals shall
always be identical to the total created
according to the tabulation rules for the
ballot box that is the aggregation.
An assignment to each candidate, or possible
voter response, a non-negative total. It may
also contain additional data for the purpose
of election verification.
Change to read: "A [database] or set of
files that contains geographic information
about political subdivisions and
boundaries;"
Specify voting system: "Typically as part of
the PCA, a recognized testing authority also
witnesses the building of the executable
system to ensure that the qualified
executable release is built [on the voting
system being tested] from the tested
components."
A permanent record of a voter's selections that can be
checked for accuracy visually or with the use of a trusted
device, which is preserved in case of a subsequent recount.
Either say "hardware" somewhere or give examples such as
the operating system.
…a removable memory component. It may also print a
paper copy of the CVR.



A device that transfers the vote data on a paper ballot to an
equivalent electronic representation.
Delete last sentence of definition.
… records votes, countes votes, and produces a tabulation
of the vote count using paper ballots.
Delete comma, change "which" to "that"
verifiable audit trail




       change Definition to "Voter Verifiable Audit
       Record: A human-readable record of all of a
       voters selections presented to the voter before
       the vote is cast. Also called Voter Verifiable
       Record."
We suggest keeping this important part of
the FEC definition with appropriate
grammatical corrections.
The following references need to be added to
the listing:

Cryptographic Modules

FIPS 140-1: Security Requirements for
Cryptographic Modules, January 4, 1994.

FIPS 140-2: Security Requirements for
Cryptographic Modules, May 25, 2001. Change
Notices 2, 3 and 4: 12/03/2002

Cryptographic Algorithms

FIPS 197: Advanced Encryption Standard
(AES). FIPS 197 specifies the AES algorithm.

FIPS 46-3 and FIPS 81: Data Encryption
Standard (DES) and DES Modes of Operation.
FIPS 46-3 specifies the DES and Triple DES
algorithms.

FIPS 186-2 and FIPS 180-1: Digital Signature
Standard (DSS) and Secure Hash Standard
(SHS), which specify the DSA, RSA, ECDSA,
and SHA-1 algorithms

FIPS 185: Escrowed Encryption Standard
(EES), which specifies the Skipjack
algorithm
Add:
HAVA - Help America Vote Act
VSS - Voting System Standard

ANSI (p. 52) - American National Standards
Institute; CISPR - International Special
Committee on Radio Interference; CCTL -
Common Criteria Testing Laboratory; DAC (p.
152) - Discretionary Access Control; ERD -
Entity Relationship Diagram (see comment 163
below); FIPS (p. 150) - Federal Information
Processing Standards; HIPO (p. 122) - [NOTE
that I think this acronymn may have been
misused in 7.5.7.1. If correctly used then
need definition but I would strongly
recommend deletion.]; ICMP (p. 32) -
Internet Control Message Protocol; ISSA -
Information Systems Security Association;
MIL-STD - US Military Standard; NSA -
National Security Agency; SPL (p.47)
[Note:This acronym possibly misused. If
correctly used need definition.]; ST (p.
153) - Security Target; STD (p. 128) -
Software Test Description; STP (p. 128) -
Software Test Plan; TSF (p. 145) [Note: This
acronym may have been misused. If correctly
used need definition.]
Adapt a convention for the common gender
pronoun and use it consistently throughout
the standard.
Change all instances of “state” in the
political sense (as opposed to, say, a
“state machine”) to, perhaps, “election
jurisdiction”.
Add dates to all URLs.
Change all occurrences of “and/or” to “or”.



Change (or add to) “during system operation”
to “without physically modifying the
device”.



Add a reference in section 2 for the
standard, and use that reference here.


Strike the definition; it is not a special
type of election.


It would be better to describe this as a
voting system, as it seems to do later in
the successive paragraphs.




Change approprated to appropriate
Delete the second sentence of the
definition.

Revise the list of definitions accordingly.
Insert in the appropriate sequence: Voter
Verifiable Paper Audit Record – An audit
record having the following characteristics:
1. The record is viewed by the voter and
capable of being viewed by a recounter or
auditor in plain language on the exact same
medium on which it is stored for purposes of
the audit. Paper is the principal example of
wuch a medium.   2. There is no technology
or functionality between the voter (or
recounter or auditor) and the record
physically capable of altering the content
of the data as viewed by the voter,
recounter or auditor. An example of allowed
technology is pure optical magnification.
An example of disallowed technology is any
form of electronic interpretation of
formatted data, except for persons lacking
sufficient vision to view the record through
magnification.

Change “i.e.” to “e.g.”

Remove the common definition of “software
verification and validation” from both these
and make it a definition in its own right.
Reference IEEE Std. 1012-1998, “IEEE
Standard for Software Verification and
Validation”; perhaps use the definition from
that document. 2
The outline implied by the comment should be
expanded into a few paragraphs of text.




Change to: "Provide for a possible recount
or audit by providing a separate record of
the cast vote, which the voter has the
opportunity to review for correctness."




This clause [is it really a clause?] describes the organization
of a generic voting system.

Make it clear where this section is out of
scope but included for general information.
See comments below.
Rewrite sentence to read: This clause
[provides an overview of] the components
that comprise [an election management
system, vote recording systems covered by
these standards, and ancilliary election
reporting systems.
The diagram is unclear, specify the meaning of the arrows in
a footnote.




Change all sentences specifying something to
“shall”, e.g., in 4.2.1, “The EMS allows the
user” becomes “The EMS shall allow the
user…”.




If you want to describe other voting system
aspects not in scope of the draft for
context, they should be in the Appendix
After initial sentence add: The EMS
described below is not covered by the
current standards. However, an EMS will
normally be required and used to generate
ballots and program the voting equipment
that is defined by these standards. Not
every EMS will necessarily have all features
described in this section.
Change the verb “must” to “shall”, e.g., in
4.2.1, “The environment in which all
databases in the subsystem are maintained
must include all necessary provisions for
security and access control” becomes “The
environment in which all databases in the
subsystem are maintained shall include all
necessary provisions for security and access
control.”




We have no preference, either the Scope
should be expanded or this material should
be described elsewhere.
The Control Subsystem consists of the physical devices and
software [that] accomplish and validate the following operations.
* Recording an image of the ballot cast that identically
reflects the choices made by the voter.

* Allow for the alert of poll watchers and election judges and
officials if a voter believes that the equipment is preventing
them from casting a ballot of their choosing.
Add informational annex with illustrative
examples, as attached



· Protecting the secrecy of the vote such
that the vote may not be observed [or
overheard] during the voter’s selection of
preferences, during the casting of the
ballot, and as the voted ballot is
transmitted for recording on a storage
device, or in the recording of the CVR
Change “Signifying…” to "Except in cases
where a paper ballot is deposited in a
ballot box, signifying…"




use correct section

Rewrite as: …tallying the absentee votes by allocating them
back to the voter's precinct, or by creating subtallies within
the voter's precinct. (Note: If this comment is out of
scope, then this entire subsection is out of scope and it
should be deleted.)

Add definition: This means [of
consolidation] must comply with the security
and procedural requirements that apply to
the system as a whole and to the individual
counting devices.
Change sentence to not include the specified
time period, but indicate only that a period
is required. A cross reference to 5.2.2.2
can be added.
Change “means” to “mechanism” in both
sentences.



what is rolled back Some things should not
be rolled back, such as the permanent
counter. Maybe other data such as this
should be allowed not to roll back.
Reword to: “The printed report shall contain
all that is in the device audit log. The
device audit log shall contain all exception
conditions encountered since the earlier of
either the installation of the ballot
configuration (election file) or
commencement of testing for the election to
which the said audit log applies.
Change to: “A voting system proides a means
for obtaining a printed report of the votes
counted on each counting device”




Add the phrase "vendor-independent" into "must provide a
means" -- to read "must provide a vendor-independent
means" in each of these sections.


Strike this requirement. As long as there
are the appropriate paper tapes and
electronic audit trails, the consolidation
is only an efficiency issue, not an
integrity issue.
Change sentence to not include the specified
time period, but indicate only that a period
is required. A cross reference to 5.2.2.2
can be added.
Change to: The printed report shall have
the ability to contain all information
generated by one or more system audit logs.




If you want to describe other voting system
aspects not in scope of the draft for
context, they should be in the Appendix
add "if applicable" to the end of the
sentence.




The audit log shall also contain, for each
of these items, a checksum with strong error
detection properties, e.g., an MD-5 hash
code. The working group should discuss
minimum specifications and, possibly,
prevention of spoofing.
Data in any alternate files must be 100% consistent with all
information maintained in the primary file.




Add sentence: Nor shall the system allow
alteration or modification of the contents
of the alternate file in any manner that
affects election results.
Add : “Such access shall not alter recorded
election data. To ensure this, a checksum
of election data shall be recorded in the
audit log upon closing of the polls and a
timestamp of all changes to the election
data shall be recoded in the audit log.”
Eliminate the bullet assuming that the
concerns are addressed by prohibitions
elsewhere in the standard.

Move “development libraries, device drivers,
operating systems, and monitors” to bullet
3.
* Documentation describing how an update is to be certified
and performed, should there be a declared or discovered
defect in the voting system, software, hardware, or firmware,
or any COTS products used in or in the development of the
system that could compromise its operation as an election
device.
Change should to shall: The voting system
[shall] include the following documentation:

Replace “Diskette, tapes, or compact disks
containing copies” with “Copies on permanent
machine readable media”.
Add a “makefile or build script” after
“utilities”; add “and build” after “read”.

Change to: “CASE tool output or other
documentation of all data and program flows,
interactions of threads, uses of shared
data, and timing analyses of all software
components, in compliance with IEEE Std
1016™-1998, “IEEE Recommended Practice for
Software Design Descriptions”, and IEEE Std
1471™-2000, “IEEE Recommended Practice for
Architectural Description of Software
Intensive Systems”.
Require compliance with section 4.3.5
(“Documentation Requirements”) of IEEE Std
1228-1994, “IEEE Standard for Software
Safety Plans”.
Require compliance with section 4.32
(“Software Documentation and Source
Listings”) of IEEE Std 982.1™-1988, “IEEE
Standard Dictionary of Measures to Produce
Reliable Software”.
Require compliance with sections 2
(“Software Safety Design Analysis”) and 3
(“Software Safety Code Analysis”) of IEEE
Std 1228-1994, “IEEE Standard for Software
Safety Plans”.
After “identification” add “with a checksum
with strong error detection properties,
e.g., an MD-5 hash code”; the working group
should discuss minimum specifications.

After “identification” add “ and a copy on
permanent machine readable media”.


The bullets for “operator’s manual” and
“user manuals” should either be replaced by
or annotated with a reference to IEEE Std
1063-2001 and a requirement that the
documentation comply with IEEE Std 1063-2001
.
Eliminate (or move to where it’s in context)
“or by providing information on how to
securely configure a particular IT product
within a system.”

Change vendors” to “COTS vendors” or “voting
equipment vendors”.



Paragraph break with the sentence beginning
"COTS products require updates…"
Develop a set of meaningful requirements.


Develop a set of meaningful requirements.

Develop a set of meaningful requirements.




This subsection [?] lists generic threats to which a voting
system may be subject. It is, of course, not possible to
enumerate all threats, but this establishes a lower bound on
Develop that must be defended against.
the threats a set of strong requirements.
Add a description of required security
measures for general access.
Delete or clarify this item.



Define uncommon acronyms at first use, and
perhaps at subsequent uses separated by more
than a single page.
Indent and number the 3 points.

Change “data” to “date”.
[Minimum r]equirements for the content of
audit records are described in Section
[4.5.4] of the Standards.
eliminate or replace second reference in
same sentence
Reduce the scope of the introductory text in
5.1.3.5.1.
Eliminate the section.

Change “may” to “shall”.
Change “scenario consist” to “scenario
consists”.
Change “consisting” to “comprised”.
Remove 'DRE'




renumber this article
Change to 5.3-6 through 5.3-7.
Add figure number.
Align 9 and 10 with earlier entries.
User adjustibility in a polling place is not
a good idea. Requirements should be able to
be met with multiple ballot styles for the
voter to choose at the beginning of a voting
session.
use the word "selections"
As indicated

The system shall check user inputs for
acceptability, e.g., check for inputs that
seem to be in error (such as putting [an]
Arabic number in a name field) and alert
voters to the error.
add item "e" before: Electric power
……level".
make 26 high case
Superscript 26 at end of sentence.
When memory devices are used to store votes
and which voters or poll workers will handle
as part of normal use, these memory devices
shall be tested both [while] installed and
separately from the host equipment.

Make footnote 26 superscripted.
Should be a) and b)
Renumber to be consistent.
should be I/O


Remove the word "punching or"

delete punch card
remove "punch or "

Working group should examine list for errors and omissions.

Requirements for configuration management
apply [to all voting systems subject to the
Standards] regardless of the specific
technologies employed . These system
components include [but are not limited to]:

The Quality Assurance Program shall, at a
minimum, address the topics [indicated]
below.
The vendor shall provide a description of
the procedures and related conventions for [
] maintaining information about
configuration management [ ].
a. [Parts] and materials to be used in
voting systems and components [are suitable]
for the intended application.
b. [Special] tests, if needed, [are
designed] to evaluate the part or material
under conditions accurately simulating the
actual operating environment;
The vendor shall declare the scope of the
system’s functional capabilities, thereby
[summarizing and] establishing the
performance, design, test, manufacture, and
acceptance context for the system.
a. Glossary: A listing and brief definition
of all software module names and variable
names, with reference to their locations in
the software structure. Abbreviations,
acronyms, and terms should be included, if
they are [not included in these Standards or
are] uncommon in data processing and
software development or are used in an
unorthodox semantic;
c. Program Analysis: The results of
software configuration [and] algorithm
analysis[,] and selection, timing studies,
and hardware interface studies that are
reflected in the final software design and
coding.
Rename the section to, perhaps, “History of
Voting Equipment in the USA”




Such ballots [were] then counted by hand,
and a tally of votes [ ] delivered to the
election authority.
A growth in the number of elective offices
and public issues, in the numbers of
political parties and candidates offered,
and in the number of voters led to the
development of [ ] mechanical voting
device[s that] would automatically tally the
votes – thereby, it was thought, eliminating
human error or chicanery while speeding the
counting process. The mechanical device[s]
require[d], however, that choices be arrayed
on a fairly large panel and that voters pull
down levers corresponding to their choices.
The levers trigger individual counters
mounted in the sealed back panel of the
device that are, at the end of the day, read
by election officials.
IBM punchcard technology gave rise to two
types of punchcard voting systems. The
Votomatic device requires voters to insert
an IBM punch card behind the spine of a
booklet that lists the choices and, with
each turn of a page, advance the voter
across the columns of the card. Voters
indicate their choices by inserting a stylus
in a corresponding hole in the spine of the
booklet[ that] dislodges a prescored chad.
After the card is removed from the booklet,
an electronic card reader can then rapidly
read the resulting punchcard ballot. The
[card] readers feed into a computerized vote
tally system. The [alternative] Datavote
device lists the voter’s choices on the
punchcard itself (usually on both sides of
multiple cards). The voter inserts the card
in what amounts to a keypunch device that,
when a lever is depressed next to the
voter’s selection, cuts a hole along the
side of the card. Again, the card is
[mechanically] read and the votes
[electronically] tallied. [ Reading the]
punchcards can occur either in a central
location to which ballots are delivered, at
the polling place after the close of polls,
The IEEE began its effort [as an independent
body] in 2001, giving project authorization
for IEEE P1583 at the June 2001 meeting of
the Standards Board.
These mechanical lever machines[, which by
no means eliminated election chicanery,]
began to be replaced in the mid 20th Century
by a variety of contemporary technologies.

Substitute current version of section 1.1
for Abstract.
... standard and is not itself a mandatory
or regulatory requirement.
add: universal access,


Do you want me to take a crack at rewriting
the abstract? Also, keywords are incomplete,
e.g., DRE, direct recording equipment,
tabulator.
It includes the equipment used to display
and cast a ballot and for precinct ballot
tabulation where the voter [or an election
official] inserts a voted paper based
ballot.
Change to read: This includes preparation of
the equipment for election use, accuracy
testing of the prepared equipment,
transporting of the equipment to and from
the polling site, storage of the equipment
between elections and use of the equipment
by [election officials] in supporting its
use in the election functions.



Make tables 7.5" wide and column widths:
Column 1 - .6
Column 2 - 3.7
Coumn 3 - .7
Column 4 - 2.5
leave out Increased air discharge from 15 to
25 kV and replcae by NC




Change font to Time Roman 10 pt.
Vince Lipsio to rewrite sections 5.6 and 6.6
and submit for review. Rewrite Annex C. Fix
problems listed below, use metric units
consistently, and correct typos. Then submit
a v 5.1 for final ballot.




This will require global changes in the document. It is not
amenable to line-by-line editing.




See comments below on particular sections.
All text related to privacy should be replaced
by a Protection Profile and language referencing
the Common Criteria, including the choice of
testing labs.



All text related to usability should be replaced
by a definition of tests according to the
comment.



All text related to reliability, safety, and quality should
be replaced by references to the
appropriate standards, such as those listed in
the comment.




Add these principles explicitly at the beginning
and amend generally to conform.
Modify draft as appropriate.




Make all statements about how a system should
be constructed advisory. Add a set of
performance criteria at the beginning.




All text related to integrity should be replaced
by a Protection Profile and language referencing
the Common Criteria, including the choice of
testing labs.




Need statement abut System Performance.
For 2002 Technology Standards and Machine
Performance,   the processor of a voting
machine should be greater than a 386. A
Pentium level would be more appropriate.
These inconistencies should be corrected.
     Proposed Resolution
(by the Chair on each comment submitted)
NC - Used in this context the existing definition should stay
the same. Ballot Scanners used in the polling place can only
read the ballot style and layout that the voter uses to cast the
ballot. There is no way for a scanner to differentiate a ballot
ver

Mercuri - Resolution is incorrect. Either the word
"marksense" should be removed from the definition or the
words "or voter verifiable" should be added as per comment,
in order that scanners may be used to confirm all forms of
NC - Needs proposed wording change.
NC - Definition is a the same as in
36CFR1193 and 36CFR1194. It is well
established in federal regulations.
NC - Needs proposed wording change.
NC - Section 4 does not give requirements but rather
describes the voting system generically.

In standards the text is normative and illustrations and
diagrams are illustrative and secondary to the text. While
diagrams should accurately illustrate the t

Mercuri - The diagram is unclear, inconsistent and confusing.
Need to correct as per my comment and proposed change.
NC - Out of scope for this document

Mercuri - The document addresses tallying of absentee votes
incorrectly and allows for the creation of "ghost precincts."
Since the document has gone out of scope in this section,
the bullet should be corrected to address the tallying
appropriately or the entire se




NC - Needs proposed wording change.
NC - Both out of scope and not possible until P1622
completes its work.

Mercuri - Since assessment of these reports is critical to the
security controls applied to the voting system, it is
inappropriate for vendor-proprietary reports to be used.
NC - This section is descriptive and not giving specifications.

Mercuri - Since the paragraph specifies that the integrity of
the primary file must not be affected, it is appropriate to add
the caveat that data in any alternate files must be consistent
with that in the primary file.




NC - Out of scope. This is determined by the relevent
election authority, such as NASED or the individual state
officials.

Mercuri - The lack of any specification regarding updates and
configuration management is a serious security flaw that
must be addressed by the standard.
NC - Needs proposed wording change.


NC - Needs proposed wording change.

NC - Needs proposed wording change.




NC - Needs proposed wording change.



NC - Needs proposed wording change.
NC - Needs proposed wording change.
NC - Needs proposed wording change.
NC - What is universal access? Is the
intent "Universal Design" or "Design for
All"?




AC


Action for editing committee.


Action for editing committee.

Action for editing committee.



Action for editing committee.

Action for editing committee.
NC - Comments proposals will be reviewed
when submitted.




NC - Return to commentor for specific
wording suggestion for the draft.




NC - Return to commentor for specific
wording suggestion for the draft.




NC - Return to commentor for specific
wording suggestion for the draft.


NC - Return to commentor for specific
wording suggestion for the draft.


Action for editing committee.


NC - Not a comment. Editorial error in entry.
Action for editing committee




NC - See Annex C "Protection Profile"




NC - See Sections 5.3 & 6.3.




NC - See relevant sections for reliability,
safety and quality. Comment is already
reflected in existing text.




NC - Return to commentor for specific
wording suggestion for the draft.
NC - Return to commentor for specific
wording suggestion for the draft.




NC - Return to commentor for specific
wording suggestion for the draft.




NC - See Annex C "Protection Profile"




NC - Return to commentor for specific
wording suggestion for the draft.
NC - Needs proposed wording change.




Action for editing committee.




NC - Need the commentor to provide specific wording
suggestions for the document.
Reference Information   Reference Information
C - Definition chaned as follows:                                 The final sentence was not added in
                                                                  V5.0. Need to correct as you said you
93.80. Recall Issues (with Options): The A process that           would.
allows voters to remove their elected representatives from
office prior to the expiration of their terms of office. Often,
the recall involves not only the questio
   IEEE P1583
COMMENT SUBMISSION FORM

           Date: 9-30-03 P1583 Draft 5.0 August 2003
                 Document:
            Commenter        Clause/           Paragraph          Type of                       Comment
            and Number     Subclause         Figure/ Table       comment
                                                                 (General/
   #                                                             Technical/
                                                                  Editorial)   COTS - "These devices and software are
       1 df1                     3        No. 26                     E
                                                                               exempted from certain portions of the
                                                                               qualification testing process so long as
                                                                               such products are not modified in any manner
                                                                               for use in the voting system."
       2   PPLX-001              3        Section 3.                 E         In discussing the definition of COTS, this
                                          Definition # 26                      section goes on to say, “These devices and
                                                                               software are exempted from certain portions
                                                                               of the qualification testing process so long
                                                                               as such products are not modified in any
                                                                               manner for use in the voting system.” In
                                                                               general it is not a good idea to discuss
                                                                               policy in a definition. In particular,
                                                                               doing so here raises the question, which
                                                                               portions of the testing process are
                                                                               “certain” portions from which testing is
                                                                               exempted.
       3   Lipsio-6D            3.26                         E                 Second sentence is not part of the
                                                                               definition. Whether or not my later
                                                                               comments on COTS are accepted, “These
                                                                               devices and software are exempted from
                                                                               certain portions of the qualification
                                                                               testing process so long as such products are
                                                                               not modified in any manner for use in the
                                                                               voting system” does not belong in the
                                                                               definition.
4 Corry-023             5.1.1      p. 20, last para.   E   Last three sentences should be separate
                                                           paragraph.




5    Lipsio-80         5.1.3.6.5                       E   COTS software was already covered in 5.1.1.

6 wfw -001                3              26            E   COTS, whether modified or not must be tested
                                                           at least to system level.
7    VCW-02             5.1.1      2nd to last para    E   The COTS products may also be subject to a security
                                                           evaluation themselves; such evaluations can support the
                                                           voting system evaluation process.
8    MercuriD50 -        7.13                          T   Provision is made in the standard for update for COTS
     078 (new)                                             products releases, but there is no such provision for
                                                           updating or decertifying non-COTS voting system
                                                           components if such have been revealed to be insecure.
9 schneidewind - 002      3        Pg. 10              T   COTS Hardware and software should not be
                                   Line 26                 exempted from qualification testing.

                                   Definition 26           This exemption should not be included in
                                                           Definitions. The exemption is not a
                                                           definition.

10   Lipsio-7B         5.1.2.3                         E   Some cross-reference of how the standard
                                                           addresses each threat would be useful.
11   df12                 D                            E   Only sorted by current VSS 2002. Later,
                                                           most will want this sorted by this standard.
                                                           This section should also be provided with
                                                           the same information sorted by the numbers
                                                           in this standard.
12   df13                         D                         G       Annex D does not include the new items that
                                                                    were not included in VSS 2002. (The ITA's
                                                                    need to do this anyway -- why not have
                                                                    everyone work off the same information and
                                                                    not have all manufacturers and ITA's do this
                                                                    separately?)
13   HD-023                       D                         G       There are many incorrect cross references

14   MercuriD50 - 030             D     entire list       General   List needs to be compared with FEC VSS to ensure
     (formerly mercuri-                                             correctness of correlation.
     085)



15 Gough-003                      Gen                               The most specific area of concern with this document is that
                                                                    it does not consider how a DRE system would handle
                                                                    provisional ballots. Voted provisional ballots would need to
                                                                    be stored on the same memory card, but in a separate
                                                                    memory file than the regular votes being cast on Election
                                                                    Day. Each provisional voter's stored ballot would need to be
                                                                    linked to an identification number. This number would also
                                                                    appear on the required provisional voting affidavit, which
                                                                    must be completed before the provisional ballot is cast. The
                                                                    regular vote totals would be transmitted Election night, but
                                                                    not the provisional ballot votes.

                                                                    Following the election, the Board would conduct a
                                                                    verification process for all provisional voters, using the
                                                                    information contained on the provisional voting affidavits.
                                                                    The Board would need the ability to accept or deny each
                                                                    individual provisional voter's ballot during this process. The
                                                                    Board would also need the ability to accept a provisional
                                                                    ballot in part, if the provisional voter is registered but voted
                                                                    in a wrong precinct containing different districts. The system
                                                                    would need to be secure so that an individual provisional
                                                      T             voter's ballot could be processed, but not viewed or printed.
16 Gough-004
                                                                    Provisional ballots are defined on page 8.
                          3.7.4
                                                                    However, the definition for 3.24 on page 5,
                                                                    "Challenged Ballot," also refers to provisional
                                                      T             ballots.
17 Gough-008   4.4.1               On page 15 under Precinct Voting, would need to add
                                   provisional voting (PV) information to include the recording
                                   of the cast ballot in a separate memory file, linked to the
                                   provisional voter identification number. The provisional
                                   voter ballot counter (PVBC) would be incremented for every
                                   record added to this memory file. The provisional voters'
                                   cast ballot information would be accumulated on one
                               T   memory card, but would not be tabulated, nor would the
18 Gough-005
                                   On page 36 under Processing Accuracy, it states
               5.2.2.1.b
                                   that following an election a consolidated report
                                   would be generated containing absentee,
                               T   provisional and other votes.
19 Gough-006
                                   On page 70 under Voting Variations, provisional
               5.6.5.2
                                   ballots are listed as a variation that may not be
                                   supported by a voting system due to the different
                               T   lawschart on page 251 indicates that provisional
                                   The in the 50 states.
20 Gough-007   7.2.2
                                   voting is covered in HAVA, Sections 302a-d. The
                                   chart also indicates that the 2002 FEC VSS and
                                   this document do not cover the implementation of
                               T   provisional ballots.
21 Gough-009             4.5       Beginning on page 16 under Reporting Subsystem, would
                                   need to add PV information to include the printing of the
                                   PVBC number only. No PV voting results would be
                               T   tabulated or printed until after the post election PV
22 Gough-010   4.5.5               verification process.
                                   4.5.5 On page 17 under Access to Election Data, would
                                   need to include information on limited access to the PV
                                   memory files, which would be necessary in order to
                               T
23 Gough-011
                                   authorize the tabulation of the valid cast provisional ballots.
               5.1.3.2.5.d         On page 26 under Vote Secrecy (DRE Systems), processing
                                   provisional ballots after an election would rely on a linked
                                   association between voter identification number and the
                               T   CVR. The security issues raised in this section would need
24 Gough-012                       to be modified.
                                   On page 30 under Time, Sequence, and Preservation of
               5.1.3.5.6.a
                                   Audit Trails, the provisional voting concerns in this
                               T   paragraph are the same as 5.1.3.2.5.d.
25 Gough-013   5.2.1.2             On page 35 under DRE System Standards, the voting
                                   devices would also need to retain a redundant copy of the
                               T   provisional ballot CVRs.
26 Gough-014   5.2.2.2         5.2.2.2 On page 36 under Memory Stability, the DRE
                               systems would also use redundant memories to store
                           T   provisional ballot CVRs.
27 Gough-015   5.5.3.2.a       On page 37 under DRE Systems Standards, the same
                               considerations would be applied to provisional ballots.
                           T
28 Gough-016   5.2.7.b         On page 39 under Availability for DRE Systems, the same
                               listed considerations would be applied to provisional ballots,
                               except that the votes from the consolidation of provisional
                           T   ballot CVRs from multiple units would not be tabulated, and
29 Gough-017   5.6.3           only the total PVBC number wouldDocument Retention, the
                               5.6.3 On page 68 under Data and be reported.
                               Board would need the ability to enter data into the third
                               linked field (AF) of every stored provisional ballot in order to
                               tabulate the valid provisional ballots. However, the first
                               linked field (identification number) would be read-only
                           T   information and the second field (provisional voter CVR)
30 Gough-018   5.6.5.1         would be 69 under Functions, the provisionalor printed.
                               On page protected from being altered, read ballot
                               information would also need to be stored in an indestructible
                               location. However, the votes would not be tabulated and the
                           T   information would be stored in the previously mentioned
31 Gough-019   5.6.5.2         three-linked-memory-field format.
                               5.6.5.2 On page 69 under Variations, this document should
                               address all variations under this section, especially
                               provisional ballots, in order to have a working solution to
                           T
32 Gough-020
                               accommodate the different laws in the 50 states.
               5.6.6           5.6.6 On page 70 under Ballot Counter, each voting unit
                               would need to have their own ballot counter for their
                               redundant storage of the CVRs, and there would be one
                               master ballot counter for the device that consolidates the
                               information from all voting units. Separate counters
                           T   following this same logic would be provided for provisional
33 Gough-021   5.6.8           5.6.8 On page processing and Functions, this section would
                               ballots and the 72 under Voting storage of this information.
                               include the previously mentioned voting functions for
                           T   provisional ballots.
34 Gough-022   5.6.8.2.b       On page 74 under Activating the Ballot (DRE Systems),
                               provisional ballots would be added to the ballot type list.
                           T
35 Gough-023   5.6.8.3.3       5.6.8.3.3 On page 75 under DRE Systems Standards, this
                               section would incorporate previously outlined provisional
                           T   voting standards.
36 Gough-024   5.6.9                                          On page 76 under Post-Voting Functions, processing the
                                                              provisional ballots would be listed under this section.
                                                   T
37 Gough-025   5.6.9.2                                        On page 77 under Producing Reports, the only Election Day
                                                              report that would be generated for provisional ballots would
                                                              be the total number of ballots cast. Following the post-
                                                              election verification process, provisional vote totals would be
                                                              generated as well as a report of all provisional voter
                                                   T          identification numbers along with the corresponding
38 Gough-026   7.6.2                                          information entered Access Control Measures, this section
                                                              On page 126 under in the AF.
                                                              would include the previously mentioned post-election access
                                                    T         that would be necessary to process provisional ballots.
39 Corry-149             7.1                             T
                                   2nd para., last sentence   Presently the test agency is required to do
                                                              the vendor's work for them if the vendor
                                                              doesn't want to be bothered.



40 Corry-150             7.1.1     Last diamond         T     No mention is made of security during
                                                              development. Development phase is a
                                                              vulnerable time for security failures.


41 Corry-151           7.1.1.1     d.                   T     Need source code to evaluate software
                                                              design.
42 Corry-152           7.1.1.1     New para.            T     Need to know the developmental history and
                                                              testing done by the vendor.

43   RGH 118           7.1.1.1.a                        E     Need clarification on what is required in a
                                                              "System Configuration Overview." Does this
                                                              refer to the system overview in sec. 7.2? If
                                                              so, shouldn't the language be consistent
                                                              from one section to the next?
44 Corry-153           7.1.1.3     2nd para.            E     A summary would suffice as well as an
                                                              abstract. Give them choice.



45   RGH 119           7.1.1.3                          G     What is required in a document abstract? Is
                                                              a TDP table of contents sufficient or does
                                                              the TDP require a summary of each included
                                                              documnt?
46   Dill-42              7.1.1.3                         T


47   MercuriD50 - 026     7.1.2     last sentence       General   Since some states rely soley on the federal certification,
     (formerly mercuri-                                           escrow should be required, not recommended.
     066)

48   Dill-43              7.1.3                           T       Technical data package should be public.



49   RGH 120              7.10.1                          G       Personnel training requirements are
                                                                  dependent on system configuration and
                                                                  customer requirements and cannot be
                                                                  accurately documented.
50 Corry-190              7.10.1    First paragraph       T       Breakdown is needed as to requirements per
                                                                  election jurisdiction and per precinct.




51 Corry-191              7.10.1    Last paragraph        T       Training levels and backgrounds of people
                                                                  working on voting systems are critical
                                                                  factors in both maintenance and security.

52   RGH 121              7.10.2                          G       Training requirements should be decided by
                                                                  client jurisdictions rather than vendors.
53 Corry-192              7.10.2    First sentence        T       Orientation and training must be in both
                                                                  system operation and security.


54 Corry-193               7.11     5th diamond           E       New versions can only be released to
                                                                  customers after certification. That needs to
                                                                  be specified.
55   RGH 112               7.11                           E       Clarification is required on what
                                                                  constitutes a "discrete system component."
56   MercuriD50 - 029      7.11     between bullets 3   General   Need to identify customers who need updates.
     (formerly mercuri-             and 4
     079)
57 Corry-194   7.11.1    1st para., c.   T   Versions must be uniquely named for
                                             identification.
58 Corry-196   7.11.2    2nd para., c    T   Again, versions need to be uniquely named.
59 Corry-198   7.11.7    a.              T   "tools" is amibuous

60 Corry-199   7.11.7    b.              T   "tools" is amibuous



61 Corry-200   7.11.7    c.              T   "tools" is amibuous

62 Corry-204   7.12..3   c.              T   Need name and contact information if this is
                                             to be of any value.
63   RGH 122   7.13.b                    G   Should change releases include references to
                                             internal configuration item identifiers?
64 Corry-154   7.2.2     a.              T   Need information about supported languages
                                             if applicable, e.g., DRE system.




65 Corry-156    7.3      c.              T   Ambiguous who the "user" is. Think the
                                             administrator or election officials should
                                             be the only ones who can bypass or
                                             deactivate functions.
66 Corry-157    7.3      d.              T   Again "user" is ambiguous.



67 Corry-158    7.3      e.              T   Again "user" is ambiguous.




68 Corry-159   7.4.2     Add e.          T   Usually there are test points and test
                                             procedures for such equipment for
                                             troubleshooting.
69 Corry-160   7.5.1     First sentence   T   Software can't run without the hardware, and
                                              different hardware will do different things.




70   RGH 123   7.5.3.d                    E   Need a better definition of what constitutes
                                              a "software item." Is this a subsystem (such
                                              as an election reporting program)? Or an
                                              individual software requirement. Also,
                                              subsection 1) requires clarification. What
                                              is meant by "Software requirements performed
                                              by the item?"
71 Corry-161   7.5.6.1   Add d)           T   Need to know the error codes produced by the
                                              system and what to do for each.

72 Corry-162   7.5.6.2   Add h)           T   Nothing about test routines and expected
                                              outputs.
73 Corry-163   7.5.7.1   First sentence   T   I am not sure what HIPOs are in this context
                                              and I can't see that such jargon adds to the
                                              reader's understanding here. Conversely, it
                                              is very likely voting systems will
                                              incorporate databases and it is essential
                                              that the entity relationship diagrams (ERD)
                                              be included in any overview.
74   RGH 124   7.5.7.2   f3               G   Documenting the "response and response time"
                                              for each system input seems excessive.

75 Corry-164   7.5.7.2   a.               T   No mention of included libraries used in the
                                              design decisions.

76   RGH 125   7.5.8     d                E   What information is required in an "entity
                                              relationhsip diagram?"
77 Corry-165   7.5.8     c.               T   Databases don't include files. They do
                                              include many other features that must be
                                              described.




78   RGH 111   7.5.9.1                    T   only applies to interfaces defined as being
                                              for the purpose of an EDI
79 Corry-166              7.5.9.2   b.4)                 T       Nanoseconds? Spare me. The MKS SI standard
                                                                 unit for time is a second.



80 Corry-167              7.5.9.2   c.2)                  T      Message encryption is part of formatting.
81 Corry-170                7.6                           E
                                    3rd para., 1st sentence      Incomplete sentence.



82   Dill-3                7.6                           T       The technical data package needs to include
                                                                 a threat analysis.




83   Dill-4                7.6                           T       Evaluation requires knowledge of extent to
                                                                 which system depends on “security through
                                                                 obscurity”




84
     Sklein-043            7.6      All                  T       The documentation required in 5.1.3.1 needs
                                                                 to be identified in the TDP.




85   MercuriD50 - 027     7.6.1     Add sentences at   General   It is important to also include vendor-related access to
     (formerly mercuri-             end                          access control policy.
     073)
86 Corry-171              7.6.2     2nd para.            E       Repeated word "access".



87   RGH 126              7.6.2                          G       Vendor can provide suggested procedures and
                                                                 tools for an effective access control policy
                                                                 but access control is ultimately the
                                                                 responsibility of the client jurisdiction.
88 Corry-172              7.6.5     b.2)                 E       Clarify that policies and procedures need to
                                                                 remain current as well as effective over
                                                                 time.
89 Corry-174   7.6.5   b. 6)                E     Detailed, as in obfuscation, usually isn't
                                                  helpful in technical documents. Add
                                                  requirement for clarity. Also, these
                                                  activities "are" prohibited not "should be."



90   RGH 127   7.6.5                        G     Telecommunications and data transmission
                                                  security is addressed on a client by client
                                                  basis and can only be documented in general
                                                  terms.
91 Corry-173   7.6.5   b. 4)                T     There are certainly many other types of
                                                  attack possible on a voting system than
                                                  denial of service. Voting system should
                                                  recognize or at least log other types of
                                                  attacks.




92 Corry-175   7.6.6   First sentence       E     Detailed, as in obfuscation, usually isn't
                                                  helpful in technical documents. Add
                                                  requirement for clarity.


93 Corry-176   7.6.6   Last paragraph       E     Sounds nice but needs a bit of beefing up.




94 Corry-177   7.7.1   Numbering             E    Numbering goes haywire after a. 3)
95 Corry-178   7.7.1                         T
                       Last para., 2nd sentence   The present wording puts an unnecessary and
                                                  unjustified burden on the ITA. If the vendor
                                                  can't supply test data the system should not
                                                  go forward.
96 Corry-179   7.8.5   b)                   E     Ambiguous who the "operator" is. Think what
                                                  is meant here is the sysadmin and election
                                                  officials, not a voter.


97   RGH 128   7.8.5   g)                   G     Delivery schedules are determined on a per
                                                  customer basis. Schedules are set based on
                                                  the needs of the individual customer.
 98 Corry-180   7.8.5     c)   T   Same ambiguity as 7.8.5.b) as to "operator"
                                   with additional problem that intervening in
                                   the voting system operations must not
                                   introduce any errors into balloting that has
                                   occurred.



 99 Corry-181   7.8.6     a)   E   I don't understand what is meant by system
                                   "acquisition" in this context. Suggest the
                                   word be deleted.




100 Corry-182   7.8.6     b    T   Upgrades must be tested and they are a great
                                   time to introduce hacks so they must be done
                                   in a secure manner.



101 Corry-183   7.8.7     c)   T   Recovery procedures are essential here.




102 Corry-184   7.8.7     d)   T   Again ambiguous use of "operator" Also,
                                   security procedures must be enforced by the
                                   election jurisdiction.


103 Corry-185   7.9.1     d)   T   Other systems besides DRE may transmit
                                   election data over a network.

104   RGH 129   7.9.2.1        G   a) database functions and preventative
                                   maintenance depend on the equipment
                                   available at a jurisdiction. The tasks
                                   detailed in subsection a) vary from customer
                                   to customer and cannot be documented. B) The
                                   number and skill level required to perform
                                   tasks is dependent on client experience.
105   RGH 130              7.9.2.2     a and b         G       Should steps to correct software
                                                               deficiencies and configuration management
                                                               processes be included in end-user
                                                               documentation?
106 Corry-186              7.9.4.1     b)              T       Parts and materials must be identified by
                                                               both size and location.
107 Corry-189               7.9.5      c)              E       Clarify.


108   MercuriD50 - 028      7.9.5      end of list   General   Add facilities for ballot and audit storage retention.
      (formerly mercuri-
      078)


109 Corry-187               7.9.5      a)              T       Supplies and requisite maintenance are not
                                                               included.



110 Corry-188               7.9.5      b)              T       The factor here is distance from the
                                                               maintenance people to the voting systems
                                                               they may need to service.


111   Lipsio-72              3.98                      E       Voter-verified has been used in an IEEE
                                                               publication while "Voter-verified" has not,
                                                               giving the former precedence. This is not
                                                               merely semantic because it was a source of
                                                               confusion in the non-resolution of a number
                                                               of comments made to the previous draft.

112   Dill-19              5.1.3.2.5   item a          E       Paper record specification s overly prescriptive.


113   Dill-20              5.1.3.2.5   item b          E       Paper record specification s overly prescriptive


114   Dill-21              5.1.3.2.5   item d          E       Instructions for paper ballots are unclear. Does the printer
                                                               have to randomize the order of the ballots?
115   HD-006               5.1.3.2.5   e               G       text includes "Vote Verifiable Audit Record"
                                                               which is inconsisent with the previous
                                                               paragraphs.
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
             Proposed Change

                                                   Proposed Resolution
                                              (by the Chair on each comment submitted)                 Special Task Group
Delete sentence. I do not believe that is                                                STG on COTS
appropriate in a reference section defining
COTS.


Remove the text in quotes.                                                               STG on COTS




Delete the second sentence.                                                              STG on COTS
Start new paragraph: [As] COTS products                     STG on COTS
require updates due to a detected security
breach or vulnerability [the] voting system
vendor must provide a method to assess the
impact of COTS updates on the voting system,
as well as a method for providing notice and
distribution of updates to purchasers[,
testing facilities, and election officials
and boards]. Where COTS products are known
to be inherently risky ([e.g.,] memory leaks
in the C++ language), vendors must
adequately describe the control methods they
have employed to ensure these risks have
been mitigated.
Eliminate “and software” from the first                     STG on COTS
paragraph and eliminate item “a”.
I would drop the last sentence.                             STG on COTS

delete second space before "voting system"                  STG on COTS



System changes that have resulted from identification of    STG on COTS
insecure voting system components must be propagated to
all systems currently deployed. (This might be more
appropriate in the configuration management section, or a
different section under maintenance.)
Eliminate the exemption.                                    STG on COTS




Find how each threat is addressed and cross-                STG on Cross References
reference that.
Add another Annex that sorts Annex D by                     STG on Cross References
P1583 sections.
Big project, but go through and add the                                                                                            STG on Cross References
specifications that are new with this
standard.



Verify and correct any cross referencing                                                                                           STG on Cross References
errors,
This must be done prior to the next draft -- insufficient time   NC - specific language to make change not provided. Agree STG on Cross References
to provide content in this comment.                              that detailed comparison is needed and any volunteer to
                                                                 help is appreciated.

                                                                 Mercuri - No time to review correlation for this draft I am
The Cook County Board is required by Illinois                    willing to offer assistance with this section prior to the next
                                                                 draft. Recommend a task group be appointed to develop a
                                                                 NC -                                                              STG on Provisional Balloting
State law to provide provisional voting on Election
                                                                 wording proposal for this issue.
Day. Even though other states without provisional
voting may use this document, it would be
beneficial if provisional voting was addressed.
The following references do not cover all the
sections in the document that would be affected by
DRE provisional voting, however, they are
believed to be the most important sections that
would need to be amended.

Sections in the document that mention provisional
ballots and would require coordinated change are
identified by comments Gough-008 to Gough-
(sections 4.4.1, 4.5, 4.5.5, 5.1.3.2.5.d, 5.1.3.5.6.a,
5.2.1.2, 5.2.2.2, 5.5.3.2.a, 5.2.7.b, 5.6.3, 5.6.5.1,
5.6.5.2, 5.6.6, 5.6.8, 5.6.8.2.b, 5.6.8.3.3, 5.6.9,
5.6.9.2, 7.6.2). Additional sections that may deal
with provisional balloting and may require change
                                                                 NC - See Gough-003. Recommend a task group be                     STG on Provisional Balloting
This section in the document mentions provisional appointed to develop a wording proposal for this issue.
ballots and may require change to deal
appropriately with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
                                                       appointed to develop a wording proposal for this issue.




This section in the document will require change to
appropriately support provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
This section in the document mentions provisional appointed to develop a wording proposal for this issue.
ballots and may require change to deal
appropriately with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
This section in the document mentions provisional appointed to develop a wording proposal for this issue.
ballots and may require change to deal
appropriately with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
                                                       appointed to develop a wording proposal for this issue.
This section in the document mentions provisional
ballots and may require change to deal
appropriately with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
This section in the document mentions provisional appointed to develop a wording proposal for this issue.
ballots and requirse change to deal appropriately
with provisional ballots.
This section in the document mentions provisional NC - See Gough-003. Recommend a task group be                  STG on Provisional Balloting
ballots and requirse change to deal appropriately appointed to develop a wording proposal for this issue.
with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
This section in the document mentions provisional      appointed to develop a wording proposal for this issue.
ballots and requirse change to deal appropriately
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
This section in the document mentions provisional appointed to develop a wording proposal for this issue.
ballots and requirse change to deal appropriately
with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
                                                       appointed to develop a wording proposal for this issue.

This section in the document mentions provisional
ballots and requirse change to deal appropriately
with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
This section in the document mentions provisional appointed to develop a wording proposal for this issue.
ballots and requirse change to deal appropriately
with provisional ballots.
This section in the document mentions provisional NC - See Gough-003. Recommend a task group be                  STG on Provisional Balloting
ballots and requirse change to deal appropriately appointed to develop a wording proposal for this issue.
with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
                                                       appointed to develop a wording proposal for this issue.

This section in the document mentions provisional
ballots and requirse change to deal appropriately
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
This section in the document mentions provisional      NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
ballots and requirse change to deal appropriately      appointed to develop a wording proposal for this issue.
with provisional ballots.
This section in the document mentions provisional NC - See Gough-003. Recommend a task group be                  STG on Provisional Balloting
ballots and requirse change to deal appropriately appointed to develop a wording proposal for this issue.
with provisional ballots.
                                                       NC - See Gough-003. Recommend a task group be             STG on Provisional Balloting
                                                       appointed to develop a wording proposal for this issue.

This section in the document mentions provisional
ballots and requirse change to deal appropriately
with provisional ballots.
This section in the document mentions provisional NC - See Gough-003. Recommend a task group be                  STG on Provisional Balloting
ballots and requirse change to deal appropriately appointed to develop a wording proposal for this issue.
with provisional ballots.
Rewrite last sentence to read: [If the                                                                           STG on TDP
vendor's developmental test data is
incomplete, the test agency [may require
additional developmental testing by the
vendor before conducting qualification
testing.]
Add security requirement. ¨ Vendor practices                                                                     STG on TDP
for managing the configuration [and
security] of the system during development
and for modifications to the system
throughout its life cycle.
Reword: d. Software design[, source code,]                                                                       STG on TDP
and specifications;
Either new sub-paragraph n. or insert after                                                                      STG on TDP
f.: Vendor developmental history and
testing.
                                                                                                                 STG on TDP




The TDP shall include a detailed table of                                                                        STG on TDP
contents for the required documents, an
abstract [or summary] of each document and a
listing of each of the informational
sections and appendices presented.
                                                                                                                 STG on TDP
All contents of the technical data package shall be                                                                     STG on TDP
submitted in electronic formats that can be read and
displayed by widely available software applications.
Therefore, it should be required that all technical        NC - Out of scope. Add to list for consideration in future   STG on TDP
documentation presented for certification and acceptance   documents.
testing be placed in escrow such that it is accessible
throughout all continued use of the voting system by       Mercuri - Comment stands, escrow should be required, not
purchasing municipalities.package shall be
The technical data                                         recommended.                                                 STG on TDP
available in its electronic format to any
U.S. citizen who requests it, at nominal
cost.
                                                                                                                        STG on TDP



The vendor shall specify the number of                                                                                  STG on TDP
personnel and skill level required to
perform each of the following functions [for
their voting system with a breakdown as to
requirements per election jurisdiction and
per precinct]:
Personnel recommendations will also include                                                                             STG on TDP
notes specifying that all individuals with
access to the voting system undergo a
background check and be U.S. citizens.
                                                                                                                        STG on TDP

The vendor shall specify requirements for                                                                               STG on TDP
the orientation and training [in both system
operation and security] of the following
personnel:
5th diamond: Releasing new versions of the                                                                              STG on TDP
system to customers [after the new versions
are certified by the testing authority];
                                                                                                                        STG on TDP

* Identification and notification of customers who may     NC - Outside scope of the TDP                                STG on TDP
require updates due to system changes that could affect
proper operations.                                         Mercuri - Maintenance must include configuration
                                                           management information propagated to the purchasers
                                                           where changes could affect security or reliability of the
                                                           voting system. Comment should be reconsidered by
                                                           working group.
c. [Uniquely] name versions.                   STG on TDP

c. [Uniquely] name versions.                   STG on TDP
a. Specific [hardware and software] used,      STG on TDP
current version, and operating environment;
b. Physical location of the [hardware and      STG on TDP
software], including designation of computer
directories[, databases,] and files;

c. Procedures and training materials for       STG on TDP
using the [voting system].
c. [Name and contact information of the]       STG on TDP
individual who conducted the test;
                                               STG on TDP

a. The performance characteristics of each     STG on TDP
operating mode and function in terms of
expected and maximum speed, throughput
capacity, start-up time, maximum volume
(maximum number of voting positions[,]
maximum number of ballot styles supported[,
and languages supported if relevant, e.g.,
DRE system), and processing frequency;
c. Required capabilities that may be           STG on TDP
bypassed or deactivated during installation
or operation by [election officials] shall
be clearly indicated;
d. Additional capabilities that function       STG on TDP
only when activated during installation or
operation by [election officials] shall be
clearly indicated; and
e. Additional capabilities that normally are   STG on TDP
active but may be bypassed or deactivated
during installation or operation by
[election officials] shall be clearly
indicated.
Add e.: Test points and procedures.            STG on TDP
The vendor shall describe the function or      STG on TDP
functions that are performed by the
[hardware and contained] software programs
that comprise the system, including software
used to support the telecommunications
capabilities of the system, if applicable.

                                               STG on TDP




Add d) A tabulation of all error codes and     STG on TDP
messages produced together with the required
fix for each.
Add h) Test routines and expected outputs.     STG on TDP

Delete HIPOs and add ERD: This overview        STG on TDP
shall include such items as flowcharts,
[entity relationship diagrams (ERD), data
flow diagrams, and other graphical
techniques that facilitate understanding of
the programming specifications.

                                               STG on TDP


Add: a. Module and unit design decisions, if   STG on TDP
any, such as algorithms [or libraries] used;

                                               STG on TDP

c. Identification and description of all       STG on TDP
database entities and how they are
implemented physically (e.g., tables, disk
partitions, segment layout and distribution
of tables and indexes across drives, disk
mirroring, triggers, indexes, roles,
auditing, stored procedures, documented
logical and physical ERD, and associated
system scripts, e.g., Korn shell);
refer to comments                              STG on TDP
4) [MKS SI] units of measurement (such as                                             STG on TDP
meters, kilograms, [seconds) and other units
potentially associated with the vendor's
voting system, e.g. currency such as
dollars];
2) Message formatting [and encryption];                                               STG on TDP
Information submitted by the vendor is [to]                                           STG on TDP
be used by test authorities to assist in
developing and executing the system
qualification test plan.
The security specification shall including a                                          STG on TDP
document analyzing each of the specific
threats of section 5.1.2.3, including a
description of the defenses against those
threats, the consequences of failure of the
defense, and the available options for
recovering from such failure.
The specification shall also include a list                                           STG on TDP
of all data that must be kept secret to
ensure the security of the system, and a
list of the roles of those holding these
secrets (e.g., should poll workers have a
password fo altering election data at the
precinct).
Insert a new section 7.6.1 titled “System                                             STG on TDP
Design” and stating “The vendor shall
provide the documentation identified in
5.1.3.1 explaining how the system is
designed to counter and protect against the
threats identified in 5.1.2.3.
Circumstances where vendors or their agents have access N.C. - Covered in 5.1.3.2.4   STG on TDP
to the equipment must also be detailed in the access control
policy and procedures documentation.                         Mercuri -
The vendor also shall define and provide a                                            STG on TDP
detailed description of the methods used to
preclude unauthorized access to the [ ]
control capabilities of the system itself.
                                                                                      STG on TDP



b. 2) Policies and processes used by the                                              STG on TDP
vendor to ensure that such protection is
updated to remain [current and] effective
over time;
b. 6) A [clear, i.e. sans unessential          STG on TDP
jargon, and detailed description of all
activities that [are] prohibited during
system setup and during the timeframe for
voting operations, including both the hours
when polls are open and when polls are
closed.
                                               STG on TDP



b 4) A detailed description of the system      STG on TDP
capabilities and procedures to be employed
by the jurisdiction to diagnose [or at least
log] the occurrence of [ ] denial of service
[or similar] attack[s], to use an alternate
method of voting, to determine when it is
appropriate to resume voting over the
network, and to consolidate votes cast using
the alternate method;
The vendor shall provide a clear, i.e. sans    STG on TDP
unessential jargon, and detailed description
of the following additional procedures
required for use by the purchasing
jurisdiction:
This documentation shall be prepared such      STG on TDP
that these requirements can be [effectively]
integrated by [an election] jurisdiction
into local administrative and operating
procedures.
Resequence in proper order.                    STG on TDP
Delete the second sentence. Generating test    STG on TDP
data for the vendor should not be an ITA
responsibility.

b) Provides procedures that clearly enable     STG on TDP
the [system administrator and election
officials] to assess the correct flow of
system functions (as evidenced by system-
generated status and information messages);
                                               STG on TDP
c) Provides procedures that clearly enable     STG on TDP
the [system administrator or election
officials] to intervene the system
operations to recover from an abnormal
system state [without introducing errors in
the existing ballot counts or inadvertently
or deliberately destroying ballot records or
images;
a) Defines the procedures required to          STG on TDP
support system [ ] installation[ ] and
readiness testing (these procedures may be
provided by reference, if they are contained
either in the system hardware
specifications, or in other vendor
documentation provided to the testing
authority and to system users);
b) Describe procedures for providing           STG on TDP
technical support, system maintenance and
correction of defects, and for incorporating
[and testing] hardware upgrades and new
software releases [in a secure manner].

c) Detailed Examples: Detailed scenarios       STG on TDP
[and remedial procedures] that outline
correct system responses to faulty operator
input. Alternative procedures may be
specified depending on the system state;
d) Manufacturer's Recommended Security         STG on TDP
Procedures: This appendix shall contain the
security procedures that are to be executed
by the system [administrator and enforced by
the election jurisdiction].
d) How transmission of [election] data over    STG on TDP
a network are performed (DRE systems[ and
others] where applicable);
                                               STG on TDP
                                                                                                                       STG on TDP



b) Size [and location];                                                                                                STG on TDP

c) Organizational affiliation (i.e.,                                                                                   STG on TDP
[election] jurisdiction, vendor) of
qualified maintenance personnel.
d) Recommendation regarding secure storage and retention NC - Outside scope of this standard                           STG on TDP
of ballot materials (whether paper or electronic) and other
audit records.                                              Mercuri - Section 7.9 pertains to system maintenance and
                                                            therefore requires material on ballot and audit storage
                                                            retention and maintenance. Comment should be
a) Recommended number and locations of spare reconsidered by working group.                                            STG on TDP
devices[, ] components[, or supplies] to be
kept on hand for repair [or maintenance]
purposes during periods of system operation;

b) Recommended number and locations                                                                                    STG on TDP
[(distance)] of qualified maintenance
personnel who need to be available to
support repair calls during system
operation;
Define "Voter-verified" as a primary term                                                                              STG on Voter Verified Paper Ballot
and, if "Voter-verified" be retained, make
that a synonym for "Voter-verified".




for voter verification, ensure that the votes on the paper                                                             STG on Voter Verified Paper Ballot
record can be concealed from others during and after the
voting process.
… vor voter verification, ensure that paper copies of                                                                  STG on Voter Verified Paper Ballot
cancelled ballots do not reveal the voter's selections, by
providing means to destroy or conceal the ballot.
 the order and times of casting the votes can be obscured                                                              STG on Voter Verified Paper Ballot
either during the process of storing or retrieving them.
Either add in parenthesis or substitute                                                                                STG on Voter Verified Paper Ballot
"paper copy of voter selections"or change
previous sections.
Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

                Date: 9-30-03 P1583 Draft 5.0 August 2003
                      Document:
                 Commenter Clause/ Subclause        Paragraph     Type of
                and Number                       Figure/ Table   comment
                                                                 (General/
       #                                                         Technical/
                                                                  Editorial)




            5
            6
            7
            8
            9
           10
           11
           12
           13
           14
           15
           16
           17
           18
           19
           20
           21
           22
           23
           24
           25
           26
           27
           28
           29
           30
           31
           32
           33
           34
           35
           36
           37
           38
           39
           40
           41
           42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
Comment
Proposed Change
     Proposed Resolution
(by the Chair on each comment submitted)
Reference Information   Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

               Date: 9-30-03 P1583 Draft 5.0 August 2003
                     Document:
                Commenter Clause/ Subclause        Paragraph     Type of
               and Number                       Figure/ Table   comment
                                                                (General/
       #                                                        Technical/
                                                                 Editorial)
           1
               Sklein-005           5.1       Para 5.1.3.2.5        T




           2
               Sklein-006           5.1       Para 5.1.3            T




           3   Alice - 016         5.1.1                            G


           4   RGH 008             5.1.1      last pp               G


           5   Simons - 001        5.1.1      bulleted list         G



           6   Simons - 002        5.1.1                            G
                                              the sentence that reads, "The security countermeasures i
7    MercuriD50 - 031     5.1.1   Paragraph following       General
     (formerly mercuri-           bulleted list
     087)

8    Aragon - 09          5.1.1        third bullet           T




9 Corry-022               5.1.1                         T
                                  p.20, 3rd para., 3rd sentence




10   Lipsio-10            5.1.1   Para. 1               T


11   Lipsio-11            5.1.1   Para. 5               T


12   Lipsio-12            5.1.1   Para. 5               T




13   Lipsio-13            5.1.1   Para. 6               T




14   Lipsio-14            5.1.1   Para. 7               T
15   Lipsio-15            5.1.1     Para. 7               T




16   Lipsio-16            5.1.1     Para. 7               T


17   Lipsio-17            5.1.1     Para. 7               T




18   RGH 007              5.1.1     last paragraph              T

19   Lipsio-18            5.1.2     Bullet 2              T




20   Lipsio-19            5.1.2     Bullet 3              T


21   RGH 009              5.1.2.2   2nd pp                      G



22   Lipsio-1A            5.1.2.2                         T



23   MercuriD50 - 032     5.1.2.3   List of Assumptions       General
     (formerly mercuri-
     090)




24   MercuriD50 - 033     5.1.2.3   E.                        General
     (formerly mercuri-
     093)
25   Adler-001     5.1.2.3   1st paragraph         T




26   Adler-002     5.1.2.3   New 1st paragraph      T




27   Adler-003     5.1.2.3   Threats                T




28   Aragon - 10   5.1.2.3   List of Assumptions   T



29 Corry-024       5.1.2.3   No. 2                  T



30   Dill-15       5.1.2.3   A-2                   T




31   Dill-16       5.1.2.3   A                     T


32   Dill-17       5.1.2.3   Add E-5               T


33   Dill-18       5.1.2.3   E-4                   T
34
     Sklein-058    5.1.2.3   5.1.2.3               T




35
     Sklein-001    5.1.2.3   A-2(h)                T/E
36   Adler-005    5.1.3   New paragraph   T




37
     Sklein-002   5.1.3   All             T




38
     Sklein-003   5.1.3   All             T
39
     Sklein-056          5.1.3    All                  T




40   Adler-004          5.1.3.1   New 2nd paragraph    T




41   Simons - 006       5.1.3.1   the entire section   T




42 GHM - 001        5.1.3.1                            T
43   MercuriD50 - 034         5.1.3.1.1        Paragraph            General
     (formerly mercuri-
     096)




44   MercuriD50 - 035         5.1.3.1.3        End of section       General
     (formerly mercuri-
     097)

45 Corry-025               5.1.3.1.3 and *.4                          G




46   GHM - 004            5.1.3.15.1           e                      T




47   GHM - 002            5.1.3.2.1                                   T




48 Corry-026                  5.1.3.2.3        Missing g. as approvedT in previous comments no. 105 on s




49   Lipsio-1B                5.1.3.2.3                         T


50   RGH 010                  5.1.3.2.3        items d & e            T




51   RGH 011                  5.1.3.2.3        a                      T


52 Corry-027                  5.1.3.2.4        f.                     T
53 Corry-028        5.1.3.2.4   Missing h. as approvedT in previous comments no. 108 on




54   Simons - 009   5.1.3.2.4   item c               T


55   Simons - 010   5.1.3.2.5   a)                   G




56   Simons - 011   5.1.3.2.5   b)                   G




57   Simons - 012   5.1.3.2.5   c)                   G




58   Simons - 013   5.1.3.2.5   d)                   G




59   Simons - 014   5.1.3.2.5   e)                   G
60   PPLX-013           5.1.3.2.5       Vote Secrecy         G
                                        (DRE Systems)
                                        Section
                                        5.1.3.2.5




61 MercuriD50 - 036         5.1.3.2.5   d.                General
   (formerly mercuri-
   099)
62 Adler-015                5.1.3.2.5   a) - d)              T




63 Corry-029                5.1.3.2.5                         as
                                        Missing changes to c. T approved   in previous comments




64   RGH 013                5.1.3.2.5   d.                   T

65   vcw-04                 5.1.3.2.5   a) - d)              T




66   vcw-05                 5.1.3.2.5   e)                   T
67   wfw - 003          5.1.3.2.5         e.          T




68   GHM - 003      5.1.3.2.5                          T


69   PPLX-014       5.1.3.2.5       Vote Secrecy    T and E
                                    (DRE Systems)
                                    Section
                                    5.1.3.2.5
                                    Subsection e




70   wfw - 004          5.1.3.3                       T




71   Simons - 015       5.1.3.4.1   a)                 G

72   Simons - 016       5.1.3.4.1   e)                 G
73   MercuriD50 - 071     5.1.3.4.1   first paragraph       General
     (formerly mercuri-
     165)




74   MercuriD50 - 005     5.1.3.4.1   d.                    General
     (formerly mercuri-
     009)


75   MercuriD50 - 037     5.1.3.4.1   e.                    General
     (formerly mercuri-
     100)




76   Lipsio-1C            5.1.3.4.1   Item “a”          T

77   Lipsio-1D            5.1.3.4.1   Item “a”          T

78   Lipsio-1E            5.1.3.4.1   Item “a”          T




79   Lipsio-1F            5.1.3.4.1   Item “a”          T




80   Lipsio-20            5.1.3.4.1   Item “b”          T
81   Lipsio-21            5.1.3.4.1   Item “c”          T
82   Lipsio-22            5.1.3.4.1   Item “d”         T




83   Lipsio-23            5.1.3.4.1   Item “e”         T

84   RGH 014              5.1.3.4.1   item d                 T


85   Simons - 017         5.1.3.4.2   the entire section     G


86   RGH 015              5.1.3.4.2   par. 3                G/T



87   MercuriD50 - 039     5.1.3.4.2   last paragraph       General
     (formerly mercuri-
     102)




88   MercuriD50 - 038     5.1.3.4.2   next to last         General
     (formerly mercuri-               paragraph
     101)


89   Lipsio-24            5.1.3.4.2   Para. 1          T




90   Lipsio-25            5.1.3.4.2   Para. 1          T




91   Lipsio-26            5.1.3.4.2   Para. 1          T
92   Lipsio-27            5.1.3.4.2   Para. 2           T

93   Dill-23              5.1.3.4.3                            E

94   Simons - 018         5.1.3.4.3   the entire section       G


95   MercuriD50 - 040     5.1.3.4.3   paragraph              General
     (formerly mercuri-
     103)




96   wfw - 005            5.1.3.4.3   First paragraph       General




97 Corry-030              5.1.3.4.3   2nd sentence             T




98   Lipsio-28            5.1.3.4.3                     T




99   Lipsio-29            5.1.3.4.4   Item “c”          T
100   Lipsio-2A       5.1.3.4.4   Item “f”        T



101   Lipsio-7C       5.1.3.4.5   Item “a”            E

102   Lipsio-2B       5.1.3.4.6   Item “b”        T



103   Lipsio-2C       5.1.3.4.6   Item “b”        T




104   Adler-022       5.1.3.5.1   1st paragraph       G




105   df7             5.1.3.5.1   Paragraph 4         G




106   HD-007          5.1.3.5.1   3rd paragraph       G




107   PPLX-015    5.1.3.5.1       System Audit        G
                                  Purpose and
                                  Context.
                                  Section
                                  5.1.3.5.1
108   Simons - 019       5.1.3.5.1                         T
                                     the first and second sentences




109
      Sklein-053         5.1.3.5.1   First paragraph       T/E




110   Simons - 020       5.1.3.5.3   the first sentence     G



111   GHM - 005      5.1.3.5.3                              G




112   RGH 018            5.1.3.5.4   item a            note to Bob

113   HD-021             5.1.3.5.4   b.                     T




114   RGH 019            5.1.3.5.4   item b                 T
115   RGH 020                  5.1.3.5.4     item c            T



116   RGH 021                  5.1.3.5.4          b.           T




117   vcw-06                   5.1.3.5.4     b)                T




118   GHM - 006            5.1.3.5.4                           T




119   HD-008                   5.1.3.5.5     b.3               T




120   Jhulshof-001             5.1.3.5.5     b5                T

121   RGH 024                  5.1.3.5.5     b.2               T


122   RGH 022                  5.1.3.5.5     items b1 & b3    T/E



123   RGH 023                5.1.3.5.5.a.3                     T


124   MercuriD50 - 042         5.1.3.5.6     f)              General
      (formerly mercuri-
      106)
125   MercuriD50 - 041     5.1.3.5.6   between d) and e)       General
      (formerly mercuri-
      105)


126   RGH 025              5.1.3.5.6   item d              note to Bob




127   Dill-41              5.1.3.5.6   item h                    T


128   Jhulshof-002         5.1.3.5.6   b5                        T

129   Jhulshof-003         5.1.3.5.6   d                         T

130   Lipsio-2D            5.1.3.6                         T




131   RGH 026              5.1.3.6     last 3 bullets            T



132
      Sklein-004           5.1.3.6     Several                  T/E




133 Corry-032              5.1.3.6.1   2nd sentence              T



134   RGH 027              5.1.3.6.2   entire clause            G/T
135   MercuriD50 - 043     5.1.3.6.2   paragraph              General
      (formerly mercuri-
      107)

136   MercuriD50 - 046     5.1.3.6.2   end of paragraph       General
      (formerly mercuri-
      111)
137   Lipsio-7E            5.1.3.6.3   Item “a”                 E

138   vcw-07               5.1.3.6.3   a)                       E




139   vcw-08               5.1.3.6.3   d)                       E




140   HD-010               5.1.3.6.3   d.                       G

141   Lipsio-2E            5.1.3.6.3   Item “b”           T


142   RGH 028              5.1.3.6.3   items a-d                T




143   RGH 029              5.1.3.6.3   b,d                      T


144   HD-009               5.1.3.6.3   a.                       T

145   Lipsio-2F            5.1.3.6.4                      T




146   Lipsio-30            5.1.3.6.4                      T




147   MercuriD50 - 044     5.1.3.6.5   e.                     General
      (formerly mercuri-
      109)
148   Lipsio-81            5.1.3.6.7                      E




149   MercuriD50 - 045     5.1.3.6.7     d. 2)          General
      (formerly mercuri-
      110)




150   Dill-24              5.1.3.6.8                      E
151   Lipsio-83            5.1.3.6.8                      E

152   RGH 030              5.1.3.7.1     item c          G/E
153   RGH 031              5.1.3.7.1     a                T


154 Corry-033              5.1.3.7.2     2nd sentence     T



155   Dill-44              5.1.3.7.2                      T




156   Adler-006             5.1.3.8      New section      T




157   Adler-007            5.1.3.8.1     New section      T




158   Adler-008            5.1.3.8.1.1   New section      T
159   Adler-009   5.1.3.8.1.2   New section     T




160   Adler-010   5.1.3.8.2     New section     T




161   Adler-011   5.1.3.8.2.1   New section     T



162   Adler-012   5.1.3.8.2.2   New section     T




163   Adler-013   5.1.3.8.2.3   New section     T



164   vcw-03       5.1.5.2      4th paragraph   E
165
      Sklein-037   6     Second bullet          T



166 JL - 005       6.0                          E




167   Dill-30      6.1                          T

168   Dill-31      6.1   Paragraph 3            T


169   Dill-32      6.1   Paragraph 3            T


170   Dill-33      6.1   Paragraph 4            T
171   Lipsio-0C    6.1   Para. 3            T




172
      Sklein-038   6.1   Fourth paragraph       T


173
      Sklein-039   6.1   All                    T
174
      Sklein-059                  6.1           All                 T




175   RGH 086              6.1 par. 3 & 6.1.2                     G/T/E
                                 par. 1


176
      Sklein-040                 6.1.1          Last paragraph      T



177   RGH 087                    6.1.2          items e & f        G/T



178   HD-017                     6.1.3          e)                  G


179   RGH 088                    6.1.3          item e             G/E



180   MercuriD50 - 014           6.1.3          e)                General
      (formerly mercuri-
      038)

181   Dill-34                    6.1.3          Item c              T




182
      Sklein-041                 6.1.4          First paragraph     T

183
      Sklein-042                 6.1.4          First paragraph     T
184   Adler-014    6.1.5   New section             T




185 MercuriD50 -    C      Entire section          E
    002 (new)              C.7.1
186 wfw -014        C            all            General


187   vcw-26        C      Table 4 and beyond      T




188 Corry-212       C      Title                   T
189 Corry-213            C       Entire section   T




190 MercuriD50 - 001     C       Entire Annex     T
      (new)




191   MercuriD50 -       C       C.7.3            T
      003 (new)




192
      Sklein-050         C       All              T




193
      Sklein-008       C.3.1.1   A.CONNECT        T




194
      Sklein-034       C.3.1.1   All              T

195
      Sklein-009       C.3.1.3   A.MANAGE         T
196
      Sklein-010   C.3.1.4   A.LOCATE          T




197
      Sklein-011   C.3.1.4   A.PROTECT         T




198
      Sklein-012   C.3.3.1   All               T




199   vcw-12       C.3.3.2                     E
200 diaz - 001     C.3.3.2                     T




201
      Sklein-013   C.3.3.3   All               T


202
      Sklein-014   C.3.4     P.ACCESS_LEVEL    T

203
      Sklein-015   C.3.4     P.ADMINISTRATOR   T
204
      Sklein-016   C.3.4   P.ALERT              T




205
      Sklein-017   C.3.4   P.ASSURANCE          T

206
      Sklein-018   C.3.4   P.DATA_AUTHENTICATION T

207
      Sklein-019   C.3.4   P.PHI_ACCESS         T




208
      Sklein-020   C.3.4   P.SECURE_TRANSMISSION T

209
      Sklein-021   C.3.4   P.SESSION_TERMINATION T




210
      Sklein-022   C.3.4   P.VOTER_ANONYMITY    T

211   vcw-13       C.4.1   O.Install            T
212 diaz - 002     C.4.1   O.INSTALL            T
213
      Sklein-023   C.4.1   O.ABORT_SESSION      T

214
      Sklein-025   C.4.1   O.AUDIT_RECORD       T

215
      Sklein-026   C.4.1   O.AUDIT_REVIEW       T




216
      Sklein-027   C.4.1   O.EVENT              T
217
      Sklein-028   C.4.1   O.INSTALL               T




218
      Sklein-029   C.4.1   O.VOTE_VALIDATION       T



219
      Sklein-030   C.4.1   All                     T



220
      Sklein-024   C.4.1   O.AUTH_VOTER           T/E
221 diaz - 003     C.4.2   Para 1                  T

222 diaz - 004     C.4.2   OE.PHYSICAL            T

223
      Sklein-031   C.4.2   OE_PHI_ACCESS           T




224
      Sklein-032   C.4.2   OE.USER_TERMINATION     T


225   vcw-14       C.5.1   Security Requirments   T
226 diaz - 005     C.5.1   FCF.VA                 T




227 diaz - 006     C.5.1                          T



228
      Sklein-033   C.5.1   All                     T
229 diaz - 007     C.5.1.1.1.2.1                         T

230 diaz - 008     C.5.1.1.2.1.1                         T




231 diaz - 019   C.5.1.5.2.3.2




232   vcw-15         C.5.1.5.3                           T

233   vcw-16        C.5.1.5.3.1                          T

234   vcw-17       C.5.1.5.3.1.1                         T
235   vcw-18       C.5.1.5.3.1.1   after C.5.1.5.3.1.1   T


236   vcw-20       C.5.1.6.4.1.3                         E



237 vcw.23            C.5.2                              T
238 vcw-21            C.5.2         Class AGD:           T
                                    Guidance
                                    documents
239   vcw-22          C.5.2                              T


240 diaz - 009        C.5.2                              T



241 diaz - 010        C.5.2                              T
242 diaz - 011          C.5.2            T




243 diaz - 012          C.5.2            T




244
      Sklein-035        C.5.2      All   T




245 diaz - 020     C.5.2.1.1.1.5




246   vcw-24           C.5.2.1.2         T
247   vcw-25         c.5.2.2.2.1.1                   T




248 diaz - 021     C.5.2.2.2.1.1




249
      Sklein-036      C.5.6.2.6.1    All             T




250 diaz - 013             C.7                       T



251 diaz - 014             C.7.1     FAU_GEN.1       T
                                     P.ACCOUNTABIL
                                     ITY
252 diaz - 015             C.7.1     FAU_GEN.2
253 diaz - 016             C.7.1     FAU_GEN.2
254 diaz - 017             C.7.1     FAU_GEN.2
255 diaz - 025     C.7.1             FCS_CKM.1

256 diaz - 026     C.7.1             FCS_CKM.1

257 diaz - 027     C.7.1             FDP_DAU.1
258 diaz - 028     C.7.1             FMT_MSA.1
259 diaz - 029   C.7.1           FMT_MSA.2




260 diaz - 030   C.7.1           FMT_MSA.3




261 diaz - 031   C.7.1           FMT_MSA.3

262 diaz - 032   C.7.1           FMT_SMR.1
263 diaz - 033   C.7.1           FMT_SMR.1
264 diaz - 034   C.7.1           FTP_ITC.1

265 diaz - 035   C.7.2

266 diaz - 036   C.7.2           table

267 diaz - 037   C.7.2           table


268 diaz - 038   C.7.2           table


269 diaz - 039   C.7.2           table



270 diaz - 040   C.7.2           table



271 diaz - 018           C.7.3
272 diaz - 022   C.7.3   Para 2




273 diaz - 023   C.7.3   Para 3

274 diaz - 024   C.7.3   Para 4
                 Comment



Randomization used for privacy protection
must be based on random events or random
noise




Need to incorporate security-relevant
requirements on operating system features
and software style that had been previously
contained in software section. There
especially needs to be protection against
improper input or buffer overflow allowing a
malicious perpetrator to input data and or
executable code.
How can Challenge/Provisional ballots be
verified if not associated to a voter?

"risks have been mitigated" is incomplete


The standard is supposed to address
unauthorized modification of the system.
But there is no full proof way to detect any
unauthorized modification.
This is a far too vague and does nothing to
address the security issues.
Need to add material at end of paragraph.




Original comment language "independent auditability" should
be acceptable. (mercuri-086 in 4.3 comments). The word
"audit" in common parlance is so frequently preceded by
"independent" (or some synonym) as to indicate common
practice, and therefore does not "make policy". The originally
suggested language is permissive: If the equipment
facilitates an independent audit, policymakers would still be
free to permit collusion in the audit, so policy is not
constrained; whereas if it does not provide that capability,
jurisdictions will lack it even if policy permits or requires it.
Therefore to the extent that the original comment does go to
COTS that be properly than discourages
policy, may fact favors rather installed and the inclusion
configured but still not meet requirements
unless latest security patches are
installed.




“acceptable” is not quantifiable and,
therefore, does not belong in a
specification.
In a dedicated device, the operating system
should be, specifically, a real-time
operating system.
The treatment of COTS products contradicts
section 5.1.2.2, “Elements of Security
Outside of Vendor Control”.


A voting machine is device in its own right
(or at least must be tested as such), not an
“IT system” made up of sundry components
thrown together. “specific requirements”
refer to what this draft endeavors to
specify.
There is implied a lack of testing in “COTS
products require updates due to a detected
security breach or vulnerability”; nothing
that requires an update should pass testing.
“The voting system vendor must provide a
method to assess the impact of COTS updates
on the voting system, as well as a method
for providing notice and distribution of
updates to purchasers” is inconsistent with
IEEE Std 1012-1998.




Memory leaks are the result of using C++
language inappropriately; they are not a
risk of a COTS C++ compiler.
Memory leaks can not be tolerated in a real-
time system; normal design of real-time
systems prohibits dynamically allocating
memory after initialization or freeing
dynamically allocated memory.
Memory leaks in C++ is not an example of an
inherent risk in COTS products.
“General Purpose Computing Equipment” is not
controllable, that is, there is no guarantee
that such a device is equivalent to the
device used in V&V, and therefore should be
prohibited (unless each individual unit is
fully tested).
“Any components developed by a voting
jurisdiction” fall outside of this
standard’s scope
Need to consider security certification in
the context of full system (procedures and
technology)

“Data communications security” and “Risk,
response and recovery” are not “outside of
vendor control” and do not, therefore,
belong here.
Add additional assumptions.




Add activation of malicious code
Add additional threat




Add paragraph on assets (i.e., the goal of
the attack)




Threats are based on the stage when a
vulnerability could be exploited instead of
the asset goal. I believe a comment was made
at the Denver meeting to "follow the vote"
which aligns with this philosophy.

Rework of 4.3 comment (mercuri-090). The duty cycle of
voting equipment differs from many other computer systems.
The difference is legitimate matter to consider in the threat
model and should not be dismissed as policy. a
Motivation alone isn't necessarily
security risk. One needs training and
ability as well.

System administrators or others could modify the source
tree.




Development phase could actually include anyone who has
custody of the software prior to installation. Depending on the
procedures in use, this could include the ITAs.



Others may have access to voting machines
Add a threat that a voter or election
official surreptitiously connects an
external device to a voting machine and
tampers with the machine or its data using
functiionality resident on the external
device.




“correctly” should be “incorrectly”
Election Verification




It needs to be made clear that voting
equipment should be dedicated to voting, or
if not dedicated should be converted to
voting by a procedure similar to conversion
between unclassified and classified
processing under “System High Mode” of
chapter 8 (Automated Information Systems) of
the National Industrial Security Program
Operation Manual.




There needs to be a section expressing the
controversy about the suitability of all-
electronic systems to provide protection
against combinations of malicious activity,
human error, and/or equipment failure that
could cause a ballot to be recorded that
differs from the intent of the voter. The
section should also point out that HAVA can
be interpreted as requiring a voter
verifiable paper audit trail.
Voter verified paper needs to be mandatory
under certain circumstances




Requirements assume that all threats can be
prevented instead of allowing for, in
addition, detection and deterrence.




The basic requirement that the vendor design
to counter and defend against the threats is
inadequate, since there is no way to protect
against all of the threats without a voter
verifiable ballot.

---- vendor provides documentation explaining how each of
the threats in Section 5.1.2.3 is countered by the system
design. --- This documentation should be checked by
computer professionals not associated with the companies
who make detailed checks of the software used in the voting
machines. The computer companies should then provide the
more improved software to the county election officials..
Wireless connectivity should specifically be precluded here,
or somewhere else appropriate in this section.




Need to note audit trail for access




These sub-sub-subsections were apparently
deleted and/or renumbered.




System audit generated creation and maintenance of audit
records
reduces the chance of error associated with audit records ---
this is true if the software has been programmed properly.
However, if problems arise due to programming errors or
stealth programming the inaccuracies and capabilities of the
The vendors shall specify the features can be much greater.
access control mechanisms to provide effective voting
system security. --- This information should be checked by
computer professionals not associated with the companies to
Passwords validity.
guarantee its cannot be nulls or common words,
e.g., password, secret, family names, etc.,
and shall contain at least one numeric or
non-alpha character, e.g., ~ ! @ # $ % ^ & *
( ).

Insufficient security.


Authentication of components which
communicate with each other REQUIRES the use
of unencrypted key storage (unless an
operator supplies a key at the time of each
exchange), rendering items d & e
incompatible.
Can't identify each person who has access
granted, it violates secrecy of ballot

Reason needed for role-based accounts.
Default superuser accounts, e.g., root, in
all software or operating systems must be
disabled.




There should be minimal security algorithmic
and key length requirements for the
encryption.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
Because voter verifiable paper ballots are
the only way to protect against the security
threats listed in section 5.1.2.3, voter
verifiable paper ballots must be required,
not mentioned as an option.
This section says, “and if a paper copy of
voter selections is printed for voter
verification, deposit the detached paper
copy into a sealed, opaque “ballot box”.

This section mixes a couple of ideas and is
too prescriptive. First, we point out that
that a printed paper copy may be printed as
the ballot, and not as a later
“verification” or audit. In such a case,
there is strong reason for the voter to take
the ballot away from the voting booth (but
not out of the polling place) for deposit on
one central ballot box. The language of
this section prohibits that. The important
point is that this section must allow either
for the voting station or the voter to
deposit the printed record.

Add ballot integrity.


Prescriptive language (e.g., "a paper copy
of voter selections is printed for voter
verification") which is inappropriate in a
performance standard. The language should,
and already does, indicate that any feature
shall "ensure vote secrecy."
Ballot image may contain embedded code that
could identify sequence even though it does
not have a specific field that does so.


Should also include provisional ballots

"Immediately after the voter chooses to cast his or her
ballot, record all voter selections as a CVR in the
memory to be used for vote counting and audit data
(including CVRs), erase selections from all visual
indicators, buffers and all other temporary storage, and
if a paper copy of voter selections is printed for voter
verification, deposit the detached paper copy into a
sealed, opaque “ballot box”; and" The highlighted
portion indicates only one possible of "voter
verification". Placing a "paper copy" of the ballot into a
In systems providing voter interaction in multiple
languages, the CVR and all copies thereof, and the
Voter Verifiable Audit Record if provided, shall be free
of indications of the language selected by the voter,
except where the system provides and the voter
explicitly selects a multiple-language presentation of
the Voter Verifiable Audit Record.
Paragraph implies that voter verifiable
ballot might be available. Current
technology does not support a cost effective
solution to provide this capability. In
other words it is not reasonalble to provide
this capability.
Voter Secrecy - DRE System section e should read ----
Voter Verification Audit Record must be provided (not may
be provided) so the voter is sure his vote is counted properly
“In systems providing voter interaction in
multiple languages, the CVR and all copies
thereof,
and the Voter Verifiable Audit Record if
provided, shall be free of indications of
the language
selected by the voter, except where the
system provides and the voter explicitly
selects a
multiple-language presentation of the Voter
Verifiable Audit Record.”

This section has an ambiguity in it.

What is meant by “the voter explicitly
selects a multiple-language presentation?”
Does this mean that voter chooses to see a
presentation in more than one language
(“multiple language”) or does it mean that
the voting system supports more than one
language but the voter only sees the one
language the voter selected? We think the
latter is what is intended and suggest that
clarification. Also, we are not sure what
the intent of this element is. If it is
intended to help protect privacy we support
that intent. On the other hand, the ability
to determine what language a group of voters
voted in could be important. One reason for
this is to find out if there are any
systematic problems with the ballot based on
The last sentence is totally open ended and
not useable in its current form. There are
physical tamper resistant methodologies
available on most election systems, but this
sentence is unsupportable or testable.
The only way that the firmware can be
verified is by making it open source.
The vendor is told to provide a mechanism
for security access to the voting devices
after successful completion of election day
testing and the method to identify operaion
of an override feature during the election
cycle if provided. This is not adequate;
minimal security requirements should be
stated in this document.
Add a paragraph that includes the ability of the audit trail to
be used for checking tallying and data collection.




Some manufacturers allow firmware to be changed between
elections, this could occur during system operation must be
prohibited from being changed during system operation
during an election.

Allowance of an override feature is a significant security risk
that should not be permitted under any circumstances.




What applies to firmware applies to all
software.
For security reasons, the identification of
software level must be automatic.
For security reasons, the identification of
software level must be validated.




For security reasons, the software must not
be able to be altered without authorization
.


Ambiguity in “acceptable by NIST”.
Bootstraps must and monitors always reside
permanently as firmware. (Unless we want to
provide for using toggle switches to enter
the bootstrap each time the device is
powered up.)
This makes no sense and is contrary to
common practice in embedded systems; the
operating system and application code
(“election-specific programming” here) are
linked together and installed on the same
ROM(s); when resident on unalterable ROMs,
the bootstrap typically is likewise so
linked in.
Unnecessary and unacceptable security risk
in allowing an override feature.
What is the purpose of keeping election-
specific programming separate from bootstrap
or operating system code?
There is no way to adequately test against
all possible bugs and malicious code in
COTS.
Some systems may not be capable of auditing
"…all process executions and terminations,
and…the alteration or deletion of any
memory, file or database object or entry."
Add additional phrase pertaining to voter initiated deletions, if
recorded.




The voter actually alters election data -- this must not be
audited because of violation of ballot privacy. Why would
anyone be allowed to alter or delete election data?


This contradicts section 5.1.3.6.8. Allowing
unrelated tasks to run simultaneously with
the election software is an absurdly
unacceptable risk that negates and V&V
performed.


Use of “servers and workstations” can not be
permitted, both because of unknown other
software that may be present, and because
the hardware is not controllable, that is,
there is no guarantee that such a device is
equivalent to the device used in V&V, and
therefore should be prohibited (unless each
individual unit is fully tested).
“Unauthorized network connections” must be
disallowed by design.
The system shall be designed such that these
precautions should not be needed.
How is the system going to protect against Trojan horses,
etc.? I don't think there is any way to do it.
The only way to protect against malicious
software is by using voter verifiable
ballots
Add time to logic bombs and also additional material
pertaining to any actions by voters or administrators that
could activate malicious code.




This far reaching all purpose statement adds
confusion rather than entlightening the
reader. Protections described may or may not
be potential threats based on the voting
system design.
Need to specify single or combination of
keypresses and to specify other types of
processes.




By mandated that a voting machine be
designed as a secure system and by
disallowing it to be constructed of sundry
other parts (such as general-purpose
computers, Microsoft Windows, discarded
bleach bottles, pipe cleaners, Microsoft
Access, et cetera), this section is not only
not needed but is a travesty.
Ambiguous (“Prior to” could mean at V&V),
insufficient (any memory used could affect
the performance) and no specification for
what to do in the event of a failure.
“Public network” is not defined.



Sub-items 2-5 all contain the words “in
human readable format” and item 1 should.
There are numerous problems with using
“local time and date”.


“A means to verify that date and time are
correctly set prior to any election” is
inadequate.




Define "indestructible" since it could be
interpreted conservatively to mandate
physical medium that is infeasible.

What's the performance standard for the
audit trail? Should it be single (double,
triple) fault tolerant?
There is a possible conflict between the
last sentence in paragraph 4 of 5.1.3.5.1
"The Technical Data Package shall be subject
to public scrutiny and may not be regarded
as proprietary." and section 7.1.2 which
states the need for putting the same
information into escrow. The implication in
7.1.2 is that the information is not open to
public scrutiny .
Document states "described in section 4" .
Should identify specific section 4.5.4.
(Could also reference section 5.6.4 in
Software and functionality for more
application audit details.
The technical data package, as currently
defined includes database layout
descriptions and program logic flow.
Furthermore, some may consider that the
source code is part of the TDP.   Even if
the source code is not considered part of
the TDP vendors must disclose a great deal
of proprietary information in the TDP.
Making the TDP public will permit anyone to
duplicate a vendor’s system. Doing so
removes the incentive to create good
election systems and this requirement will
have the consequence of degrading the
quality of our electoral systems.
The second sentence says that election audit
trails "present a concrete, indestructible
archival record of all system activity
related to the vote tally, and are essential
for public confidence in the accuracy of the
tally, for recounts, and for evidence in the
event of criminal or civil litigation."
This is a false claim. The audit trail is
meaningless in as far as a recount is
concerned, because there is no way of
knowing if the vote recorded in the machine
accurately reflects the intent of the voter
unless there is a voter verifiable ballot.
Also, the existence of an audit trail should
not be sufficient for generating public
confidence in the accuracy of the tally.

The paragraph should reflect the proper
access control requirements for audit data
that were erroneously stated in Appendix C.




The sentence makes a false claim that audit
trails are essential to ensure the integrity
of a voting system and to retain public
confidence in the election process.
 ---- vendors shall supplement it ( they are referring to the
design construction) with information relevant to the
operation of their specific systems. ---- This supplements
should be conducted by professional computer scientists not
association with the vendor company before the company
provides the voting machines instructions to the county
The M650 conduct the voting and to the operators.
officials whodoes not include the "readiness
report" in its audit log…
The restriction in parenthesis prevents a
function that is provided in many DRE
systems today that is required for ITA
stress testing and is used by jurisdictions
in election specific L&A testing. Note that
this section is in conflict with 5.6.7.2.1
(section f and the concluding paragraphs
below) which allow automatic ballot
generation as long as it can't occur during
election day voting. THIS INCLUSION IS THE
REASON FOR MY NO VOTE. My vote would be Yes
if this were changed.

This is not possible. Correct counting logic
can only be verified by human operators.
With any moderately advanced operating
system, it is not possible for software to
check "all data paths and memory locations"
which might be used in vote recording.
It says that " The ballot interpretation logic shall test and
record the correct installation of ballot styles or formats on
voting devices for the voting precincts at the polling location
and that the ballot logic produces a correct count for each
candidate and issue on the ballot (NOTE: The system shall
not automatically generate voted ballots)".Note that this
section is in conflict with 5.6.7.2.1f) and the paragraphs
below which allow it as long as it can't happen with real
voting. This restriction would severly hamper ITA as well as
"The ballot interpretation logic shall test and record the
correct installation of ballot styles or formats on voting
devices for the voting precincts at the polling location
and that the ballot logic produces a correct count for
each candidate and issue on the ballot (NOTE: The
system shall not automatically generate voted ballots);"
unreasonably restricts systems from conducting test
ballot runs to verify election readiness and shall verify
---- Prior to opening of polls, a system processreliability.
hardware and software status and general readiness for
audit status. ---- The lack of readiness in certain Florida
districts lead many people to leave without voting because
they had other important things to do such as going to work.
Two hour delay in order to vote due to equipment failure is
inexcusable for equipment not working and appropriate
The audit log cannot contain a verification
solutions for this condition must be made available.
that the voting system is in the location
for which it was programmed. This must be a
pollworker function. It can only identify
the location for which it was programmed.

this registration may not have a timestamp
voter privacy
Unless you are talking GPS, the voting
system cannot verify it is in the proper
location
How can the system verify that date/time and
location are correct? Must there be an
atomic clock and GPS receiver attached to
each system?
Codes may be all an audit log contains so
that printing tapes can be an efficient
consideration
replace practical with possible
The use of a secure time/date stamp protocol should
supplement the human readable format.




The audit record "shall be available for
review at all times by authorized election
officials" -- is the iVo audit log available
on screen to an "authorized" operator?

Does this require that electronic copies be made of paper
records?

no real time clock can activate fraudulent
code
audit trail shall not include cvr

Missing requirement.




This is for a "Local Area" Network,
presumably isolated from any external
connection. These security concerns are
ridiculous.
The term “datagram” normally refers to a
message transmitted using the Universal
Datagram Protocol (UDP). Unless it is the
intent of the standard to require use of UDP
(e.g., as distinct from TCP) or to limit its
statements to messages transmitted using
UDP, another synonym for “message” should be
selected and used in this section.

Devices connected to the voting system must
be certified as well. I also question how
this might be enforced?

Voting systems using sonic or physical (via
liquid, for instance) transmission of data
are exempted.
The use of wireless and open air transmissions leaves data
available for monitoring, corruption, and/or jamming,
regardless of error detection etc.. This practice should be
prohibited.
Need to add to access control end of paragraph.



“Datagram” is not the correct term here as
it implies use of UDP.
"The network shall be so configured that only
datagrams authorized and required by the voting system
appear on the physical network medium and that
datagrams from the voting system are not transmitted to
non-voting systems." is not gramatically correct based
on the preceeding sentence. keys will be managed to
               "Encryption
            ensure the keys are not compromised and
            that the keys are changed on a periodic
            basis." is not with introductory
Wording not consistent gramatically correct.
sentence "…systems shall:
Ambiguity in “for sensitive data”.


Item a essentially requires an isolated
network. Why not state that directly? In
either case, items b-d are unnecessary
overkill once the item a requirement is met.

References to encryption and keys are
specific technology implementations

Wording not consistent with introductory
sentence "…systems shall:
Additional requirement to eliminate damage
that could fall through from the existing
requirements.




Additional requirement to eliminate damage
that could fall through from the existing
requirements.



Eliminate "antennas"
This is unnecessary if all communications
are initiated by the voting device.




Reporting of breaches needs to be added.




Item is unclear.
This contradicts section 5.1.3.4.2.

Grammar be confused.
References to encryption and keys are
specific technology implementations

Devices connected to the voting system must
be certified as well. I also question how
this might be enforced?




Election Verification




Voter Verification (cast-as-intended)




Intent Capture
Casting in the Ballot Box




Election Verification (counted-as-cast)




Sealing the Ballot Box



Scrutiny of Sealed Ballot Box




Results Reproduction



Missing "to" before "require additional security
measures"
Expert-based analytic evaluation should
apply as well to information security


The quality of some figures in the draft is poor. For example,
Figure C-1 in subclause 6.2.2.1. Many of the other figures in
Clause 6 are also not very good. They will need to be fixed prior to
submitting the draft for approval by the Standards Board. Failure to
provide publishable figures may result in a disapproval or
conditional approval by the Standards Board and will delay
The standard needs to include a design review and source
code review.
The last sentence of this paragraph would seem to apply to
all systems, not just systems that use public
telecommications networks to transmit official results.
Why is this restricted to threats as of the time the system
was submitted for qualification? Serious new threats are
often exploited almost immediately upon being discovered.
What attacks should be simulated?
Missing requirement (cf. section 5.1.3.6).




However, this does not relieve the testing
authority from addressing the threats
identified in 5.1.2.3 (A-1) and (A-3)
The requirement of 5.1.3.1 needs to be
evaluated by expert analysis
If the system does not include a voter-
verifiable paper audit trail, the test
procedure needs to include evaluation of the
suitability of the system to operate without
such a function.




6.1 states "public" telecommunications
networks, but 6.1.2 doesn't make this
distinction. Only public networks require
these measures.
Protection during storage between elections
also needs to be included here.


It is not possible, in most cases, to
distribute or install new system releases in
response to threats, because certification
is usually required.
Both sentences use the phrase "be
reexamination." The sentence doesn't read
properly.
Grammatical errors: "All subsequent
changes…shall be reexamination. All
changes…shall also be reexamination." Is
this a typo? What's the intended meaning?
Any changes should require reexamination.




We can be more specific about the design information for
custom chips.



EAL-2 is insufficient for the needed
protection
The “generic PP” has no obvious purpose.
P1583 is the draft standard for evaluating
voting systems. The evaluation process does
not provide an opportunity for some entity
to take a “generic PP” defined in this
standard and develop a “specific PP” to be
responded to by a machine vendor.
Design Review Board




Remove references to inappropriate entities

I am confused about the purpose of Annex C.
It needs more clarifications on how it is to
be used, either in design or testing.
Make changes as indicated.




Title isn't clear as to purpose of this
annex.
First, it is my understanding from Berger
that this is now Annex D and a new Annex C
has been added that I have not seen. Second,
undefined conventions, jargon, and
formatting apparently from the security
community are used throughout that make the
meaning and understanding of this annex
impossible as written. Third, there are so
many flaws in the information presented as
to make the annex worthless as currently
presented.
Choice of EAL2 is inappropriate to the
voting system setting.



No mitigation is provided to known Common
Criteria flaw regarding removal of audit
trail data that includes actions by the
voters (a known threat agent group) in order
not to violate ballot privacy.



Protection against collusionary signaling by
a voter to a vote buyer or vote intimidator
to bypass vote privacy should include
consideration of covert channels for
accomplishing the signaling.



This assumption is incorrect. Asumptions
regarding controlled access facilities may
not apply during storage between elections
or at numerous other times, such as between
the time of delivery to the polling place
and the opening of the polls.
The assumptions in 5.1.2.3 need to be
reflected in the PP assumptions.
This assumption is incorrect. Malicious
insiders are among the potential threat
agents. The TOE must provide means of
enforcing separation of duties to preclude
the need for trusting a single individual
not to perform a malicious act.
This assumption is incorrect. Assumptions
regarding controlled access facilities may
not apply during storage between elections.
Also, there are numerous other opportunities
for unauthorized physical access to the
voting machines, such as the time between
delivery to the polling place prior to
election day and the arrival of the polling
place election officials on election day.
Even the detailed monitoring of the polling
place by election officials when they are
busy with numerous voters may not satisfy
this assumption.
This assumption is incorrect. Asumptions
regarding controlled access facilities may
not apply during storage between elections
and at other times. The TOE must be capable
of providing some of its own protection

The threat agents identified do not track
the threat agents implied in 5.3.2.1




"Threats To Be Addressed By The Operating Environment" is not specific about enviroment.
Change title from:

C.3.3.2 Threats To Be Addressed By The
Operating Environment

to:

"C.3.3.2 Threats To Be Addressed By The
Operating Environment (Non-IT Environment)
The threats identified in this section do
not clearly track the threats identified in
5.3.2.1.
This policy statement does not make sense in
the context of a voting system.
In general, administrative procedures should
be such that at least two people participate
in every procedure.
This policy statement does not make sense in
the context of a voting system. TOE
activity can not be monitored during periods
of storage and delivery to polling places
and storage ares. There is nobody present
to hear an audible alarm or see a visual
alarm. A combination of TOE features and
externally applied/inspected seals is used
when the TOE is being stored or delivered.
The requirements must reflect this.

The Voting System Standard is this document,
not some external reference.
This is a requirement of the TOE, not of
organizational policy
Before and after the voting, the general
public has a need and right to know all
information in the voting machine, other
that security protective passwords and keys.
However, only authorized election officials
have a need-to-perform the functions needed
to supply that information to the general
public.
This is a requirement of the TOE, not of
organizational policy
This policy statement does not make sense in
the context of a voting system.



Anonymity must be maintained even if the
voter attempts to collude to violate it
O.INSTALL
This appears to be a Non-IT Environment objective.
This objective, as written, does not make
sense in the context of a voting system.
The Voting System Standard is this document,
not some external reference.
Before and after the voting, the general
public has a need and right to know all
information in the voting machine, other
that security protective passwords and keys.
This explicitly includes all audit data,
which is part of the public record of the
election.
This security objective does not make sense
in the context of a voting system. TOE
activity can not be monitored during periods
of storage and delivery to polling places
and storage ares. There is nobody present
to hear an audible alarm or see a visual
alarm. A combination of TOE features and
externally applied/inspected seals is used
when the TOE is being stored or delivered.
This security objective does not make sense
in the context of a voting system. The
system is initially delivered to a
warehouse, where after acceptance it remains
in storage until it is needed for an
election. It is then taken from the
warehouse and delivered to a polling place,
where it is set up and used according to
procedures. It is not clearly ever
“installed” anywhere. After the election,
the system is returned to the warehouse
where it is stored until needed for the next
election and delivered to a (likely
different) polling place.
This statement makes improper use of the
term “CVR”. The CVR is the record of cast
votes. Votes should not be cast until they
have been validated by the voter.
Need to add a new objective on non-
traceability of the cast vote to the voter.


This statement does not say anything.
Most of these objectives appear to be Non-IT objectives. For
instance, which SFR meets OE.ADMIN_TRAINING?
This is an example of an IT security objective that could
be met by an SFR; FPT_PHP
Before and after the voting, the general
public has a need and right to know all
information in the voting machine, other
that security protective passwords and keys.
However, only authorized election officials
have a need-to-perform the functions needed
to supply that information to the general
public.
Before and after an election, the election
and audit data in the TOE is part of the
public record of the election.
Component missing from Security Requirements
This is an explicitly stated requirement, therefore
Section 1 needs to include the conventions that are used
in the PP; assignments, selection, iteration, refinements,
and explicitly stated requirement formats
Add to table:

FMT_SMF.1 - Specification of Management
Functions
Beside being incomplete (with places having
text to be determined (assignment or
selection), these functional requirements
are based on incomplete or incorrect
assumptions, threats, policies, and
objectives. Accordingly, they need to be
completely rewritten
This ‘refinement’ of the requirement needs to be
explained. See comments diaz - 005
All operations need to be identified; assignments,
selections, refinements. For example, a completed
assignment would be bold text within brackets and a
selection would be underlined text. to comply with
Add the following requirement
International Interpretation RI - #65




Add

Add

Add
Add


misspelled authorized


Class AVA: Vulnerability assessment
Change to Administator


Class ATE: Tests add ATE_IND.2


There International Interpretations that apply to several
SARs, and as such need to be annotated. I have
included a few of them.
Change from:

AGD_ADM.1 Election Official / Administrator
guidance

to:

AGD_ADM.1 Administrator guidance
Change from:

ATE_COV.1 Evidence of coverage
ATE_FUN.1 Functional testing

to:

ATE_COV.1 Evidence of coverage
ATE_FUN.1 Functional testing
ATE_IND.2 Independent testing
Change from:

AVA_SOF.1 Strength of TOE security function
evaluation
AVA_VLA.1 Vendor vulnerability analysis
ATE_IND.2 Independent testing

to:

AVA_SOF.1 Strength of TOE security function
evaluation
AVA_VLA.1 Vendor vulnerability analysis
The justification for using EAL-2, as
reflected in C.7.3 is absolutely inadequate
and incorrect. The fundamental requirement
is to protect the integrity of the voting
process from well-financed, sophisticated,
motivated attackers. There is no
requirement that this be attempted with an
EAL-2 operating system, and in fact it may
not be feasible to provide the needed
protection with such a system. Furthermore,
because the standards is primarily for US
voting systems, there is no need to remain
within the EAL structure prescribed by the
Common Criteria international mutual
recognition agreement. Modifying the
assurance requirements from those in the
standard EAL's would have only the effect of
requiring international users to separately
determine their acceptance of US
certification – a non-issue for this
standard.
Add the following requirement to comply with
International Interpretation RI - #3




Add heading
"The documentation shall describe the steps necessary
for secure setup, ballot generation, and opening and
closing of the polls, start-up of the TOE." does not
adequately describe this funciton.
Modify the following requirement to comply
with International Interpretation RI - #51




This is a major example of where EAL-2 is
inadequate. EAL-2 linits the vendor's
search for vulnerabilities to obvious
vulnerabilities and does not require the
search to be systematic. A minimum level
should be AVA_VLA2 or AVA_VLA3 which require
documentation of the disposition of
identified vulnerabilities and that the
search for vulnerabilities be systematic.
There are objectives, threats, assumptions, and policies
listed in Sections 3 and Sections 4 that have not been
mapped.
This policy is not defined in Section C.3.4


O.ENTITY_AUTHENTICATION objective is not
defined in Section C.3.4.
O.ELECTION objective is not defined in Section
OFFICIAL_AUTHENTICATION - Is this an objective
or a policy? It is not listed in either Section.
O.MESSAGE_AUTHENTICATION - This objective is
not defined in Section C.4.
P.AUTHENTICATED_MESSAGE - This policy is not
in Section C.3.4
P.AGREEMENT - This policy is not in Section
C.3.4
O.INSTALL - This objective is dealing with
the TOE being delivered and installed
correctly, whereas this security requirement
is to ensure the policy identified is
enforced and to restrict the ability to
modify the security attributes of that
policy to the authorized administrator.
O.INSTALL - This requirement is to ensure
only secure values are accepted for the
security attributes (of identified policies)
which does not meet this objective.

O.INSTALL - This requirement is to allow the
authorized administrator the ability to
define alternate values (of the identified
policies) which does not meet this
objective.
P.AUTHORIZED_ELECTION OFFICIALS - This
policy is not in Section C.3.4.
P.AUTHORIZATION_RECORDS    - This policy is
not in Section C.3.4.
P.TERMINATION_RECORD    - This policy is not
in Section C.3.4.
P.ENTITY_AUTHENTICATION - This policy is
not in Section C.3.4.
Assumption Rational - Missing the rationale
to support the mapping
CountersThreats - T.PHI_ACCESS - This is not
a threat in Section C.3.3.3.
Environmental Security Objectives -
OE.AGREEMENT - This environment objective is
not in Section C.4.2.
Environmental Security Objectives -
OE.RELEASE - This environment objective is
not in Section C.4.2.
Environmental Security Objectives -
OE.ELECTION OFFICIAL_TRAINING - This
environment objective is not in Section
C.4.2.
Environmental Security Objectives -
OE.ELECTION OFFICIAL.TERMINATION - This
environment objective is not in Section
C.4.2.
Add as 2nd paragraph:

This PP contains the assurance requirements from the
CC EAL2 assurance package. The EAL chosen is
based on the statement of the security environment
(assumptions, threats and organizational policy) and the
security objectives defined in this ST. The sufficiency
of the EAL chosen (EAL2) is justified based on those
aspects of the environment that have impact upon the
assurance needed in the TOE. The administrative staff
is conscientious, non-hostile and well trained
(A.MANAGE, OE.ADMIN_TRAINING). The TOE is
physically protected (OE.PHYSICAL), and properly
and securely configured (O.INSTALL). Given these
aspects, a TOE based on good commercial development
practices is sufficient. EAL 2 is an appropriate level of
The high level design is description of the
TOE, its subsystems, and how the security
functions are implemented. The high level
design does not include the underlying
operating system.
I am not sure I understand this statement or what it is trying to say.

I am not sure this is an appropriate
statement. And I am not sure what the last
sentence means; augmenting the assurance?
             Proposed Change



Add item e as follows: “ Randomization used
for privacy protection must be based on
random noise or random events and not solely
on pseudo-random algorithms. Such
algorithms are in fact deterministic and
provide repeatable values when started from
the same initial conditions or “seed
parameters”. Repeatable sequence values can
facilitate linking of votes to individual
voters if the sequence of voters using a
machine is known.”
Add a section stating “The operating system,
the language processors, and/or the software
shall be designed to prevent excessive or
improper input from allowing the input of
data or executable code by a malicious
perpetrator.


Delete. Until further assessment of
Challenged/Provisional relationship to
voter.
add " or minimized (ex., use bounds checking
sw to analyze and identify memory leaks)"

Add a requirement for a voter verifiable
paper ballot.


Replace sentence with the following:
"Underlying products, such as operating
systems, database systems, firewalls,
network devices, web browsers, smart cards,
biometric devices, general purpose
application components, libraries, and
hardware platforms, that are crucial to the
correct and secure operation of the entire
system must be thoroughly tested. This
includes COTS systems. In addition, there
must be a line by line code review of ALL
software that interacts with the voting
system in any fashion. This is required
because of the potential risk of malicious
code."
All of these activities must be fully documented and retained
for all certified systems.



Third bullet change to:
"To ensure the integrity and independent auditability of the
ballots as cast by the voters."




Notwithstanding the fact that system
certifiers can rely upon the prior
validations of the individual components of
the system [ ] provided they are properly
installed and configured [with the latest
security patches], there must still be an
evaluation of the integrated system to make
certain that security holes have not been
left or created during the integration
process.
Mandate compliance with IEEE Std 1228-1994,
“IEEE Standard for Software Safety Plans”

Change “operating systems” to “RTOS s (Real-
Time Operating Systems)”.

Change “COTS product may” to “COTS products
shall”. Mandate compliance with section
4.3.11 (“Previously developed or purchased
software”) of IEEE Std 1228-1994, “IEEE
Standard for Software Safety Plans”.
Change “IT systems are procured and
constructed to meet specific requirements
and typically use existing” to “The system
may”.


Mandate that testing preclude any security
breach or vulnerability; mandate compliance
with section 4.3.11 (“Previously developed
or purchased software”) of IEEE Std 1228-
1994, “IEEE Standard for Software Safety
Plans”. Mandate COTS be subject to the
specifications of IEEE Std 1008™-1987
(R1993), “IEEE Standard for Software Unit
Testing”. Add reference to IEEE Std 982.1™-
1988, “IEEE Standard Dictionary of Measures
to Produce Reliable Software”.
Bring into conformance with Annex D (“V&V of
reusable software“) of IEEE Std 1012-1998,
“IEEE Standard for Software Verification and
Validation”, e.g., “Reusable software (in
part or whole) includes software from
software libraries, custom software
developed for other applications, legacy
software, or commercial-off-the-shelf (COTS)
software. The V&V tasks of Table 1 are
applied to reusable software just as they
are applied to newly developed software.
However, the inputs for these tasks may not
be available for reusable software, reducing
visibility into the software products and
processes.“
Eliminate “(ex. memory leaks in the C++
language)”

Eliminate “ex. memory leaks”.




More appropriate would be "security
vulnerabilities in Microsoft products".
Strike the words “General Purpose Computing
Equipment”.




Strike the words “Any components developed
by a voting jurisdiction”

add "Certification should be based on both
the technology and procedural aspects
comprising a system as defined in the
vendor's documentation (see Sec 5.1.3.1)."
Strike the bullets “Data communications
security” and “Risk, response and recovery”.


3. The systems may be unattended for periods of time when
they could be at greater risk. 4. The need for anonymity of
voter ballot reduces or entirely removes many traditional
forms of auditing commonly used for other electronic
systems (such as ATMs in banks).


E-4. A voter or election official is able to activate Trojan
horse or other malicious code that has been previously
installed, in order to affect or manipulate ballot contents or
vote totals.
4. The persons attempting to compromise the
election process could be insiders with full
knowledge of the election system including,
but not limited to, political operatives,
vendor personnel, polling place workers, or
election administrators.
For elections, the principal asset is
governmental power. That power is
transferred by the results of counting voted
secret ballots. Hence, integrity of the
voted ballot is critical through the entire
process from capturing the voter's intent,
casting it into the ballot box, counting it
to produce the election results, and finally
retaining it to resolve disputes.

The principal vulnerabilities to the voted
secret ballot are (1) undetected compromise
of election integrity, (2) compromise of
ballot secrecy, and (3) denial of voting
service. [need to reorder existing 5.1.2.3
threats underneath this taxonomy]
Add: "4. The systems may be unattended or in idle storage
for periods of time."

Rewirte sentence: 2. There are people who
are motivated [and have the training and
ability] to compromise the election process.

Someone with legitimate or illegitimate access to the system
source modifies the voting system software directly, or
modifies the programming environment to modify the
software. The modification could be malicious code that
affects voting directly, or it could enable the subsequent
installation of malicious code. B into "Software Development,
Perhaps we can merge A and
Testing, and Distribution"
A third party is able to activate a Trojan
Horse remotely via RF, IR, network,
telephone, or other remote control.
A voter, poll worker, technician, or election official …
Add item E-5: A voter or election official
is able to surreptitiously connect an
external device to a voting machine and
tamper with the machine or its data using
functiionality resident on the external
device. An example of such an attack would
be connection of a handheld computer to a
voting machine through a device that
simulates the physical connection of a
smartcard used by the voting machine for a
purpose such as voter authorization.
Revise per comment
Election Verification: These standards
address election verification requirements
to ensure that all ballots are counted-as-
intended. Any change to election data
throughout the voting process shall be
detected by the voting system.
Insert a section in 5.1.3 entitled
Hardware/Software Security. The section
should state: “Voting equipment should be
exclusively dedicated to voting. However,
if shared equipment is used, all persistent
storage devices and media not dedicated to
voting should be physically removed or
disconnected from the equipment prior to its
use for any activity related to voting, with
the exception of the read-only firmware that
interfaces the hardware to the operating
system. All non-persistent storage,
including all input/output channels, must be
cleared before and after the voting activity
by writing at least three cycles of the
sequence of a character, its complement, and
a random character into all addressable
locations. The hardware, software, and
documentation provided must facilitate
conversion of the system between non-voting
and voting configurations. For example, all
hard drives must be removable so voting-only
drives can be substituted for non-voting
drives, suitable software must be provided
to clear non-persistent storage and
input/output channels, and users must be
instructed to safeguard voting-only devices
Insert a new section in 5.1.3 titled “Voter
Verfiiable Paper Audit Trail”. The section
should state “There is controversy about
the suitability of all-electronic systems to
provide protection against combinations of
malicious activity, human error, and/or
equipment failure that could cause a ballot
to be recorded that differs from the intent
of the voter. Also, Section 301(a)(2)(B) of
HAVA can be interpreted as requiring a voter
verifiable paper audit trail. Accordingly,
it is recommended that all DRE systems
include a voter-verifiable, paper audit
trail satisfying the requirements for such
features defined in other sections of this
standard.”
Add to the section created under comment SK-
4 above: A voter verified paper audit trail
is mandatory for any system in which any of
the following conditions is found: 1.
Either the system software or any COTS used
as either a system component or development
tool, including compilers, libraries, and
other tools, is too complex to clearly and
thoroughly evaluate at the source code level
to ensure absence of backdoors and other
malicious code or means of introducing
malicious code. 2. All other security,
accuracy, integrity, and availability
requirements are not satisfied clearly,
easily, and without any question or
requirement for interpretation. 3. - There
are any reports or significant suspicions
that similar technology may have failed to
record all ballots exactly as cast. 4. -
There is any question whatever about the
ability of all using jurisdictiions to
easily and completely satisfy all
assumptions regarding supervision of
machines and relevant personnel at all times
machines are in use, regarding fully secure
storage of machines between elections, and
regarding other procedures intended to
Defensive countermeasures are designed to
thwart attacks that exploit the principal
vulnerabilities in attempts to compromise
the assets. Countermeasures can be designed
into the data protocol, the
software/hardware implementation, and/or the
election administrative procedures.
Countermeasures typically fall into three
categories: protection, detection, and
deterrence.
Add the following sentence: "Because there
is no full proof way to protect against all
of the threats identified in Section 5.1.2.3
via software and testing alone, the voting
system must include a voter verifiable paper
ballot or its equivalent."
Wireless connectivity, both in the development and
deployment of election systems poses significant security
risks, such that it is necessary to require that the use of
wireless communications devices be prohibited in any stage
of election equipment construction, development and use.



There should be an audit record provided in the audit trail for
all access, along with a reporting mechanism for access
violations.

Four levels of subheadings should be
sufficient. Also, current formatting makes
it difficult to keep chapters, sections,
subsections, and sub-subsections straight.
Then sub-subsections appeared to have
disappeared. That makes it very difficult to
track changes. Suggest limiting to four
levels and formatting correctly before
resubmitting for review.




Add new item "g. Capability to restrict use
of common words, e.g. password, secret, and
common dictionary words for passwords and to
enforce strong password policies by the
voting jurisdiction need to be provided by
the vendor."
Add: “and shall log each and every access
with a timestamp and all data specified in
these features”
Item e should be removed, or perhaps
substituted with direction that components
incorporating unencrypted keys must be
physically secured.


"must log all activities associated with
ballot processing and restrict access based
on the type of request being asked for"
Add reason: f. Role-based and discretionary
access control [so that superuser accounts,
e.g., root, sysadmin, sa, ora, etc.,
accounts can be deactivated.]
Add h. All default superuser accounts, root,
sa, ora, sysadmin, etc., in all operating
systems and software installed in the voting
system shall be disabled and individual
accounts with such superuser privileges as
are essential and necessary set up by the
vendor. The voting jurisdiction shall be
encouraged to disable the vender superuser
accounts and establish its own.

Add minimal requirements for key length and
algorithmic robustness.

Delete "if a paper copy of voter selections
is printed for voter verification" and
replace "deposit the detached paper copy"
with "deposit the voter verifiable paper
ballot"
Delete "if a paper copy of voter selections
is printed for voter verification" and
replace "print text on the paper copy" with
"print text on the voter verifiable paper
ballot"
Replace "any paper copy of voter selections
that may be printed for voter verification"
with "the voter verifiable paper ballot"


Replace "any paper copies of voter
selections that may be printed for voter
verification" with "any voter verifiable
paper ballot"

Replace "and the Voter Verifiable Audit
Record if provided" with "and the voter
verifiable paper ballot". Also, replace the
last occurrence of "Voter Verifiable Audit
Record" with "Voter Verifiable Paper
Ballot".
This section must allow either for the
voting station or the voter to deposit the
printed record.




…from being associated with a voter or the order that the
ballots were cast, while also ensuring ballot integrity.
Delete language.




c. Ensure that ballot image data does not
contain any fields [or codes, including but
not limited to timestamps,] that identify
the sequence that the ballots were cast or
in any way allow for voter identification;
"were cast, including provisional codes
associated with a CVR".
Delete "for voter verification" from each of the sections
since any paper record of the CVR should be handled in
this manner




Replace with "In systems providing voter interaction in
multiple languages, the CVR and all copies thereof, and
the Audit Record, shall be free of indications of the
language selected by the voter."
Remove the last sentence.




“In systems providing voter interaction in
multiple languages, the CVR and all copies
thereof, and voter verifiable audit records,
shall be recorded in which the voter has
chosen to vote.”




Drop the last sentence




Add the requirement that the firmware should
be open source.
Develop requirements for security access to
the voting devices after successful
completion of election day testing.
Voter verified ballots can be used to form part of the election
system's audit trail. These ballots can be retained for
possible later use in spot-checks or mandatory recounts, or
they can be optically/electronically scanned or hand-counted
to form or confirm the official tally of election results. Some
variations of the voter verified balloting system may permit
the voter to later "look up" or review an encrypted version of
their ballot that does not reveal its contents, but
mathematically can confirm that the cast votes were properly
entered into the official totals. A voter verified ballot can also
contain additional information that can be used by the
election officials or vote tallying system to confirm that ballots
have not been substituted, duplicately entered into the ballot
box or vote totals, orsystem butany way, but thisbeing altered
….embedded in the altered in prohibited from information
during election operation.




End sentence at "completion of election day testing."




Eliminate “If software is resident in the
system as firmware”.
Change “may be verified” to “is reported”.

Add requirement that a checksum must be
produced and verified with some central
agency, preferably using a challenge so that
the person starting the device can not know
the expected result nor produce it on
her/his own.
Require physical locks with redundant keys
(so that no individual can gain access)
protecting the innards of the device so that
any unauthorized access will result in
obvious physical damage.
Explicitly specify minimum strength.
Change “may” to “shall” for the bootstrap
and monitor.
Deleted item.




Strike the second half of the sentence.

Remove this restriction unless there is
sufficient justification.

Add the requirement that all COTS used in
any voting system must be open source.

Enabling these audits should be required
only for systems supporting them.


…largest expected data, including voter initiated deletions,
with maximum….




Sentence starting "It also ensures the existence of an audit
record…" must be changed to ensure that ballots are not
identified to the voter casting them. As well, there should be
NO WAY to alter election data, other than to record ballots.
All existing ballot data must be retained intact in the event of
any termination or halting of election software processes.
Add requirement: “The voting device shall
operate as a dedicated piece of hardware and
its hardware and software shall be tested
together.” Add reference to IEEE Std 1471™-
2000, “IEEE Recommended Practice for
Architectural Description of Software
Intensive Systems”.
Add requirement: “The voting device shall be
an embedded system consisting of dedicated
hardware and software tested together, and
complying with IEEE Std 1471™-2000, ‘IEEE
Recommended Practice for Architectural
Description of Software Intensive Systems’.”


Add requirement: “Any and all communications
between the voting device and the outside
world shall be initiated by the voting
device; furthermore, no data received from
the outside world shall be retained by the
voting device longer than is necessary to
parse the said data and drive the protocol
state machine.”
Eliminate paragraph.

Delete this item.

Add the requirement for voter verifiable
paper ballots.

…denial of service, and time and logic bombs. All
keypresses of any single or combination of voters and/or
administrators shall be precluded from adtivating any
software or firmware process other than those directly
pertaining to the election aspect being used. Data flow
analysis should be used to validate these controls.
Eliminate the first sentence, the preamble.




All [single or combination of] keypresses of
any single or combination of voters and/or
administrators shall be precluded from
activating any software or firmware
process[, e.g., an "Easter Egg" like
feature,] other than those directly
pertaining to the election aspect being
used.
Add requirement: “The voting device shall be
an embedded system consisting of dedicated
hardware and software tested together, and
complying with IEEE Std 1471™-2000, ‘IEEE
Recommended Practice for Architectural
Description of Software Intensive Systems’.”


Change to: “Upon commencement of use for an
election, the system shall perform a sanity
test of the CPU, test of the code image
against the error detection data therein, a
test of all RAM, of the real-time clock and
timers and the consistency thereof, and a
test of every other peripheral device;
failure of any of these tests shall be
logged and shall prevent use of the device
for voting; these tests shall be run
immediately prior to entering normal running
mode for voting. Also, when otherwise idle,
the system should repeatedly perform tests
of the code image and other data in memory
that can not be modified, of RAM that is not
in use, and of whatever other entities for
which tests may be devised.
Change “public network” to “communicate with
the outside world in any manner including,
but not limited to, a PSTN, VPN, or
Internet”
Move the words “in human readable format” to
the introductory paragraph of item “a”.
Specify UT (GMT) and specify allowing
setting of local time zone for display
purposes; require, however, that UT be used
for all logging.
Disallow altering of the system clock;
mandate that a defective system clock be
replaced. This is necessary to prevent
tapering with what is reckoned as during
polling hours. Also, recommend use of
checking the clock against broadcast radio
signals such as provided by NIST.




Committee needs to decide if TDP is
proprietary or not.




Change 4 to 4.5.4   Add the additional
reference?
Delete the first two sentence of section
5.1.3.5.1, because they are false. Election
audit trails in no way verify the
correctness of the reported election
results!




Add to the end of the paragraph: “Access
control of audit data is on a “need to
perform” basis to ensure that election
officials can make the data public at the
appropriate time according to election
procedures. At the time prescribed by law
and procedure, the general public has a
“need to know” the data. However, the
integrity of the data must be protected at
all times.”
Replace "essential" with "necessary but not
sufficient"




Either remove the statement in parenthesis
or substitute the statement " Automatic
ballot generation is only permitted under
the conditions defined in section
5.6.7.2.1."




Develop extremely advanced AI technology or
time travel capabilities to fetch it from
the future.
Eliminate or drastically restructure this
requirement.


We need to modify the wording to permit test modes as long
as they cannot be exercized during the voting process or
delete the parenthesis so that the language in the other
sections governs.




Replace with "The ballot interpretation logic shall test
and record the correct installation of ballot styles or
formats on voting devices for the voting precincts at the
polling location and that the ballot logic produces a
correct count for each candidate and issue on the ballot
(NOTE: The system shall only automatically generate
voted ballots when in a test mode);"




Change the wording to "Identification of the
voting location for which the voting system
was programmed."



delete

replace with identifying the election
location that is being voted by the machine

I believe these items are intended to be
verified by a human operator. The
requirement should be simply for the
information to be output.
eliminate "human readable message" or add
code can reference it in some documentation
optionally
… the greatest extent possible (in accordance with accepted
industry practices).
There shall be a secure time/date stamp protocol used as
well as a human readable format.




Reword: The system shall be capable of
producing a backup copy of any electronic
audit records…
change to relative time clock not
registering date
CVR must me excluded from review when system
is activated for elections.
Add: “Any election data transmitted from the
voting machine to a remote location, being
subject to many sundry methods of tampering,
shall not be used in any final, official
tally, but rather may only be used for a
preliminary tally; all election results
shall be constructed from data transferred
directly, without use of any network or
other open system, from a physical device
that resided in the voting device during the
election to a tallying device at the
jurisdiction’s central elections office.”

Decide and specify what is meant by "Local
Area Network."


Replace “datagram” with “message” wherever
it appears.




Additionally, only [currently certified]
devices and applications will be allowed to
interface with the voting system hardware or
software.
Can we dispense with the overspecification
of specifics? How about "Voting systems
which transmit data shall ensure the
integrity of all transmitted data"?
Only use of wired transmission should be allowed in this (and
any) section of the standards document.



Protocols to enforce these conditions must be non-
proprietary and capable of demonstrating correctness,
accuracy and integrity.
Change “datagram” to “packet” or “message”.

Replace with "Be configured so that only datagrams
authorized and required by the voting system appear on
the physical network medium and that datagrams from
the voting system are not transmitted to non-voting
systems."
Replace with "Manage encryption keys to ensure that
the keys are not compromised and that the keys are
changed on a periodic basis."
Reword as follows "Manage encryption keys
to…."
Specify some official category used by NIST
or specify a functional description of the
cryptographic strength required.
Related to comment 12, above. Physical
isolation is a better solution than
encryption 100% of the time.


replace the wording with more generic
references; it is not a given that a PKI
structure must be used
Reword as follows "Configure the local
network so that…."
Add: “Any and all threads used for
interacting with the voter and recording
data internally to the device shall have a
higher priority than any other task on the
system, excepting perhaps a monitoring, self-
check task of nominal bandwidth, so as to
prevent any external communications from
disrupting the main functionality of the
device.”
Add: “All modules buffering or processing
data communicated from an external source
shall use a pre-determined amount of RAM so
that no babbling or malicious communication
with the voting device can degrade the
voting related functionality.”
Cabling and external attached connectors.
Eliminate the section. Add a requirement:
“All communications between the voting
device and the outside world shall be
initiated by the voting device; the voting
device shall refuse to buffer or parse any
communication not requested.”
This should include appropriate and timely reporting of any
observed breaches or breach attempts during the election
setup, actual election, and post election and canvass.




Delete it
Eliminate or severely modify section
5.1.3.4.2.
Clarify the text.
replace the wording with more generic
references; it is not a given that a PKI
structure must be used
Additionally, only [currently certified]
devices and applications will be allowed to
interface with the voting system hardware or
software.
Communication interfaces to the public
telephone network or any other network
outside of the physical boundaries of the
polling place should be disabled in real
election mode.
Election verification includes voter
verification and results verification. Voter
verification shall provide mechanisms to
ensure that each ballot is captured and cast
as intended by the voter. Results
verification shall produce an irrefutable
tally to ensure that anyone can verify all
ballots were counted-as-cast.
Voter verification ensures that the voter's
ballot accurately captures the voter's
intent and is contained in the ballot box
for counting. Since only the voter knows
his/her intent, only the voter can be
responsible for voter verification.
The voter shall have the option to access
human readable information that securely and
uniquely connects the voter's intended
choices to a ballot, according to the
tabulation rules.
1. The voter shall have the option to verify
that the ballot cast by the voter is in the
sealed ballot box.

2. The voter shall have the option to
determine that the voter's ballot, as
captured in the sealed ballot box, preserves
The same connection to the voter's intended
choice as was indicated by the human
readable information presented to the voter
during the act of voting. The information
granted to the voter shall provide
irrefutable proof of any discrepancy.


Results verification shall produce an
irrefutable tally from the sealed ballot box
such that any election auditor or observer
can determine, with full confidence, that
all ballots were counted-as-cast. Measures
shall be in place to ensure that votes
cannot be added, removed, or modified.

Election policies should provide precise,
publicly accepted procedures for creating
(“sealing”) the sealed ballot box at the
proper time.
1. The sealed ballot box shall be made
available for scrutiny to any election
auditor or observer.

2. An election auditor or observer shall be
able to determine, with full confidence, for
each ballot in the sealed ballot box,
whether it is “legitimate” in the sense that
it was cast by a unique, eligible voter.

3. An election auditor or observer shall be
able to determine, with full confidence,
that no ballot in the sealed ballot box has
been undetectably deleted or changed.

An election auditor or observer shall be
able to reproduce the election results, with
full confidence, per the tabulation rules.

Wireless connectivity, both in the development and
deployment of election systems poses significant
security risks, such that it is necessary to require
additional security measures to specifically mitigate the
risks posed by the use of wireless communications
devices during all stages of the election process.
Make the first sentence read “ ... design
of the system, potential exposure to risk,
and the threats identified in Section
5.1.2.3.”




Add design review and source code review for security and
confidentiality.
Delete first sentence up to the first comma.


Delete "… from types of attacks known at the time the
system is submitted for qualification."
Specify something about the attacks
Add: “Any election data transmitted from the
voting machine to a remote location, being
subject to many sundry methods of tampering,
shall not be used in any final, official
tally, but rather may only be used for a
preliminary tally; all election results
shall be constructed from data transferred
directly, without use of any network or
other open system, from a physical device
that resided in the voting device during the
election to a tallying device at the
jurisdiction’s central elections office.”

Add the text of the comment to the
paragraph.

Insert a new section 6.1.1 titled “System
Design” and stating “The design
documentation explaining how the voting
system is designed to counter and defend
against the threats of Section 5.1.2.3 shall
be analyzed. Tests shall be conducted to
verify that the system adequately defends
against these threats. These shall include
conducted or simulated attacks based on
hypothesized scenarios based on the threats
and on analysis of vendor-identified
vulnerabilities.
Add Section 6.1.5: If the system does not
include a voter-verifiable paper audit
trail, the suitability of the system to
operate without such a function shall be
evaluated. Such evaluation shall include
review of evaluation results for software,
procedures, reported problems with similar
technology, reliability, and accuracy
evaluations to identify of conditions that
mandate the inclusion of a voter verifiable
paper audit trail.
Modify the text in 6.1.2 to refer
specifically to public networks.


Delete the “and” before “maintenance” and
between “trail” and the period, insert “,and
for system protection during storage between
elections.”
Are jurisdictions able to use uncertified
systems? If not, these two items should be
removed.

Change the phrase to "be subject to
reeaxamination."

Fix the text.



Replace words "be subject to reexamination" in both
sentences with "be required to be reexamined."



The details specified for the chip should include high level
descriptions that were used in the design, such as RTL, high-
level language functional models, gate-level netlists, and
layouts, as well as the verification environment, such as
simulation testbenches.
Delete the reference to EAL-2

The generic PP needs to be converted into a
specific PP (or more, if necessary to
address different kinds of voting systems,
such as DRE versus non-DRE) covering all
classes of voting systems.
1. A design review board shall be convened
to review the design of a voting system to
determine compliance with the election
verification standards (Section 5.1.3). The
board shall consist of independent experts
in the fields of election systems, election
administration, data security, computer
security, and other fields as deemed
appropriate by the duly designated oversight
body (e.g., Election Administration
Committee, Secretary of State).

2. The board's review shall to be based on
publicly available information.

Example: FTP_ITC.1 -- "healthcare" should be
changed to "voting system"
A purpose statement. Is this a requirement
and how do we test?

Table 4 Security Objectives and Functional
Requirements Mapping

FAU_GEN.1 Audit data generation
This component outlines what data must be
included in audit records and what events
must be audited. This component traces back
to and aids in meeting the following
objective: O.AUDIT_RECORD that supports
P.ACCOUNTABILITY and P.AUDIT_CONTROL
policies.

FAU_GEN.2 Election Official identity
association
This component ensures that events recorded
in the audit trail are not associated with
individual Voter identities. This component
traces back to and aids in meeting the
following objectives: O.AUDIT_RECORD,
O.ENTITY_AUTHENTICATION, O.ROLES, and
O.ELECTION OFFICIAL_AUTHENTICATION that
support P.VOTER_ANONYMITY policy

FAU_SAA.1 Potential Violation Analysis
This component monitors the TOE and sends an
alarm when a security violation has
occurred. This component traces back to and
aids in meeting the following objective:
O.EVENT that supports P.ALERT policy.

FAU_SAR.1 Audit ReviewGeneric [Security]
Change title to read:
Protection Profile For [Voting Systems]
Define all conventions prior to their usage.
Rewrite and reformat the annex based on
those definitions and formatting used in the
rest of the Standard. Remove or substantially
reduce the jargon that make this annex read
like the mumbo-jumbo of some cult. Submit
both the new Annex C and the revised Annex D
(currently Annex C) to the P1583 security
working group for independent review.


Requires discussion by entire working group
to determine proper EAL level, likely EAL4
with components from EAL5 and EAL6. Rewrite
Annex to match EAL level determined by
working group. See attached notes.
C.7.3.1 Since the audit trail must
necessarily be turned off during voting in
order to preserve ballot privacy, additional
security assurances must be provided in
order to maintain ballot data integrity.
Such assurances could include the addition
of an voter verifiable audit record to the
voting system.
Analyses required for assurance should
include covert channel analysis to the
extent covert channels can be used for
signaling to bypass vote privacy
requirements. This can be accomplished by
adding a portion or a modified version of a
covert channel analysis component to the PP
assurance requirements.
Revise the assumption




Revise the assumptions accordingly.

Revise the assumption
Revise the assumption




Revise the assumption




The (potential) threat agents implied in
5.3.2.1 include (malicious) vendors, their
employees, suppliers, and the suppliers'
employees; perpetrators who can access the
delivery process; providers of services in
the delivery process; election officials
(non-polling place), election officials
(polling place), voting system technicians,
intruders into voting machine storage
facilities, voters, and eavesdroppers
(listening to compromising electromagnetic
emanations or inter-equipment
communications).
Replace with "Threats To Be Addressed By The Operating Environment (Non-IT Environment)"
Eliminate the exemption.




Provide a table showing how the threats in
C.3.3.3 track the threats in 5.3.2.1

Revise the statement taking other comments
on this section into account.
Revise the statement accordingly.
Revise the statement accordingly.




Revise the statement accordingly.

Revise the statement accordingly.

Revise the statement accordingly.




Revise the statement accordingly.

Either delete the statement or revise it to
make sense. For example, no session with a
voter should automatically terminate, but a
session with an administrator might
reasonably do so.
Append the text of the comment to the
statement.
Delete
Clarify or delete the objective.

Revise the statement.

Delete everything after “natural language
format.”




Revise the statement accordingly.
Revise the statement accordingly.




Revise the statement accordingly.



Add an objective to the effect that the TOE
must preserve vote anonymity with or without
collusion of the voter to bypass it.

Clarify or delete the objective.




Revise the objective accordingly.




Delete “and election data.”


FMT_SMF.1 Specification of Management Functions




Rewrite in accordance with voting system
application and security requirements as
reflected in the remainder of the draft
standard.
Add the following Sections:

C.5.1.5.3 Specification of Management
Functions (FMT_SMF)
C.5.1.5.3.1 FMT_SMF.1 Specification of
Management Functions
C.5.1.5.3.1.1 FMT_SMF.1.1
The TSF shall be capable of performing the
following security management functions:
[assignment: list of security management
functions to be provide by the TSF].

C.5.1.5.3 Specification of Management Functions
(FMT_SMF)
C.5.1.5.3.1 FMT_SMF.1 Specification of Management
Functions
C.5.1.5.3.1.1 FMT_SMF.1.1
The TSF shall be capable of performing the following
security management functions: [assignment: list of
security management functions to be provide by the
The TSF shall provide authorized election officials with
the capability to verify the integrity of stored TSF
executable code.
Delete "ATE_IND.2 Independent testing"
AGD_ADM.1 Administrator guidance AGD_USR.1
User guidance

ATE_COV.1 Evidence of coverage
ATE_FUN.1 Functional testing               Add
"ATE_IND.2 Independent testing"
Provide assurance requirements appropriate
to the requirements, threats, and
objectives, after they have been revised and
become properly stated.




Add the following Sections:

C.5.2.1.2 International Interpretation #3
The configuration list shall uniquely
identify all configuration items that
comprise the TOE.


C.5.2.1.2 International Interpretation #3        The
configuration list shall uniquely identify all
configuration items that comprise the TOE.
Change to "The installation, generation and start-up
documentation shall describe all the steps necessary for
secure installation, generation, and start-up of the TOE.
"
Change from:

C.5.2.2.2.1.1 ADO_IGS.1.1C
The documentation shall describe the steps
necessary for secure setup, ballot
generation, and opening and closing of the
polls, start-up of the TOE.

to:

C.5.2.2.2.1.1 ADO_IGS.1.1C
The installation, generation and start-up
documentation shall describe all the steps
necessary for secure installation,
generation, and start-up of the TOE.
Upgrade the assurance requirement in
accordance with this comment and the
previous comment (SK-35).
     Proposed Resolution
(by the Chair on each comment submitted)
NC- Not appropriate place

Mercuri - Verification materials must be retained in order to
provide evidence of appropriate certification testing.
Comment should be moved to appropriate section.
3. NC - goes to policy 4. Add "3. 3. The need for anonymity
of voter ballot impacts many traditional forms of auditing
commonly used for other electronic systems (such as ATMs
in banks). "

Mercuri - "impacts" is the wrong word choice for the new
phrase and is misleading. "precludes" (or reduces or
removes) as appropriate.
C-Added is recommended

Mercuri - the word "totals." was omitted from the end of the
sentence in the draft.
C- Wireless can be deployed securely - changed to add "
Wireless connectivity, both in the development and
deployment of election systems poses significant security
risks, such that it is necessary require additional security
measures to specifically miti

Mercuri - It is not agreed that wireless can be deployed
securely. This must be taken up by the entire working group.
NC - wording is not correct

Mercuri - Access violations and violation attempts are an
important part of the security process and must be noted in
the audit trail.
C- add "while maintaining ballot integrity"

Mercuri - Phrase was not added in draft v5.0
C - VW - Add to document as second to last paragraph in
section 5.1.3.5.1

WFW - Adding the VoteHere approach as generically
described, may or may not be acceptable as another method
of voter verification. We are getting into areas that I believe
rightful

Mercuri - The wording added in 5.1.3.5.1 is overtly vague and
is intended to permit vendors to provide internal audit trails
that can not be independently recounted or verified. This
issue needs to be discussed in the working group.

NC - Specifications are not the place to add requirements.
New specifications should be proposed as additions to
Section 5.

Mercuri - This comment is now requested to be placed at
5.1.3.4.1 d)
NC- if operation of an override is identifyable then tampering
can be identified. If an override is needed to correct a DOS
attack on the device (ie take a voting location out of service
by activating tamper protection, then it should be identifyable.

Mercuri - Override feature could be exploited. Comment
must be addressed by working group.
NC - a "voter initiated deletion" would not add to the storage
requirements in a significant way.

Mercuri - A voter initiated deletion may still create an
additional record, depending upon implementation. Sufficient
deletions could impact data requirements adversely.
NC - specific language to make change not provided.
Current language is adequate.

Mercuri - OS and DB audits are insufficient to provide the
ensurances identified in this section. The paragraph is
inadequate and incorrect and must be reworked.
C - Added"All keypresses of any single or combination of
voters and/or administrators shall be precluded from
adtivating any software or firmware process other than those
directly pertaining to the election aspect being used."

Mercuri - Add "time and logic bombs" Also need to address
data flow analysis to validate these controls.
C as provided

Mercuri - replace "practical" with "possible" as provided
NC - The time/date stamp is part of the Audit log and is
required to be secured from loss by current language

Mercuri - Comment was misunderstood. Secure time/date
stamping protocol must be used in addition to human-
readable format.
NC - wireless connections can be deployed with solid
security. These issues are dealt with in other sections.

Mercuri - Comment must be discussed by working group.
Wireless poses significant security risks.
NC goes to design

Mercuri - Correctness, accuracy and integrity are
performance.




NC - if used should be identified

Mercuri - Antennas should not be used since they pose a
significant security risk.
C Added new para d. "d. The vendor should provide
appropriate and timely reporting to the appropriate
jurisdiction of any observed breaches or breach attempts
during the election setup, actual election, and post election
and canvass for all systems of th

Mercuri - The jurisdiction must also report breaches if they
were the observers.

Editor - Requirement for jurisdiction are outside of scope of
C -as indicated

Mercuri - The draft is incorrect, and does not reflect the
changes as specified. "shall be reexamination" in two places
should be replaced with "be required to be reexamined"
The Operating Environment (Non-IT Environment)"
Reference Information   Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

                Date: 9-30-03 P1583 Draft 5.0 August 2003
                      Document:
                 Commenter Clause/ Subclause        Paragraph      Type of
                and Number                       Figure/ Table    comment
                                                                  (General/
       #                                                          Technical/
                                                                   Editorial)
           1    selker-1004        5.2.1.2.3   #1                     E



           2    Lipsio-84           5.2.2.3                           E

           3    RGH 038              5.3                 5            E


           4    RGH 113           5.3.10.1.3                          E


           5    RGH 046           5.3.10.2-3        4th bullet        E




           6 Corry-043             5.3.10.5    Title                  E


           7    RGH 049             5.3.1-2                           E




           8    RGH 057             5.3.3-5         7th bullet        E




           9    RGH 058              5.3.4          footnote 14       E

           10   RGH 115              5.3.5                            E
11   RGH 064              5.3.5-11       3rd bullet           E




12   RGH 067              5.3.6-6                             E


13   MercuriD50 - 076     5.3.10.2   end of subsection        T
     (formerly mercuri-
     171)




14   MercuriD50 - 077      6.3.3     Test matrix - 5.3.10.2   T
     (formerly mercuri-
     172)


15   Jhulshof-005           5.3      all section              E
16   HD-011                 5.3      7th bullet               G


17   RGH 037                5.3                3              G


18   RGH 039                5.3          7th bullet           G

19   RGH 040                5.3         Table 5.3-1           G




20   Aragon - 01            5.3      First paragraph          T


21 Corry-051                5.3      Figures 5.3-2,3,4,5,6 T



22
     Sklein-048             5.3      First paragraph          T
23   MercuriD50 - 055       5.3.1                      2   General
     (formerly mercuri-
     126)



24   MercuriD50 - 054       5.3.1      Bulleted List       General
     (formerly mercuri-
     125)


25   Jhulshof-006           5.3.1                      2   general


26   Aragon - 03          5.3.10.1               3            T




27   Jhulshof-016         5.3.10.1                     2     T

28   RGH 041              5.3.10.1                            T



29   Aragon - 04          5.3.10.1               1           T




30   RGH 042              5.3.10.1-1                          G
31   PPLX-027       5.3.10.2




                                    5.3.10.2
                                    Accessibility
                                    for Voters with
                                    No Vision,
                                    Limited Vision,
                                    Reading
                                    Problems, or
                                    Print
                                    Blindness
                                                           E
32   Jhulshof-018       5.3.10.2    3                   general
                                        dot 3 from bottom



33   Jhulshof-017       5.3.10.2                      1   T



34   RGH 114            5.3.10.2                          T




35   Aragon - 05        5.3.10.2    3, fourth bullet      T




36   wfw - 009         5.3.10.2.1       .1 (second        T
                                         sentence)
37   wfw - 010         5.3.10.2.2           0.1           E


38   RGH 043           5.3.10.2-1                         G


39   RGH 044           5.3.10.2-3       12th bullet       G

40   RGH 045           5.3.10.2-3       13th bullet       G

41   RGH 047           5.3.10.2-3                         G
42   MercuriD50 - 061     5.3.10.3             1   General
     (formerly mercuri-
     139)


43   Jhulshof-019         5.3.10.3             1     T

44   Jhulshof-021         5.3.10.4             1     T


45   Jhulshof-022         5.3.10.5             3   general



46   RGH 048              5.3.10.5                    T



47   MercuriD50 - 062     5.3.10.6             1   General
     (formerly mercuri-
     140)
48   Jhulshof-023         5.3.10.6             1   general




49   Jhulshof-025         5.3.10.6   figures       general



50 Corry-044              5.3.10.6             1      T



51 Corry-045              5.3.10.6   1.a.             T




52 Corry-046              5.3.10.6   1.b              T
53 Corry-047            5.3.10.6    1.c              T




54 Corry-048            5.3.10.6    1.d              T




55 Corry-049            5.3.10.6    1.e              T




56 Corry-050            5.3.10.6    1.g              T




57   Jhulshof-024       5.3.10.6    1 g              T


58   HD-013                 5.3.3   1. Last bullet   G




59   HD-014                 5.3.3   9. & 10.         G
60   PPLX-021       5.3.3




                                    5.3.3
                                    Information
                                    Presentation     G
61   MercuriD50 - 057      5.3.3     Add bullet under 5        General
     (formerly mercuri-
     132)

62   MercuriD50 - 056      5.3.3     Second bullet under       General
     (formerly mercuri-              2.
     129)




63   Jhulshof-007          5.3.3     table                general
                                              charactor size 5.3-1



64   Jhulshof-008          5.3.3                           7   general

65   HD-012                5.3.3     Table 5.3-1                  T




66   RGH 050               5.3.3     Table 5.3-1                  T

67   RGH 051              5.3.3.-7             2                  G



68   RGH 052              5.3.3.-7       2nd bullet               G


69   RGH 053              5.3.3-1        2nd bullet               G


70   RGH 054              5.3.3-3                                 G


71   RGH 055              5.3.3-3                                 G

72   RGH 056              5.3.3-5        4th bullet               G
73   PPLX-022             5.3.4




                                          5.3.4 Vote
                                          Selection
                                          Mechanisms         E and G
74   MercuriD50 - 074             5.3.4   7                   General
     (formerly mercuri-
     169)




75   MercuriD50 - 075             5.3.4   footnote #17       General
     (formerly mercuri-
     170)




76   MercuriD50 - 058             5.3.4   footnote #17       General
     (formerly mercuri-
     135)




77   MercuriD50 - 059             5.3.4                  8   General
     (formerly mercuri-
     136)


78   Jhulshof-009                 5.3.4                  3   general
79   Jhulshof-010    5.3.4                      8   general




80   Jhulshof-011    5.3.4     9 last dot           general

81   RGH 059        5.3.4-3                            G


82   RGH 061        5.3.4-9      4th bullet            G

83   HD-015          5.3.5                             G


84   Jhulshof-012    5.3.5                    13    general


85 Corry-037         5.3.5                      5      T




86 Corry-038         5.3.5                      6      T

87 Corry-039         5.3.5                    11       T




88 Corry-040         5.3.5     12, 2nd bullet          T



89   RGH 062        5.3.5.10   first bullet            T


90   RGH 063        5.3.5-11     1st bullet            G
91   PPLX-023             5.3.6




                                                 5.3.6
                                                 Navigation and
                                                 Interaction
                                                 with the System      E
92   MercuriD50 - 060             5.3.6                         1   General
     (formerly mercuri-
     137)




93   Jhulshof-013                 5.3.6                        1    general



94   RGH 116              5.3.6.1 and 5.3.7.6.                         T




95   RGH 065                    5.3.6-1                                G

96   RGH 066                    5.3.6-3                                G

97 Corry-041                      5.3.7                        5       T




98   HD-016                       5.3.7                        2       T




99   Jhulshof-014                 5.3.7                        2      T
100   PPLX-024       5.3.7




                                     5.3.7 System
                                     Response Time
                                     and Feedback         T and E
101   RGH 068             5.3.7.4                            T



102   RGH 069             5.3.7-4                            G
103   PPLX-025       5.3.8




                                     5.3.8
                                     Preventing and
                                     Minimizing
                                     Voter Errors           G
104   Jhulshof-015           5.3.8                    3     T



105   RGH 070             5.3.8.5                            T

106   wfw -007            5.3.8.5                           T


107   RGH 071             5.3.8-5                            G
108   PPLX-026             5.3.9           5.3.9 Help and
                                           Indications of
                                           Degraded
                                           Conditions           G
109   wfw - 008                5.3.9.2           All            T




110   df10                         6.3     All Sections         E



111   MercuriD50 - 019             6.3     Bulleted list      General
      (formerly mercuri-
      044)




112 Corry-125                      6.3                          T



113 Corry-126                      6.3                          T



114 Corry-096                  6.3..3.3                         T

115 Corry-090                  6.3.1.1                           E
                                           2nd para., 2nd sentence




116   RGH 090                      6.3.2                        G




117   RGH 091                      6.3.3                        E
118 Corry-091     6.3.3      Entire section   G




119 Corry-092    6.3.3.1                      T

120 Corry-093    6.3.3.1                      T

121 Corry-094    6.3.3.1                      T

122 Corry-124   6.3.3.10.1   Item 3           E



123 Corry-122   6.3.3.10.1                    T

124 Corry-123   6.3.3.10.1                    T


125   RGH 092   6.3.3.10.2     5.3.10.2-4     G




126 Corry-127   6.3.3.10.2                    T




127 Corry-128   6.3.3.10.2                    T


128 Corry-129   6.3.3.10.3                    T



129 Corry-130   6.3.3.10.3                    T

130 Corry-131   6.3.3.10.4                    T
131 Corry-132          6.3.3.10.5                              T



132 Corry-133          6.3.3.10.5                              T



133 Corry-134          6.3.3.10.5                              T


134 Corry-095            6.3.3.2                               T


135 berger - 004   6.3.3.2 - 6.3.3.10.3                        E

136 Corry-099            6.3.3.3                                E
                                          Para. 2., first sentence
137 wfw - 013            6.3.3.3                 3              E




138   RGH 093            6.3.3.3              5.3.3-7          G




139 Corry-097            6.3.3.3                               T

140 Corry-098            6.3.3.3                               T


141   RGH 094           6.3.3.3-1                              G


142   RGH 095           6.3.3.3-1                              G




143   RGH 096            6.3.3.4              5.3.4-1          G




144 Corry-100            6.3.3.4                               T
145 Corry-101   6.3.3.4              T




146 Corry-102   6.3.3.4              T




147 Corry-103   6.3.3.4              T




148 Corry-108   6.3.3.5              E


149   RGH 097   6.3.3.5   5.3.5-10   G




150   RGH 098   6.3.3.5   5.3.5-11   G




151   RGH 099   6.3.3.5   5.3.5-12   G




152   RGH 100   6.3.3.5   5.3.5-13   G




153   RGH 101   6.3.3.5   5.3.5-14   G
154   RGH 102   6.3.3.5       5.3.5-9   G




155 Corry-104   6.3.3.5                 T



156 Corry-105   6.3.3.5                 T


157 Corry-106   6.3.3.5                 T




158 Corry-107   6.3.3.5                 T

159 Corry-109   6.3.3.6                 T


160 Corry-110   6.3.3.6                 T

161 Corry-111   6.3.3.6                 T


162 Corry-115   6.3.3.7   Item 6        E

163 Corry-116   6.3.3.7   Item 7        E

164 Corry-112   6.3.3.7                 T


165 Corry-113   6.3.3.7                 T


166 Corry-114   6.3.3.7                 T




167   RGH 103   6.3.3.8       5.3.8-1   G


168   RGH 104   6.3.3.8       5.3.8-2   G




169 Corry-117   6.3.3.8                 T
170 Corry-118   6.3.3.8                        T



171 Corry-119   6.3.3.9   Title                E



172   RGH 105   6.3.3.9       5.3.9-1          G




173   RGH 106   6.3.3.9       5.3.9-3          G




174   RGH 107   6.3.3.9       5.3.9-3          G




175 Corry-120   6.3.3.9                        T



176 Corry-121   6.3.3.9                        T


177 Corry-210     B       2nd para.            E




178 Corry-211     B       2nd para., add bullet T
                 Comment



I am not convinced that this is the best way
for a well funded operation to through an
election. This retoric does not belong in a
standards paper
Paper ballots are a type of removable
storage media.
ref: Embedded in the DRE is software... This
is not true in all DREs

…so that it is not possible to differentiate
votes from the "accessible" voting machines.

ref: the use of the word option. The audio
ballot mode was designed as a stand-alone
voting mode - separate from the visual
ballot mode and not all options are valid in
both the audio or visual mode. For example,
the write-in keyboard is a single image on
the visual ballot but on the audio ballot it
is 40 separate sound files.
No provision for voters with prosthetic
devices.

I don't understand how information
presentation and large font affect privacy.
If it is referring to these as increasing
voter independence then the wording should
change.
ref: # of colors to be used. Is this
referring to the number of colors on a
single screen or the total numbers of colors
used in the interface. If it is the total
number of colors - this is too low.
number 3 - 5 in the process sequence are
wrong.
This section should be clarified that it
pertains to "Voter" Input/Control Devices
and Feedback. Such definition should not be
misconstrued to pertain to other users such
as pollworkers, warehouse workers, and
election office administrative staff.
the inherent issue of averaging (when 2
points are selected on a touch screen and
the driver averaged the distance between the
touches and activates the nearest area)
needs to be addressed in this item.

giving the voter the option to cast a blank
ballot is not always applicable and is
defined by state law.
Clarification that voters with visual impairment have same
voter verifiability information as sighted voters (This item is
similar to but not the same as one in David Aragon's
comment set.)




Test support for new paragraph of 5.3.10.2. Propose to test
by "I" (inspection) because the requirement relates to the
logic and data path from which the data is presented, so
verification requires inspection of source code but not
expertise in human factors. (This item is the same as the one
dots are not comment set.)
in David Aragon's usable for reference
Position of parenthesis and its contents
detracts from readalbility of the sentence.

ref: information is properly grouped. why is
this a standard? This is the responsibility
of the county
why is this a standard? This is the
responsibility of the county
The required size is not defined in this
table but the required size is in section
5.3.10.2-1. I don't understand what this
table is telling me. Do we have to have the
ability to display a different font size for
someone that is 6'9" tall because they are
viewing the ballot at a greater distance?

This section can be applied to non-DRE systems also, as it
relates to input while the definition of a DRE relates to its
output and tallying capabilities.
Dimesions should not be in millimeters as
that accuracy is not realistic. Also, inches
are normally related to centimeters in
metric units.
The first sentence states that the section
is intended to apply only to DRE systems.
However, some of the principles stated in
the section apply to non-DRE systems as
well. There needs to be something said
about usability for non-DRE systems
Add ability of election officials to ensure between voter uses
that equipment is still secure and functioning properly.




Add recountability.




speech input is thread to privacy poll
workers can hear and count number of yes and
no
"should be tabulated" -- to this
recommendation for how the machine is used
in combination with others, add an analogous
requirement on the individual machine.

it must not be possible to connect two
headphone thread to privacy
NO! Can't require jurisdiction to provide
accessibility assistive device (eg.,
disposable sip 'n puff)

5.3.10 still does not give adequate guidance
to manufacturers as to how an audio
interface can be made to offer "the same
voting capabilities and options" per
5.3.10.2 (and HAVA), particularly with
respect to VVAR. Public debate has included
statements that a VVAR cannot be implemented
accessibly, or that it need not be. One
proposal and prototype from a major vendor
requires use of a personal assitive device
(hand-held scanner), which State of
California would reject on HAVA concerns.
Standard should pro-actively address this
confusion to facilitate development of
compliant systems.

In 5.3.10.1, should clarify applicability of
point 1 to VVAR.
I don't see how this can be a standard. All
voting systems including paper based system
would fail if this standard is put into
place. How will a quadriplegic use any
voting system without an assistive device?
The following bullet: “Provide auditory
output using two distinguishable voices. One
voice should be used
exclusively for communicating instructions
and the other should be used exclusively for
communicating content” is overly
prescriptive. We suggest removing it.


The requirements make no provision for
multiple languages.

Furthermore, a missing, but important
requirement allows voters who have visual
disabilities to have ballots verified or
audit trails handled in the language in
which they voted.

display choice is discriminating solution
against full face machines with printed
ballot lay out. Privacy must be guarentied
inanother way
fond size is a solution discriminating other
solutions as magnifying glasses, mirrors
increased light and high definition printing

…with graphic elements scaled
proportionately. Scaling graphical elements
is much more difficult than scaling text,
and in some cases clarity is decreased
rather than improved.
Same general comment as Aragon - 04.
Clarify that this requirement in 5.3.10.2 is
applicable to VVAR, and that a VVAR
complying with this requirement is
realizable.
Should not indicate a more preferable
approach.
We should not require or even recommend
Braille be used for the reasons stated in
the paragraph
the font height stated in this item is
defined as an option in table 5.3-1 and now
as a requirement?
ref: normal display. Does this mean the
visual ballot?
ref: audio and /or visual feedback. Is this
a combined audio and visual ballot?
does this standard apply when a voter
receipt is part of the system?
Wireless is a security and privacy risk.




wireless communication does not offer
guarantied privacy
speech input is thread to privacy poll
workers can hear and count number of yes and
no
this is a solution discriminating other
equal or better solutions like highly
accesable build in controls or controls on a
swiffle.
Additional controls to place in a voter's
lap? Cost?


Add privacy screen note.



this is a solution discriminating other
equal or better solutions like highly
accesable build in controls or controls on a
swiffle.


this is a solution discriminating other
equal or better solutions like highly
accesable build in controls or controls on a
swiffle.
All subsections need to be rewritten.
Dimenstions need to be in centimeters and
inches, not millimeters and inches.
Precision without accuracy error.
122 cm or 48 inches is further than a person
can reach sitting down. Figure 5.3.6 shows
the 122 cm (48 in) dimension as the distance
to the rear of the wheelchair wheels, not
the distance a person can to reach in front
of them. Forward reach for a person sitting
down should not exceed 60 cm (24 in).

Again, a person sitting down cannot reach
122 cm (48 in) over an obstacle sitting
down.
A person sitting cannot reach a control that
is 1.2 m (48 in) away from them.




A person in a wheelchair cannot easily reach
a control that is only 25 cm (15 in) above
the floor


Again, a person sitting down cannot reach a
control that is only 38 cm (15 in) above the
ground.



The horizontal surface referenced is
undefined.



there should only be a minimum titlt


Indicates that text should be left justified
and ragged right. Should this not depend on
where the selection target is located.
Agree that this is correct for a left
target. But should this be optionally
different for a right target (vote
indicator)?
These paragraphs and incorrectly indented.

The proposed text reads:
“2. Instructions should ascribe to the
following design practices:
· The voter should have the option to choose
from available languages (as required by the
Voting Rights Act of 1965). Translations
should be independently verified to ensure
correctness.”

The problem is that the term “available
languages” is somewhat vague.

We suggest clarification of “available
languages”.
Systems may only be used in jurisdictions
for which the systems support languages as
specified in the Voting Rights Act of 1965
as amended.
Color bias




Need to address tallying under language options.




table is a solution discriminating other
solutions as magnifying glasses, mirrors
increased light and high definition printing

there is a big array of coulor blindness

The table indicates 3 sets of values with
columns for Minimum, Preferred and
Recommended Option. (the column headers
also don't line up). It is confusing as to
what is the required specification
Too many options to test or require

why is this a requirement. The ballot design
should be such that all people regardless of
visual impairment should be able to read the
display.
contrast ratio range is too wide. Also if a
vendor meets the minimum are they considered
in compliance?
ref: bold should be limited. Bolding when
used in a deliberate fashion should be
acceptable
ref: well-known graphics. What are well-
known graphics? This could be completely
divergent things to vendors.
focus testing not well know graphics. Isn't
this taken care of in section 6.3.3.1
redundant to 8th bullet
In Paragraph 7, the draft says, “After
voters submit their votes, the system shall
inform the voter that their votes have been
properly registered and the voting process
has been concluded.” Paper ballot systems
have voters deposit votes in a passive
ballot box.. A passive ballot box may not
comply with the requirement that the
“system” inform the voter….” We suggest the
following change.

“Through the use of signs, displays or other
alerts, the voter should be assured that the
voter’s vote has been cast after the voter
submits his or her vote.”

Similarly, paragraph 10 as drafted may be
impossible for paper based systems. T

Can be implemented through voter verification




Spoiling ballots needs to be clarified. (This item appeared in
an earlier comment set for Chapter 5 as mercuri-D049.)




Spoiling ballots needs to be clarified.




Ensure deleted data is not counted.




the voter shall always see the full set of
options within a contest, if not the options
not seen in the first instant are
discriminated
this is a solution discriminating other
solutions. The existing systems work
typically with the same activator to enter
and to erase a choice there is no reason to
change this. After the choice is cancelled
the voter will be informed and can reenter
the choice if desired
unnecessary error messages will hold up the
voting process
this can not be mandated. For counties in
Pennsylvania where a vote for 20 is common
the DRE will not comply.
ref: an error message should be presented.
This is not standard practice
For alphabetic keyboard entry (as for write-
ins), is querty or alphabetic sequence of
keys preferred?
speech input is thread to privacy poll
workers can hear and count number of yes and
no
I've never seen equipment that it was
impossible to accidentally actuate. Design
should minimize possibility.


Use of acronym "SPL" may not be correct.

A height and width of 1.5 cm and separation
of 1.9 cm between touch screen areas is too
small and too close together for my fingers.
Also, such a small area and distance makes
it a difficult target for elderly who are
shaky.
Return to home position should be obvious to
voter.


labels on button or keys, s/b braille for
blind voter and no need for labels if that's
only purpose
the width of the minimum separation is too
large. If this is mandated than the number
of ballot pages will increase and have an
undue affect on voters especially ones with
short term memory issues.
The last sentence of paragraph 1 states:
“Where there are time limits imposed on the
total time spent voting, the system shall
indicate the amount of time the voter has
remaining to complete the voting process.”

We have no specific recommendation here but
wonder if this is a good idea. Voters may
spend more time in a voting booth due to
extenuating circumstances. This may best be
controlled by poll workers.

Many states have a time limit for voting.




time limits are state rules whether or not
the voter should be informed on the time
left has no place in the federal
specification
If there is a maximum time to vote, it does
not necessarily need to be solved with an
integrated timer in the voting system.
Such a feature may be useful, but could and
should be left to the vendor and customer.
For example, a separate "egg-timer"
alongside the terminal could be a simple
solution.
Must the time indicator be displayed to the
voter at all times within the interface?
Must the task status be displayed on every
page?
Type ahead capability must not be provided,
not an option.



The text indentifies "feedback (within 0.1
seconds.." yet the next section discusses
more than 1 second. I expect that this is a
typo. Also move the parenthesis since it
misleads reader into thinking the time is
mandatory yet following sections indicate
accommodation for longer times.
button can be used to enter and delete a
choice time must be longer to prevend
bouncing
Paragraph 2 states:
“The system shall provide feedback (within
0.1 second, but preferably less) in response
to user
actions.”

While well intentioned, this paragraph can
lead to deleterious operation. For example,
voters who have a tremor may touch an active
area more than once within 0.1 seconds. If
the voter trembles, the selection may be
selected then deselected and the voter may
not understand why or the voter may not
realize that this happened. This problem
can be solved by “click debouncing.” Doing
so means that response should not occur and
feedback should not be given so quickly.

Section 12.7 of the FEC 2002 VSS provides
guidelines that anticipate both of these
issues. We think that the VSS has been well
thought-out on this issue.

too prescriptive of a solution



redundant to 5.3.7-3
Paragraph 2 has two problems. As proposed,
this paragraph states:
“The system shall check user inputs for
acceptability, e.g., check for inputs that
seem to be in error
(such as putting a Arabic number in a name
field) and alert voters to the error.”

A write-in candidate with the name of John
Doe 3rd will have a serious problem with
this requirement. Furthermore, paper based
systems that allow voters to write-in in
longhand will be unduly burdened to meet
this requirement.

Various state laws forbit accepting overvote



reversible too narrow

Choosing language on most systems is
currently not reversible, without spoiling
the ballot and reactivating the machine.
are all actions defined as reversible by the
voter? OR just reversible in general?
Paragraph 6 specifies colors. We think that
red should also be allowed for warnings.
Help and instructions are ususally provided
on the opening screen and also printed
within the voting booth itself. Providing
additional context sensitive help within the
voting device screen themselves will add to
the confusion. These are not PC's!
An extra "3" is in the numbering. For
example 6.3.3.5 refers to the specification
in 5.3.5.

Add Recountably




Need section to evaluate whether password
protection has been implemented and common
words cannot be used, e.g., password,
secret, dictionary words.
Need to evaluate whether roles have been
implemented and default superuser accounts,
e.g. root, sysadmin, sa, ora, etc. have been
disabled.
I cannot see how these items can be
validated without first inspecting them.
Public confidence is dependent on access to
the bases for why testing standards are, or
are not used.Therefore it is essential to
state specifically that such reasoning is to
be in the public record.
if methodology used should address the ANSI
(NNSI NCITS 354-2001) common industry Format
for Usability reports the document should be
included in this standards document for
comment.
it would be helpful for the testing types
had accompanying definitions
I can see no reason in most cases why an
inspection should not be required by the
procedures in these tables. Also, in most
cases a usability test should be required as
a check on what may simply be someone's
opinion. When something like luminance and
contrast, or voter privacy are to be
evaluated then an expert opinion is required
as those are far beyond what most
technicians can evaluate. Thus, these tables
need extensive additions to be satisfactory
(see following comments).Also, it would be
less confusing if the tables were labeled as
such and not as sub-sub-subsections.

Expert evaluation is required for many of
these.
Voter privacy must be tested.

Usability tests must be performed to ensure
the requisite functionality.
The meaning of this sentence isn't clear.



Accessibilty for disabled voters should be
subjected to an inspection as a first step.
The use and need, or lack thereof for
personal assistive technology will require
an expert evaluation.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
usability testing are left to determine
compliance, I have concerns.
I am at a loss to understand how one would
know whether the accessibility features for
voters with limited or no vision are present
without an inspection of the equipment.

One can't know whether the large text,
audio, or Braille are useful or usable
without some form of usability testing.
Whether or not wireless coupling for hearing
aids maintains voter privacy and avoids EM
interference will require expert evaluation.

Wireless coupling for hearing aids requires
usability testing.
Whether or not the voting system is usable
by voters with no or limited speech and
other probable impairments requires and
expert evaluation as well as usability
testing.
Whether the voting system is accessible to
voters with limited movement and
coordination must be subject to an
inspection.
I really can't see how the accessibility of
the system to voters with limited movement
and coordination can be evaluated without
usability testing.
Whether or not wheelchair users can use the
voting system in the same orientation
requires some usability testing.
Both testing and usability testing are
required of the means used to activate the
ballot.
Pull the introductory sentence out of the
first cell in the table.
Footnote is numbered one. Should be no. 28.
I still dissagree with this requirement. It
is unacceptable to provide the voter the
means to adjust color or contrast on a color
DRE. It will only add to the time to vote,
provide the opportunity for malicious use by
the voter and cost and complexity to the
device. It also misinterprets the intent of
Section 508, which has this as requirement
only if the device provides for it.

missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
Adequate luminance and contrast require an
expert evaluation.
Usability testing is essential for such
features as graphics, color, luminace and
contrast, flicker, flashing, etc.
ref: the use of the word One at the
beginning of the sentence. Should this not
be "The color ballot presentation…"
It's stated in 6.3.3.3-1 footnote 28 that
the contrast ratio is 6:1 or greater. This
contradicts the statement in section 5.3.3-7
which is the contrast ratio minimum is 3 - 1
but the preferred is 7- 1.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
All the evaluation methods require an
inspection, if nothing else.
There is nothing intuitive about intuition
and only an expert can evaluate such
interactions. The multiple rules for
undervoting also require an expert
evaluation of the system capabilities. And
are we to trust a simple test to confirm
votes are correctly registered and voting is
complete? I think not!
Certainly such features as intuitive
interactions require testing. Anyone who is
married knows that what is conspicuous and
obvious to the wife is often obscure to the
husband. So that feature must be tested as
well. And feedback is oftern overlooked,
ignored, or misunderstood. So again testing
is required.
Now certainly we need to insist on testing
the usability of such features as functional
relationships, feedback, undervoting, and
vote review. Otherwise we are simply
trusting to luck that such features work for
the voter. And what works for me certainly
won't work for many others.

Items 11, 12, and 13 may or may not apply to
a given voting system. Put in disclaimer "if
applicable."
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
Without an inspection how does one ensure
controls are conveniently located, that
users get feedback, that the knobs and
switches are suitable, etc.?
Dexterity required to operate controls and
ruggedness are matters for expert
determination.
Whether controls and input devices are
conveniently located, whether the controls
are familiar to most users, whether inputs
can be triggered accidentally or
unintentionally, and whether the system is
sufficiently rugged to withstand momma's
babe in arms while she votes can only be
determined through testing.
Usability testing is essential for all items
in this list, not just the last five.
Again, an inspection is a first, essential
step to evaluating navigation and system
interaction.
Testing of the distinctions between
navigation controls appears necessary.
Whether any of these functions works for
even half the voters requires usability
testing of all listed functions.
Sentence structure and voter, not user.

Voter is who needs to be alerted.

A simple, straightforward inspection of the
system for the usability requirements listed
would save much time and effort.
Whether the voting system adequately alerts
a voter to problems and resolutions is a
matter for an expert evaluation.
This section is titled evaluation methods
for usability requirements of system
response time and feedback but not usability
testing is required. Seems a pretty basic
requirement here.
missing checkmark in testing column. The
testing authority might have additional
items on procedural items
missing checkmark in expert column. an
expert should evaluate and determine if the
standards based on HFI have been meet. If
testing and usability testing are left to
determine compliance, I have concerns.
If minimizing errors is the objective then
the system must be inspected for such
potential errors.
Minimizing errors requires usability
testing. What works for one often generates
errors by another. Let us keep Murphy's Laws
firmly in mind.
This is exactly the same title as for
section 6.3.3.8.


I disagree with the testing types identified
for this item. An expert does not need to
validate that this item is available. This
should be the role of the test and
inspection
I disagree with the testing types identified
for this item. An expert does not need to
validate that this item is available. This
should be the role of the test and
inspection
I disagree with the testing types identified
for this item. An expert does not need to
validate that this item is available. This
should be the role of the test and
inspection
A simple, straightforward inspection of the
system for preventing and minimizing errors
would be a first step in evaluating the
equipment.
How these functions can be adequately
evaluated without a usability test escapes
me. Seems a basic requirement.
Sentence structure and wording




Recommendation about candidate order on
ballot should be added.
             Proposed Change



eliminate this



At the end of the last sentence insert “or
print-outs on paper”
delete this sentence and add the following
to the previous sentence. "as the DRE
presents the ballot to the voter."



remove the word option




5.3.10.5 Accessibility for Voters with
Limited Movement[,] Coordination[, and
Prosthetic Devices]
change the wording to the sentence to:
Voting equipment shall provide features that
ensure voter independence while the voter is
voting.

change wording to "To avoid confusion it is
recommended that no more than six colors be
used.


Change the sequence to 4 is third, 5 is
forth and 3 is fifth.
Change title to "Voter Input/Control Devices
and Feedback"    Further, make an opening
statement that clarifies this point.
add wording such as "It is recognized that a
touch screen inherently averages the touch
selection. For example, when 2 points are
selected on a touch screen and the driver
averaged the distance between the touches
and activates the nearest area.
add wording to beginning of sentence "When
applicable…"

Add: 5. Where a voting system provides a voter verified
ballot in a visual form (e.g. hardcopy) a system providing an
audio output for voters shall
a. Read ballot data for the audio rendition of the voter
verified ballot from a data path common with the main output
of the Cast Vote Audit Record and different from that used to
render the information in Section 5.3.4, paragraph 6.
b. Provide means for the voter to terminate audio
presentation of the voter verified at any point, including prior
to presentation. However, the system need not provide
means to reject the ballot prior to presentation of at least
Add: 5. I




replace dots by alpha numeric numbering
Move parenthesis as follows: "...bias that
would (either intentionally or
unintentionally) encourage or impose…"
remove it


remove it

remove the table. By adding the distance
variable compliance to the standard is
unobtainable. In addition, the font height
is already defined in section 5.3.10.2-1




Delete “DRE” in the first sentence. Where a specific
requirement is applicable only to DRE, call that out in the
requirement or section heading; there may be none.
Redraft and present all units in centimeters
and inches.


Delete “DRE” in the first sentence. Either
provide a table indicating applicability of
principles to systems by type (DRE, non-DRE)
or provide an indication of applicability in
each statement.
Privacy should not prevent the ability of the election officials
to determine and ensure, between voter uses, that the
election equipment is not being tampered with (or used, by a
single voter to cast more than one ballot, for example).


* Recountability - the election totals should be readily and
independently verifiable if required by law, procedure, or
litigation.


delete speech input


Add before or after (3) another list item:
"The votes on an individual machine shall be
stored in such a way that it is not possible
to differentiate votes by whether
accessibility features were used."
delete last sentence

replace with the comment to allow connection
of their assistive devices through some
common connection (is referred to later)

Add informational annex with illustrative
examples, as attached




remove item
Please add the following:
When audio content or instructions are
provided, allow the voter to select a
language in accordance with languages
available according to the Voting Rights Act
of 1965, as amended. Also, voters shall be
permitted to verifiy the ballot in the
language in which the vote was cast.
delete this point



make no refernce to fond size but change to
Font shall besuffisiant and remove text
between brackets

Should be removed - don't be prescriptive on
the implementation



Add informational annex with illustrative
examples, as attached



Change sentence to read "Alternatively, the
text size can be made adjustable.
Change the wording to read "Braille can be
provided where practical, "

clarify


clarify

clarify

clarify
Eliminate "wireless coupling" from this section -- must be
hard-wired.




delete this point

delete speech input


delet 3



Eliminate reference through allowing
assitive devices to be connected or the unit
to be repositioned to allow better access?

h. Where possible, voter should be able to use the privacy
screen to conceal their vote choices in a similar fashion to
non-disabled voters.
delete: that will allow voters who use
wheelchairs to approach the voting station
in the same orientation as voters who do not
use a wheelchair. Insert the drawing from
fec 2002 where the wheelchair voter
approaches the voting station paralllel
 Insert the drawing from fec 2002 where the
wheelchair voter approaches the voting
station paralllel

Use cm and in as units consistently.



a. Where clear floor space only allows
forward approach to an object, the maximum
high forward reach shall be [60 cm (24
inches)]. The [maximum] low forward reach
shall be [30 cm (12 inches)].



b. Where forward reach extends over an
obstruction with knee space below, the
maximum level forward reach shall be [60 cm]
(24 inches). When the obstruction is less
than [50 cm] (20 inches) deep, the maximum
high forward reach [shall not exceed 20 cm]
(4 inches) [past the obstruction]. When
the obstruction projects [50 to 60 cm (20 to
24 inches)], the maximum high forward reach
shall [not exceed the depth of the
obstruction].
c. The position of any operable control
shall be determined with respect to a
vertical plane [that is between 107 and 122
cm (42 to 48 in) above the floor] centered
on the operable control, and at the maximum
[ ] the [control shall be within [60 cm (24
in) in front of the voter].
d. Where any operable control is [26 cm] (10
inches) or less behind the reference plane,
the height shall be 137 cm (54 inches)
maximum and [60 cm (24 inches)] minimum
above the floor.
e. Where any operable control is more than
25 cm (10 inches) and not more than 61 cm
(24 inches) behind the reference plane, the
height shall be 117 cm (46 inches) maximum
and 60 cm (24 inches) minimum above the
floor.
g. Operable controls located on horizontal
surfaces [in front of the voter] shall have
a tilt of between 10 and 20 degrees to make
them easier to see and access from a sitting
position
shall have a tilt of more than 10 and less
than 90 degrees to make them easier to see
and access from a sitting position.
Consider the potential differences of use of
a right target vs. a left target.




Correct indentations.
* Use of colors should not impose external bias on vote
selection.



Tallying should be tested using all language options to
ensure votes are registered to the appropriately selected
candidate.




remove table and 1 dot referring to
character size


define the types of coulor blindness to be
addressed
The table should either contain 2 columns,
indicating minimum and preferred, or if
applicable, there should be a minimum,
maximum and preferred column. (The column
headers should abvioulsy line up as well.)
Simplify to one distance, minimum font and
minimum zoom font values
remove the requirement and address this
issue in the contrast and luminance
requirement.

narrow the ratio range to 5 to 1 to 7 to 1


remove bolding from sentence


specifically define what well-known graphics
are (i.e. a red octagon indicates stop or
warning).
remove the sentence

remove 8th bullet
he following solution is offered:

“Once the ballot is submitted, the voter
shall be prevented from making any further
modifications to the ballot or shall be
prohibited from casting another ballot.”
Informing the voter that their votes were properly registered
may consist of the preparation of a paper ballot containing a
summary of the ballot choices the voter selected, that can be
presented to the voter for verification.


For paper-based voting systems, such as mark-sense or
voter-verified, there must be a well-identified procedure for
the voter to use to "spoil" a ballot and obtain a new one. This
must be within the constraints of laws pertaining to number of
times and reasons allowed for spoilage. Ballots "spoiled"
either physically or electronically must not be entered into the
vote tally or cast ballot audit trail.
For paper-based voting systems, such as mark-sense or
voter-verified, there must be a well-identified procedure for
the voter to use to "spoil" their ballot and obtain a new one.
This must be within the constraints of laws pertaining to
number of times and reasons allowed for spoilage. Ballots
"spoiled" either physically or electronically must not be
entered into be avote tally for cast ballot audit trail. thus
There must the method or ensuring that the data
deleted has not been entered into the vote totals.




delete when possible and start with The
system shall…
change text to:          If a voter chooses
to delete data, the system shall provide
immediate feedback that the data has in fact
been deleted and the voter must be able to
reenter his or her choices


delete and a error message should be
presented
remove item


remove portion of sentence

Specify the preferred embodiment or if
neither, identify such.

delete


5. Input devices and controls shall be
designed to [minimize or] eliminate [ ]
accidental actuation. As warranted,
instructions shall be provided indicating
their proper manner of operation.
If SPL is correct then need definition and
add it to section 3.
Change separation to 3 cm (1.2 inches) and
minimum height and width of 2 cm (0.8
inches)



· If there is a "home position," the
capability for an automatic return to that
point should be provided [that is
intuitively obvious to the voter.]
revise language to reflect comment


decrease separation to .1 in
Provide no standard here, allow poll workers
to manage this time.
States that impose a time-limit for voting must be informed
about the length of time that may be necessary for all voters
(including disabled) to cast a ballot using the system. If this
amount of time exceeds the state regulation, then an
exemption must be applied.


delete last sentence



Too prescriptive and again a system feature




clarify

clarify

While the system is processing a request
(e.g., proceeding to display a new page), no
further button presses shall be recorded or
stored, i.e., type-ahead capability shall
not be provided.
Change the number in parenthesis to 1
second. Put the parenthesis at the end of
the sentence.




change to 1.5 seconds or less
We recommend using the same language as the
VSS:

g. The system should provide feedback to
user inputs in less than a second, but if
processing takes longer, feedback should be
provided that the system is processing the
voter's input.
replace with > 10 seconds, the system should
provide feedback to alert a voter that
activity is still being performed until it
is completed
combine items




We recommend dropping paragraph 2.
last sentence should read When a attempt to
overvote is made, the syystem should not
accept the choice and alert voters to the
condition.
actions can be reversible or restarted
through a ballot cancellation process
This was a poor example, remove it.


clarify
Allow red to indicate warnings.
Restate the sentence to say help can be
provided independently from the voter
interface itself.



Re-number 6.3. The first few sections in
6.3 will have to be letters under 6.3 so
6.3.1 does not get used in the intro
discussion for the testing section.
* Recountably - the election totals should be readily and
independently verifiable if required by law, procedure, or
litigation.




Add table to evaluate password protection.



Add table to evaluate roles.



Require inspection of items 2, 3, 4, 6, 7,
and 10.
Non-applicable standards should be noted as
such, with a rationale provided [in the
public test report] for their exclusion in
non-obvious cases.

added ANSI document to standards document




added definitions for testing types
Revise requirements as in following
comments.Label tables as Table 6.3.3-1
etc.or some similar convention.




Add requirement for expert evaluation to
items 2, 3, and 4.
Add requirement to test voter privacy to
item 2.
Add requirement for usability testing to
items 2, 3, 4, and 5.
Suggest: Ensure that ballots cast through
personal assistive devices are
indistinguishable from other ballots and
cannot be traced to the voter.
Require an inspection for items 1 and 3.

Require an expert evaluation for items 1 and
3.

add check to Expert column




Require an inspection for items 1, 2, and 4.




Require usability testing for items 1, 2,
and 3.

Require expert evaluation for items 1 and 2.



Require usability testing for item 1.

Require expert evaluation and usability
testing for item 1. Other testing would also
be a good idea.
Require an inspection for items 1 through 4.



Require usability testing for items 1
through 4.


Require usability testing for item 1.


Require both testing and usability testing
of means used to activate the ballot.



Fix footnote numbering.
Remove the reference.




add check to Expert column




Require an expert evaluation for item 7.

Require usability testing for items 2, 3, 4,
5, 6, 7, 8, and 9.

change wording from "One of the …" to "The
color …"

clarify




add check to Expert column




Require an inspection for items 1, 2, 4, 6,
7, 8 and 9.
Require an expert evaluation for items 1, 5,
and 7.




Require testing of items 1, 3, and 4.




Require usability testing for items 2, 3, 4,
5, 6, 7, 8, 9, and 10.




Add disclaimer [(if applicable) to items 11,
12, and 13.

add check to Expert column




add check to Expert column




add check to Expert column




add check to Expert column




add check to Expert column
add check to Expert column




Require inspection of items 1 through 13.



Require an expert evaluation for items 2 and
8.

Require testing of items 1, 3, 5, and 8.




Require usability testing for items 1
through 8.
Require an inspection for items 1, 2, 4, 5,
and 6.

Require testing for item 4

Require usability testing of items 1, 3, 4,
5, and 6.

Alert [voter] to tasks [that] must be
completed within a time limit
Alert [voter] to problems and possible
resolutions
Require inspections for items 1 through 7.


Require an expert evaluation for item 7.


Require usability testing for items 1
through 7.



add check to testing column


add check to Expert column




Require an inspection for items 1 through 5.
Require usability testing for items 1
through 5.


If this section actually differs from
6.3.3.8 then provide unique title. Otherwise
combine the two sections into one table.

remove check from expert column and add
checks to Inspection and Testing



remove check from expert column and add
checks to Inspection and Testing



remove check from expert column and add
check to Inspection



Require an inspection in steps 1, 3, 4, and
5 (or 6,8, 9, and 10 if tables are combined)


Require usability testing for items 1
through 6 (or 6 through 11 if tables are
combined).
Therefore, the information presented below
provides guidance for the presentation of
ballots[ ]. These principles should be used
in conjunction with the general information
presentation principles in Section 5.3.3 to
make ballots more effective from a usability
point of view [for computer displays of
ballots]. It is recognized that State laws,
in many cases, govern ballot design.

Add bullet: Candidates should be listed in
some random order, not simply alphabetically
or by party affiliation, e.g, Republicans
should not always be listed first.
     Proposed Resolution
(by the Chair on each comment submitted)
NC - Requirement is not clear and it is not clear that it could
be implemented as stated.

Mercuri - Comment needs to be addressed by working
group.




NC - See response to corresponding comment on 5.3.10.2.

Mercuri - Comment needs to be addressed by working
group.
NC - out of scope for this section on voter interfaces. Other
users of voting systems will be addressed in a later revision
of the standard.

Mercuri - Need for poll workers to ensure that the voting
device is not tampered with should be addressed here in a
NC - out of scope for in section on voter interfaces. Other
footnote, or elsewherethisthe standard.
users of voting systems will be addressed in a later revision
of the standard.

Mercuri - Recountability is a requirement of election law and
must be addressed.
NC - T-Coil wireless coupling is well established and very
important for a segment of the population with hearing loss.

Mercuri - Wireless coupling may be appropriate for non-
voting applications but may void privacy requirements here.
This should be discussed.




NC - Need for privacy already a general requirement.

Mercuri - Privacy screen may need to be different for
disabled voters. This should be separately addressed.
NC - Addressed in item D042 above

Mercuri - 5.1.3.2.4 does not also mention specific times
when certain access shall be disallowed. Comment should
be reconsidered byfor this section on voter interfaces. Other
NC - out of scope working group.
users of voting systems will be addressed in a later revision
of the standard.

Mercuri - Insufficient testing for language options has been
noted in equipment failures in actual use. This must be
addressed.
C - Add the comment that "If voter verification via a paper
record of the ballot cast is provided it shall inform the voter
that their votes were properly registered and contain a
summary of the ballot choices the voter selected, that the
voter can veri

Mercuri - The change does not appear in draft v5.0
C - The end of the footnote is changed to read:

The design should make it clear where and how to vote, how
to change a vote which has not been cast and the system
shall provide feedback that the vote was accepted by the
system. The guidance in this sec

NC - Already addressed in 5.3.8, Item in
Mercuri - The change does not appear 5 draft v5.0

Mercuri - Ballot spoilage needs to be clarified, since it may
not be possible to "reverse" a paper-ballot selection, hence
"spoilage" would need to be invoked. 5.3.8 5) does not
sufficiently address this issue.
NC - out of scope for this section on voter interfaces. Other
users of voting systems will be addressed in a later revision
of the standard.

Mercuri - This also must be addressed with the spoilage
issue above.
C - Changed in incoporate time-limit indication. Exemption
item not included as it is up to local policy.

Mercuri - The time-limit indication may be confusing or
troubling to some voters. If the machine "times-out" though,
this could be very upsetting. What I meant by the comment
was that the amount of time a voter might need to prepare a
ballot may exceed the regula
No change - Recountabilitiy is out of scope for this section. It
addresses testing for Section 5.3 which addresses voter
interfaces and not those of other users. These will be
addressed in a future revision.

Mercuri - Recountability still needs to be addressed, since it
is a component of election law and impacts usability.
Reference Information   Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

                Date: 9-30-03 P1583 Draft 5.0 August 2003
                      Document:
                 Commenter Clause/ Subclause        Paragraph              Type of
                 and Number                           Figure/ Table       comment
                                                                          (General/
       #                                                                  Technical/
                                                                           Editorial)
           1    Alice - 006          5.6.2.4     a                            T




           2    MercuriD50 - 024     6.6.4.2     i.                           T
                (formerly mercuri-
                056)


           3    Jhulshof-033          5.6.6      opening                      T
           4    Jhulshof-034          5.6.6      a                            T
           5    Jhulshof-037         5.6.8.3.3   p                            T
           6    Lipsio-56              5.6                            G




           7 Corry-071                 5.6       Entire section               T


           8    Lipsio-0A              5.6                            T




           9    Lipsio-0B              5.6                            T


           10
                Sklein-007             5.6       Para 5.6.1.1                 T
11   Lipsio-34        5.6 , 6.6.4.1 , 6.6.4.1.2               T




12   Lipsio-35            5.6, 6.6.4.1, 6.6.4.1.2             T




13   Lipsio-36            5.6, 6.6.4.1, 6.6.4.1.2             T



14   Lipsio-37            5.6, 6.6.4.1, 6.6.4.1.2             T




15   Lipsio-87                    5.6.1             Para. 3         E




16   Lipsio-38                    5.6.1             para. 2   T




17   Lipsio-88                   5.6.1.1            para. 2         E


18   MercuriD50 - 064            5.6.1.1            Section       General
     (formerly mercuri-
     143)
19   Alice - 001   5.6.1.1                       T




20   Lipsio-39     5.6.1.1   Para. 2         T




21   Lipsio-3A     5.6.1.1   para. 3         T

22   Lipsio-3B     5.6.1.1   para. 3         T




23   Lipsio-3C     5.6.1.1   Para. 3         T




24   RGH 078       5.6.1.1                       T

25   Alice - 005   5.6.1.1   2nd Paragraph       T

26 berger - 008    5.6.1.2                       E
27   RGH 079              5.6.1.2                 G




28   Lipsio-3D            5.6.1.2           T




29   MercuriD50 - 069     5.6.10.2   list       General
     (formerly mercuri-
     161)

30   MercuriD50 - 068     5.6.10.2   i          General
     (formerly mercuri-
     160)
31   Lipsio-89             5.6.2                  E




32   Lipsio-8A            5.6.2.1                 E

33   Alice - 008          5.6.2.1                 G
34   Alice - 002           5.6.2.1                         T




35   HD-022                5.6.2.1                         T




36   Lipsio-8B     5.6.2.1, 5.6.2.3- 5.6.2.7               E

37   Lipsio-3E             5.6.2.2         para. 1     T




38   Lipsio-3F             5.6.2.2                     T




39   PPLX-034       5.6.2.2




                                           5.6.2.2
                                           Software
                                           Integrity       T
40   Lipsio-40             5.6.2.2.        bullet 3    T
41   Lipsio-41     5.6.2.2.   bullets      T




42   Lipsio-42     5.6.2.2.   para. 1      T



43   Lipsio-8C     5.6.2.3    Bullet “c”       E
44   Lipsio-8D     5.6.2.3    Bullet “c”       E




45   Lipsio-8E     5.6.2.3                     E


46   Alice - 010   5.6.2.3    c                G

47   Alice - 009   5.6.2.3    a                T



48   Lipsio-43     5.6.2.3    Para. 1      T




49   Lipsio-44     5.6.2.3    Para. 1      T




50   Lipsio-45     5.6.2.3    Para. 1      T
51   Lipsio-46          5.6.2.3                     T



52   RGH 117            5.6.2.3                         T




53 schneidewind - 005   5.6.2.3   Pg. 70                T



54
     Sklein-044         5.6.2.3   First paragraph       T


55
     Sklein-045         5.6.2.3   First paragraph       T




56
     Sklein-057         5.6.2.3   5.6.1.1               T
57   PPLX-035             5.6.2.3




                                              5.6.2.3
                                              Software
                                              Modularity and
                                              Programming         T & E
58   Lipsio-47             5.6.2.3, 5.6.2.7                    T
                                              Bullet “c”, Bullet “a”




59   MercuriD50 - 065          5.6.2.4        control constructs list   General
     (formerly mercuri-
     147)




60   Alice - 003               5.6.2.4                                    T
61   PPLX-036      5.6.2.4




                                   5.6.2.4 Control
                                   Constructs           T and E
62   Alice - 011       5.6.2.5     c                       G




63   Lipsio-8F         5.6.2.7     Bullet “d”              E


64 Corry-072           5.6.4.1     a)                      T




65 Corry-073           5.6.4.1     c)                      T




66   RGH 080           5.6.4.1.c                           T


67 Corry-074           5.6.4.2     After 1st sentence      T




68 Corry-075           5.6.4.2     2nd sentence            T
69 Corry-076                  5.6.5.1      c)                              E

70   Lipsio-90                5.6.5.2                                      E




71   PPLX-038             5.6.5.2          5.6.5.2 Voting
                                           Variations                      G
72   MercuriD50 - 066         5.6.5.2      Voting variations list.       General
     (formerly mercuri-
     153)




73   Lipsio-48                 5.6.6       Bullet “a”                T




74 Corry-077                  5.6.7.1      First sentence                  E


75   RGH 081                  5.6.7.1.c                                    T


76 Corry-078                  5.6.7.2.1    f) and g)                       E




77 Corry-079                  5.6.7.2.1    Second a) and b)                E


78 Corry-080                  5.6.7.2.1    Second a)                       T




79   Lipsio-49                5.6.7.2.1                              T




80   RGH 082                 5.6.7.2.1.e                                   T


81   Lipsio-91                5.6.8.1.1    Bullet “b”                      E
82   Lipsio-4A            5.6.8.1.3   Bullet “a”        T



83   RGH 083              5.6.8.2.c                            T


84   RGH 084              5.6.8.2.d                            E
85   Dill-29               5.6.8.3                             E


86 Corry-081              5.6.8.3.2   j)                       E




87   MercuriD50 - 067     5.6.8.3.2   f                     General
     (formerly mercuri-
     158)




88 Jhulshof-036           5.6.8.3.2   c                     general
89 Corry-082              5.6.8.3.3   i)                       E




90 Corry-083              5.6.8.3.3   o)                       E




91 Corry-084               5.6.9      First sentence           E




92 Corry-085               5.6.9      Second sentence          T


93   RGH 085              5.6.9.1.e                            T
94   Simons - 021          5.6.9.2    the list                 G

95 Corry-086              5.6.9.2     k)                       T



96   Lipsio-75              6.6                                E
97    Lipsio-57             6.6                        G




98 Corry-146                6.6     Entire section           T




99    Alice - 004          6.6..2                            G


100   MercuriD50 - 021     6.6.1    End of paragraph       General
      (formerly mercuri-
      047)




101   MercuriD50 - 022     6.6.2    Paragraphs 2-4         General
      (formerly mercuri-
      048)




102   Dill-35              6.6.2                             T


103   Dill-36              6.6.2    Paragraph 3              T

104   Lipsio-4C            6.6.2    para. 1            T



105   Lipsio-4D            6.6.2    para. 3 & 4        T




106   Lipsio-4E            6.6.2                       T
107   Lipsio-4F               6.6.2                            T




108   Lipsio-50               6.6.2                            T


109 schneidewind - 007        6.6.2       Pg. 107                    T




110 schneidewind - 001        6.6.2       Pg. 107                    T




111   Lipsio-51            6.6.3, 6.6.4                        T

112   MercuriD50 - 023        6.6.4       End of paragraph         General
      (formerly mercuri-
      049)




113   Dill-37                6.6.4.1      Replacement rule discussion E



114   Lipsio-92              6.6.4.1                                 E


115   Alice - 012            6.6.4.1                                 G



116   Alice - 007            6.6.4.1                                 T




117   Lipsio-52             6.6.4.1.2                          T

118 Corry-147                6.6.4.2      r.                         E

119   Lipsio-93              6.6.4.2                                 E
120   Alice - 013          6.6.4.2                            T




121   Dill-38              6.6.4.2     item c                 T
122   HD-018               6.6.4.2     I.                     T



123   Lipsio-53            6.6.4.2     bullet “n”       T



124   Lipsio-54            6.6.4.2                      T




125   Dill-39              6.6.4.5     item w                 T




126 Corry-148              6.6.5.1     Last two sentences     E




127   MercuriD50 - 025     6.6.5.1.2   Last paragraph       General
      (formerly mercuri-
      061)




128   Alice - 014          6.6.6.1.1                          G

129   Dill-40              6.6.6.1.2                          T
130   HD-019   6.6.6.3   G


131   HD-020   6.6.6.4   G
                        Comment



Should emphasize that only the structured
programming method, meaning controlsof
Sequential, Conditional and Loop, can be
accepted. Instead of using the specific
command IF-THEN-ELSE, DO-WHILE, CASE
Line length constraints here can impose splitting of code into
non-functionally appropriate groups. For example, a table
initialization might be longer than 240 lines in length. Add
sentence before last one.

a system is registering and not tabulating

a system is registering and not tabulating

ballot position registers are for mechanical
machienes not supported by HAVA
As agreed upon in the past with Steven
Berger, I am rewriting this section, and so
am not commenting on every detail that
methinks should be changed, but only on a
few sundry points.
This section requires rewriting from stem to
stern.

This section does not address stack
overflow.




This section does not address runtime
exceptions.

Unmodified COTS must be evaluated at the
source code level to protect against the
threats identified in 5.3.2.1 (A).
The section is written assuming a single-
threaded application and, as such, prohibits
interrupt service routines as well as multi-
tasking and also does not address the
serious consequences that can arise from the
illicit interaction of the threads.


This section is written assuming a single-
threaded application which is assumed
elsewhere throughout the document.




This section is written assuming a single-
threaded application and, therefore, does
not address the serious problems that can
arise from sharing of data by threads.
This section is written assuming a single-
threaded application and, therefore, does
not address the serious problems well-known
in real-time multi-tasking systems.


“accepted and proven industry standard
software design methods and tools” should
also reference appropriate IEEE standards.




The definition of “firmware” should also
refer to its being fixed and irremovable
without opening the machine; since self-
modifying code is explicitly excluded, there
is otherwise no meaningful reason to
distinguish firmware from other software.
“Industry standard” is not defined.


Concerns addressing use of COTS products need to be
added.
"source code generated by COTS code
development package and embedded in software
modules for compilation or interpretation
shall be provided in human readable form"
Some newer programming tools do not
necessary generate traditional source code
as reference within this clause.

“Unmodified third-party software is not
subject to code examination” is very risky
and ” is contrary to such other mission-
critical methodologies as those used by the
FDA and FAA, and contradicts what is
specified in section 5.1.3.3.2.
“formal tests” needs to be explained

“Unmodified third-party software is not
subject to code examination” is contrary to
such other mission-critical methodologies as
those used by the FDA and FAA, and
contradicts what is specified in section
5.1.3.3.2
“The vendors shall submit … a record of all
user selections made during software
installation … [and] a record of all
configuration changes made to the software
following its installation” implies that the
code is variant, excluding the possibility
of V&V having been performed on the entire
software image.
Exclude db scripts from coding
considerations
Besides it is not realistic to translate the program language
source code to be plain English.
Change 'the standards' to 'this standard' or
in some cases to 'this standard'.
Add comments indicating purposes and
instructions/intent related to source review




“examination... to verify that the code is
unmodified and that the default
configuration options have not been changed”
should be expanded to validate all legal
configurations.

Add item




Also provide for separate information from individual voting
machines, especially useful in case of malfunction.

“The software used by voting systems is
selected by the vendor” appears to mean
“COTS is selected”; else, it contradicts the
subsequent sentence. Change the opening
words from “The software” to “The COTS
software”.
 “operating system software may be designed
in assembly language” makes no sense.
Why is the writing of the operating system
in this manual? Aren't there already IEEE
Standards for Coimputer operating systems.
This section, which requires the use of
high-level for logical operations thus
precluding the use of assembly language, was
ported from the FEC 2002 standards. This
restriction was not included in the FEC 1990
standards. It assumes that good
maintainable code that meets all the other
standards requirements cannot be implemented
in assembly language. I don't believe this
is true and I've been told that the reviewer
of code for Wyle indicates that he's
reviewed prefectly good code that meets all
requirements implemented in assembly
language.
These are not requirements; they can not be
tested to.
Industry standard COTS compiler and runtime
interpreter both is not defined and assumes
that, contrary to reality, something is fail-
safe and fool-proof by virtue of being in
common use.
“to prevent accidental or deliberate
attempts to replace or modify executable
code” can be accomplished in a fool-proof
manner by disallowing writes to memory
containing the code.



The draft states: “Self-modifying, remotely
or dynamically loaded code” We do not
believe that any vendor can comply with the
section in boldface. Almost all operating
systems use some form of demand paging. We
believe that demand paging could be
construed as a form of dynamically loaded
code. Demand paging is transparent to, and
out of the control of, system designers.
“Dynamic memory” is rarely or never used in
real-time or mission critical systems
because it is indeterminate, that is, its
use can lead to a crash at unpredictable
times under circumstances that may or may
not have been tested for.
Insufficient features in “Where the
development environment … includes the
following features”


The control that the software provides
should apply not only where “Where the
development environment … includes the
following features”
“imbedded” should be “embedded”
“Library modules…” makes no sense in the
most common use of the term “library” as a
precompiled and partially linked collection
of modules; it is not useful or generally
possible to add a header to an object file.
Numerous requirements are merely suggestions
that do not conform to current industrial
practice.
Are these header comments in the coding for
whom?
All modules of a program may not be able to
stand alone. Calculations or table builds
may be performed prior to a module being
called.
“COTS software is not required to be
inspected…” is contrary to such other
mission-critical methodologies as those used
by the FDA and FAA, and contradicts what is
specified in section 5.1.3.3.2
There is implied a lack of testing in “COTS
products require updates due to a detected
security breach or vulnerability”; nothing
that requires an update should pass testing.




There is implied a lack of testing in “the
most recent version of the COTS product
incorporating all security patches” ”;
nothing that requires an update should pass
testing.
Requirements and suggestions are intermixed.



"…COTS software …must be the most recent
verion of the COTS product …"     The most
recent version is not always stable enough
to deploy and may not be compatible with the
other aspects of the application.   The
vendor must have the latitude to employ the
COTS versions and upgrades at the
appropriate time.
Why specify that COTS software must be
designed in a modular or object oriented
fashion and not inspect it for compliance?

COTS must meet the requirements of 5.1.3.1


COTS virus detection programs are not
available for all operating systems.




COTS evaluated should include compilers,
libraries, and any other software tools used
in system development and capable of
introducing backdoors or other malicious
code.
This section of the draft has this language:
“However, COTS software is not required to
be inspected for compliance with this
requirement but must be the most recent
version of the COTS product incorporating
all security patches,” [emphasis added]
This section may be ambiguous.    Must the
latest version always be incorporated or
only the latest version of security patches?
What if the security patch is not relevant
to the particular operation.

In any case, forcing the latest version of
COTS software is a configuration control
nightmare and will result in endless re-
qualification. One interpretation of this
section is that software written to run on
Windows 2000 must be rewritten and re-
qualified to run on Windows XP even if it
runs perfectly well on Win2000. An even
worse interpretation requires vendors to
update hard disk controllers with new
firmware and drivers every time a new
software version is available. We don’t
think this is intended nor desirable.

Note the term “Module”. The term Module is
used here as it is used in the FEC VSS and
we believe this usage to be non-standard. A
module should be a collection of related
subroutines and functions. A module may
“Headers are optional for modules of fewer
than ten executable lines” is inconsistent
with common practice in mission-critical
software and, for example, violate FAA
rules (at least as of 1993) where all
modules must contain 20 or fewer lines of
code, have a full header, and no more than 1
module is allowed in a source file.
Additions as described.




What does "No other constructs be used to
control program logic and execution" mean?
This specification prohibits the GoTo
construct; a good idea. However, some
programming languages require GoTo for so-
called ON conditions. For example consider
Visual Basic (a language expressly allowed):

On Error GoTo ErrorHander

Other languages may have other “On”
conditions:

On Timer GoTo TimeoutRoutine



Why should names differ by more than a
single character? What does this do for the
logic? If there are multiple porgrams that
have the same general function, but
reference different tables, changing the
name would make more sense and relate the
programs.
“so that its executable lines can be clearly
understood” implies that the rest of the
lines do not need to be understood
Specifies reporting errors to election
officials and voters but no requirement to
put these messages in audit log.



What is an "easily understood language" is
undefined.




eliminate or printed - may not be for
performance reasons

Status messages for the voter must be in the
language selected by the voter.




Messages need to be stored in audit log as
well as memory. Also, it may not be
necessary to store them in memory if they
are stored in the audit log.
No location is "indestructible" Suggest use
of "permanent."
“of the 50 states” excludes US Territories
or the possibility pf election laws being
more local (as is the case in many places
where local elections use proportional
representation or ranked-order voting).

Please add Cumulative Voting.
Additions as described.




The ballot counter need only be recorded at
commencement of an election; it need not be
0. It may be a good security feature if it
is never reset so that both testing and
tampering are recorded.
Programs must be certified before being
installed.

how does fw match ballot styles
automatically to poll place intended?

Correct English. Ensure rather than assure.




These sub-sub-subsections should be numbered
g) and h) to be consistent with 5.6.7.2.1
Common Standards
Elements need to be certified as well as
proven.



For security and to detect tampering, a
checksum of the code’s image must be emitted
and validated.



consolidated reports within a poll place but
for a specific equipment type - can have
mixed equipment
Wording leaves something to be desired.
Need to log all attempted actuations,
authorized or not, successful or not.


machine may not automatically prevent ballot
not entitled, some of it is procedural

can be automatic or procedural
This subsection seems to duplicate information that is
discussed elsewhere. This lengthens the standard and risks
inconsistency.
Incomplete sentence. Doesn't specify who
should get the message if a jam.




Add "appropriately"




voting secrecy is abused
No provision for paper verified ballots.




Missing right parentheses.




Sentence structure and punctuation.




Applies only if results transmitted to
another facility.

can be automatic or procedural
The requirement of Voter Verifiable Paper
Ballots is missing.
Applies only to DRE machines.



Much of the verbiage in this section is
redundant to wording in section 5.6.
As agreed upon in the past with Steven
Berger, I am rewriting this section, and so
am not commenting on every detail that
methinks should be changed, but only on a
few sundry points.
This section should be completely rewritten
to more exacting standards. Vince Lipsion
has offered to do that and I will defer
detailed comments and acceptance until I see
his version. Some minor comments follow on
existing version.
What is human readable form? Why is it
required?

The paragraph does not reflect current computer science
theory regarding inability of confirm proper functionality via
source code review. Clarification needs to be added. Citation
to Ken Thompson 1984 CACM "Reflections on Trusting
Trust" paper could be added here.




The decision by the FEC to exempt COTS products from
inspection has created a serious security flaw. It should not
be imperative that the IEEE standard continue to reflect this
inappropriate practice. All exemptions for COTS product
review should be removed from this standard.




If COTS hardware or software is in the trusted subset, it must
be treated exactly like software or hardware designed by the
vendor.
It is not possible to find all malicious code or back doors.

“All software components ... shall be tested
… after every update or modification is
completed” allows a loophole or is
ambiguous.
“Unmodified, general purpose COTS non-voting
software ...is not subject to code
examination...is not subject to the full
code review and testing” is contrary to such
other mission-critical methodologies as
those used by the FDA and FAA, and
contradicts what is specified in section
5.1.3.3.2.
Insufficient specification.
Insufficient specification.




Insufficient specification.


COTS software must work in conjunction with
the voting application software. Therefore,
it should be subjected to the same rigor of
testing as the application software.

How do you know that the COTS software has
not been modified?
COTS software should not be exempt from code
inspection.

Insufficient specification.

Add statement regarding review for malicious code.




This kind of thing was popular in the 60's and 70's, but looks
really archaic to me. I think everyone knows the basic
control constructs now. This just unnecessarily lengthens
the document. noting to do with testing and so
These have
belong with their less verbose redundancies
in section 5.6.2.4.
Flowcharts not necessary for persons with
programming knowledge.


Control Constructs should not limited to the
listed programming language commands and
should not be so detailed. Because some
newer high level computer languages use
different commands but do the similar thing.
For example PL/SQL uses FOR lopp instead of
DO-WHILE, JAVA uses SWITCH instead of CASE.

“Unless obvious from the process” is too
subjective to predicate a requirement on.
Put illustration of levels into table so
that it stays together.
These have noting to do with testing and so
belong with their less verbose redundancies
in section 5.6.2.6.
Hoew does PL/SQL,SQL,Oracle Developer fit
into the coding convention?




I don't know what a macro "exit point"
The last sentence "The vendor shall justify
any module lengths…" does not identify who
determines the validity of the
justification.
Warning about “mixed mode operations” should
explicitly note (for the C Programming
Language) consideration of mixing signed
with unsigned types.
Entities specific to the C Programming
Language are freely intermixed with generic
programming conventions; also, most of these
conventions are really best practices, not
requirements.



In some cases, it would be better if assertion violations were
logged. The exceptions are when there are too many of
them, or when voter privacy would be compromised.




Make last two sentences seperate paragraphs.




Need to add sentence at beginning of this paragraph,
preceeding "In this situation…"




Who is perfroming the system build for
testing?
Use of "concurrently read" seems to require that both sides
of a ballot be read simultaneously. This is too prescriptive.
This section is essentially the same as
section 6.1.3

This section is essentially the same as
section 6.1.4
                      Proposed Change




Modules should be constructed as to be grouped according
to functionality. The vendor shall justify any module
lengths….


device registers ballots shall provide a
public and a protective counter. The
must be set to zero before any ballots or
cvr are registered
delete

Re-write of section (in process).




Vince Lipsio to rewrite section 5.6 and
submit for review. Comments on specific text
in this section follows.
Add section specifying the monitoring of
stacks at run-time. Example wording
(paraphrased from a real SRS approved by
the FDA): “Each stack’s high water mark
shall be monitored at least once a second;
60% usage of any stack shall be logged as a
software anomaly and 80% usage of any stack
shall result in a fatal software error .”
Add section specifying minimal requirements
for the handling of hardware and software
exceptions.
Delete “Unmodified third-party software is not subject to code
examination; however,” and replace it with “All third party software
shall be subject to source code an d other examination to preclude the
presence of trap doors, hard-coded passwords, vulnerabilities and
other non-deliberate errors, deliberate errors allowing the introduction
of malicious code, and malicious code of any kind, especially
malicious code intended to trigger upon use of the software in voting
systems.”
Eliminate reference to single-entry, single
exit procedures. Eliminate prohibition of
infinite loops. Add sections do deal with
shared data and semaphore requirements. Add
section to deal with race conditions, dead
tasks, tasks being permanently excluded from
running due to erroneous pre-emption or
waits, etc.
Eliminate prohibition of infinite loops
(because these are the typical
implementation of a task); eliminate
reference to single-entry, single exit
procedures (or substitute something that
makes sense in the context of a multi-
threaded system).
Add section to specify safe use of shared
data and requirements for semaphores.


Add section to deal with race conditions,
dead tasks, task deadlocking, tasks being
permanently excluded from running due to
erroneous preemption or waits, and other
such inherent problems of multi-tasking in a
real-time system.
Reference (indirectly here in the scope,
directly throughout the section) at least
the following: IEEE Std 1228-1994, IEEE Std
829™-1998, IEEE Std 1028™-1997, IEEE Std
1471™-2000, IEEE Std 1016™-1998, IEEE Std
14143.1™-2000, IEEE Std 1061™-1998, IEEE Std
1061™-1998, IEEE Std 1008™-1987 (R1993), and
IEEE Std 1228-1994.
Add the phrase “and embedded in the device
so as to require physical modification to
the device to alter” to the definition of
firmware.


Eliminate all such subjective references or
move them to an annex of recommended
practices.
COTS products, especially software libraries, are a
vulnerable attack point and must be subject to risks
assessment prior to use in voting products. Configuration
management should include vendor updates and alerts when
flaws are detected that could compromise election
operations or cast ballot data integrity. Object code modules
should be provided such that compiled versions of programs
can be compared.
Delete this clause




Ensure compliance with section 4.3.11
(“Previously developed or purchased
software”) of IEEE Std 1228-1994, “IEEE
Standard for Software Safety Plans”.


A more complete description of “formal
tests” should be given.
Eliminate the phrase “Unmodified third-party
software is not subject to code examination”




Require a single, invariant code image for
the system, thusly forcing all these
concerns to be addressed when the software
is built, before testing and V&V, and
require a checksum of the validated and
released binary image to be recorded and
checked upon initialization of the system.

add comments
The intention of the Voting Systems
Standards Source Inspection Process for
Independent Test Authorities (ITAs) is to
prevent deviant or malicious code from being
introduced into the voting process; to
safeguard from external threats being able
to effect unintended changes to voting
processes or data corruption to occur as a
natural part of voting processes. The ITA
is obligated to identify any such threats
and the submitting vendor is expected to
resolve these occurences before approval.
In order to review this code properly, an
ITA must be able to follow the code clearly
to ascertain whether any of these defects
have occurred - this has been addressed as
readability and maintainability guidelines
in the standards. When an ITA cannot
understand the code that has been submitted
for review sufficiently to determine if any
dangers exist, their request for changes and
improvements for readability should be
enumerated using items defined in the
guidelines. The ITA reviewing the source
has the lattitude to request changes using
these guidelines only when they feel the
Add: “and all configuration items allowed by
the system shall be validated. Furthermore,
each configuration item shall be range
checked when fetched so that hardware
malfunction or tampering can not allow an
untested configuration to be used.
* Ensure and confirm that extracted ior duplicated nformation
is identical to that on the original cast ballot storage medium.



Provide independent reports for each ballot casting device as
well as consolidated one.




Change “designed” to “implemented” or
“coded”.
Delete. Should have its own IEEE Manual and
refernce that manual.
Since assembly language is not considered a
high level language, Delete the inclusion
of Assembly language within this section.
Alternatively, if an operating system
software may be designed in assembly
language such operating system software is
required to meet all the same provisions or
requirements any other operating system
software is subject to.
This restriction should be removed as long
as a modern compiler is used that permits
the source code to written symbolically and
it follows all the rules required of the
source code written in higher level
languages.




Move sections to an addendum

Require all tools, including compilers and
interpreters, to be validated and verified
in the same manner as application software.


Add requirement that all executable code
shall either reside on ROM or shall be write-
protected while the device is operational;
if the latter, the code’s checksum shall be
generated after disabling the write-enable
signal to the memory containing the code
image.




Prohibit dynamically allocated memory or, at
least, force graceful system reset if
dynamic memory exhausted.
Add bullets to check all cases in a
switch/case statement, and to check all
subscripts when referencing an array, and
range-check all data when writing or
fetching.
Add: “else, the application software shall
explicitly check to ensure against such
situations”.

Change “imbedded” to “embedded”
Change “header” to “readMe file”.




Nothing   presently; I'll enumerate these and
convert   them into recommended best practices
at some   future time.
Delete.   Common in structured programming.

All modules thant construct a function
should be tested together, including library
modules. Actual modules should be used for
all test versions.
Eliminate the section, or, better yet,
reverse its sense.



Mandate that testing preclude any security
breach or vulnerability; mandate compliance
with section 4.3.11 (“Previously developed
or purchased software”) of IEEE Std 1228-
1994, “IEEE Standard for Software Safety
Plans”. Mandate COTS be subject to the
specifications of IEEE Std 1008™-1987
(R1993), “IEEE Standard for Software Unit
Testing”. Add reference to IEEE Std 982.1™-
1988, “IEEE Standard Dictionary of Measures
to Produce Reliable Software”.
Bring into conformance with Annex D (“V&V of
reusable software“) of IEEE Std 1012-1998,
“IEEE Standard for Software Verification and
Validation”, e.g., “Reusable software (in
part or whole) includes software from
software libraries, custom software
developed for other applications, legacy
software, or commercial-off-the-shelf (COTS)
software. The V&V tasks of Table 1 are
applied to reusable software just as they
are applied to newly developed software.
However, the inputs for these tasks may not
be available for reusable software, reducing
visibility into the software products and
processes.“
Change everything that can be tested against
to a “shall” sentence with objective
criteria. Move everything else to an
addendum of recommended practices.
Remove this clause.




Either eliminate the requirement or inspect
for compliance.


In the second sentence, after “security
requirements defined in” insert “Section
5.1.3.1 and”.
In the second sentence, replace the comma
after “security patches” with “and”.
Replace “and must be tested” by “. In
complying with the requirement of 5.1.3.1,
the vendor must document how the COTS has
been defended against the threats identified
in 5.1.2.3 (A-1), (A-3), (B-1) and (B-2),
such as by testing”.
COTS to be evaluated shall include
compilers, libraries, and any other software
tools used in system development and capable
of introducing backdoors or other malicious
code.
This section has several problems. The
module usage should be changed to subrouting
or function, remove the strict requirement
of only one exit per subroutine or function.
Change so the most recent version of COTS is
not required.
Eliminate “Headers are optional for modules
of fewer than ten executable lines”.




While should be added to Do-While. Recursion should be
allowed only if detailed description of need added.
Concurrent process flow should be allowed if detailed
description of need added. Assignments within branch tests
should be avoided (ex. use of if (a=b) should not be used,
rather, do the assignment outside of the test).



Delete this clause. Different languages
have different constraints which perform
this same function. Such limitation in the
standard, tends to restrict or lock out
newer programming techniques, in favor of
older techniques.    Is the intent here to
issue a new standard fot older technologies?
The specification should be revised to state
that GoTos should be disallowed except where
required by the programming language.
Delete. Names should be more than one
position and be discriptive, but a name that
changes by one position does not change the
functionality .



Change “its executable” to “all”.


Add requirement to include in audit log: a)
The system shall generate, store, and report
to the election officials and, where
appropriated, the voter all error messages
as they occur [and all such messages shall
be written to the audit log];
Add definition: c) All error messages
requiring intervention by an operator,
precinct official or voter shall be
displayed or printed unambiguously in
[English and the language selected if the
error affects the voter or their vote], or
by means of other suitable visual indicators
without compromising voter privacy;

replace with idea printed error messages
could be codified and referenced for
performance/execution constraints
Add after first sentence: The system shall
display and report critical status messages
using unambiguous indicators or English
language text. [If voter interaction to
correct the error is required the system
will also display the error message in the
language selected by the voter.]
The system need not display non-critical
status messages at the time of occurrence
and may be stored in memory [and the audit
log] to be recovered after ballot processing
has been completed.
c) Register and accumulate votes in a secure and
[permanent] location;
Change “50 states” to “sundry
jurisdictions”.




Add cumulative voting
r) No vote selected s) Blank ballot cast. * All of these
variations must be implemented in such fashion that it is
readily discernable that: a) votes are registered to the
appropriately selected candidate and b) tallies reflect the
algorithm properly (as in N of M or IRV), considering under-
and over-votes correctly.


Change “set to zero” to “recorded in the
audit log”.



All systems shall provide a means of
installing ballots and [certified] programs
on each piece of polling place equipment...
election data is organized to link only
allowable ballotstyles to poll place
specific locations
f) Segregate test data from actual voting
data, either procedurally or by
hardware/software features and [ensure] that
reported results cannot combine actual
voting data and test data.
Renumber to be consistent.


Renumbered: g) These elements shall be
capable of being tested separately, and
shall be proven [and certified] to be
reliable verification tools prior to their
use; and
Add bullet requiring a checksum must be
emitted and verified with some central
agency, preferably using a challenge so that
the person starting the device can not know
the expected result nor produce it on
her/his own.
refer to comments


Change to: “Automatic disabling of any
device until self-test has successfully been
completed.”
Add “and all attempts of such actuation,
whether or not successful, shall be entered,
with a timestamp, into the device’s audit
log”.
election data is organized to link only
allowable ballotstyles to poll place
specific locations
add "within the same voting session"
Delete section.


j) Prevent or detect the attempted feed of
overlapping ballots, and if detected halt
the reading of the ballot and provide a
message to [election officials, the
operator, and the audit log] identifying the
condition;
…without correction, appropriately tallying all validly voted…




delete or appropriate election official
i) For electronic image displays [or paper
verified ballots], prompt the voter to
confirm the voter's choices before casting
the ballot, signifying to the voter that
casting the ballot is irrevocable and
directing the voter to confirm the voter’s
intention to cast the ballot;
o) Provide a capability to retrieve CVRs in
a form readable by humans (in accordance
with the requirements as specified in the
DRE System requirements subsections of
Section 5.2[)] without compromising voter
privacy;
All systems must provide a means to close
the polling place, provide capabilities to
accumulate and [report] results [ ] for the
jurisdiction, and to print audit trail[ ]
reports.
If the system provides the capability to
transmit results [to another facility],
additional requirements apply.
refer to comments
Add the following: "Produce a Voter
Verifiable Paper Ballot;"
k) Ensure that extracted or duplicated
information is identical to that on the
original Cast Vote Record storage medium
[for DRE machines].
Identify the redundant wording and eliminate
it.
Re-write of section (in process).




Lipsio to rewrite and submit for review and
comments. Defer acceptance of this section
until that is done.



Delete this clause.             The computer or
machine should be able to read and execute
the code, not a human.
Current computer science theory recognizes that no amount
of source code review and functional testing is capable of
confirming that a system is free from bugs, defects, or
nefarious code that could compromise election operations.
Therefore, it is imperative that systems include sufficient
features necessary to ensure that ballots cast are recorded
and tabulated as per their voters intentions.


Remove all exemptions for COTS product review from this
standard on the grounds that such pose a serious security
flaw. COTS products shall be presented in their entirety for
open review in the same way that vendor software is
examined.




Specify that the COTS exclusion only applies
to system components outside the trusted
subset.
Specify that inspection can be done to ATTEMPT to find
back doors, etc.
Rephrase so that it is clear that upon
modification of any component, the entire
system must undergo regression testing.

Eliminate the sections; ensure compliance
with section 4.3.11 (“Previously developed
or purchased software”) of IEEE Std 1228-
1994, “IEEE Standard for Software Safety
Plans”.



Require compliance with IEEE Std. 1012-1998,
“IEEE Standard for Software Verification and
Validation”.
Add requirement that the system conform to a
“Required Reliability Rating” of “Very high
“ (assuming the absence of a paper audit
trail ) as defined in section 4.33.2 of a
IEEE Std 982.1™-1988, “IEEE Standard
Dictionary of Measures to Produce Reliable
Software”.
Add requirement that the system conform to
IEEE Std 829™-1998, “IEEE Standard for
Software Test Documentation”.
Eliminate the exemption of COTS software
from the testing requirement.



Eliminate the exemption.




Require compliance with IEEE Std 1028™-1997,
“IEEE Standard for Software Reviews”.
Source code should be reviewed to ascertain the existence
or availability of malicious code, trap doors, Easter eggs, or
other program features that could be used to compromise or
exploit the system and/or the ballot data.



Cut the replacement rule, figures, flow
charts, etc.


Combine into section 5.6.2.4; eliminate this
section.

Delete. Not necessary for persons with
proramming knowledge. Programs and
flowcharts may look different, but have the
same results.




Either define “obvious” in an objectively
verifiable manner, or eliminate the phrase.
Make this a table.

Combine into section 5.6.2.6; eliminate this
section.
These old languages limits programming
ability. Newer languages are more
efficient,effective, and give greater
performance.
Structed programminig is required. No GO
statements allowed. Program should make
calls and return to main module.

Delete this item.
Modify the sentence to state " The vendor
shall justify to the satisfaction of the
testing authority any module lengths…."

After “mixed mode operations” add “,
including mixing of signed and unsigned data
in C or C++ or other languages where this is
tantamount to mixing modes”
Convert the language-independent items that
are testable into “shall” statements and
move the remaining items to an addendum. I
suggest an addendum that contains guidelines
for using the C Programming Language since,
assembly language aside, that is the least
fool-proof language in common use.

Say that assertion violations should be
added to the audit log. There should be
limits placed on the number of times a given
assertion is logged to avoid exceeding
system resources, and information should not
be logged if it can compromise
confidentiality.
New para.: All functional testing shall be
performed with executable code that has
either been compiled from the reviewed source
code or has been verified to match this
compiled code. New para.: Code that is
modified as a result of testing, or for any
other reason, shall be resubmitted for
verification and qualification testing.]
Where any module affects the operation of another (such as
a module that checks data limits from another module), there
must be in-context examination.




Delete. Does not state who is doing the
system build. Vendor or Tester?
Say that the ballot must be scanned so that data from
multiple pages or both sides of the ballot are kept together in
the cast vote record.
This section should reference section 6.1.3
and the wording in section 6.1.3 should be
merged between the 2 sections.
This section should reference section 6.1.4
and the wording in section 6.1.4 should be
merged between the 2 sections.
     Proposed Resolution
(by the Chair on each comment submitted)
NC - This is covered in section 5.1.1 as shown by the
following excerpt: "COTS products require updates due to a
detected security breach or vulnerability. The voting system
vendor must provide a method to assess the impact of
COTS updates on the votin

Mercuri - COTS products themselves should be subject to
thorough evaluation, not just their updates. COTS provide a
significant security risk. This must be addressed by the
C - Section added as follows: k) Ensure that extracted or
duplicated nformation is identical to that on the original Cast
Vote Record storage medium.

Mercuri - 5.6.10 missingreportpdfreport is already covered by
NC - The individual unit from is document???
item C).

Mercuri - 5.6.10 missing from pdf document?
NC - While there is agreement that including language for
the points identified would be an improvement, specific
language changes that would be directly usable have not
been suggested.

Mercuri - Corrections should be as follows: a) ...If-Then-Else,
While, Do-While, ... f) Concurrent process flow should be
allowed only if detailed description of need has been
documented. g) Assignments within branch tests should be
C - suggested r) is included in l) tabulation of overvotes and
undervotes. The following section has been added: r)
Casting of a totally blank ballot (if permitted)

Mercuri - Need to add: The TDP must reflect the
implementation of these voting variations in such fashion that
it is readily discernable that a) all votes are registered to the
appropriately selected candidate(s) and b) tallies reflect the
C - Wording changed as follows: f) Allow the voter to vote a
new ballot in the event of a detected exception or, if the
equipment and state law permit, allow the voter to submit the
ballot ‘as is’ without correction appropriately tabulating all
validly v

Mercuri - Change was not reflected in draft v5.0
NC - This is the testing section and it is not appropriate to
include standards requirements here. Although there is no
degree of testing or source code inspection can 100%
assure perfect system operation, testing and inspection
should be as complete as

Mercuri - Since software testing in inherently insufficient,
there should be a comment here regarding the flawed nature
of source code review. The first sentence should be included
NC - Only unmodified COTS is exempted. This is a drastic
change that permeates the spec and cannot be considered
at this time. Also, if required, a vendor cannot control COTS
source availability which would also limit vendor choices in
system design.

Mercuri - Unmodified COTS is not exempt from serious
security flaws, as evidenced in the continual update patches
that must be downloaded for Microsoft operating systems, for
example. The exemption for COTS products was erroneous
NC - Review for malicious code is a standards requirement
issue and not a definition in the testing area. It should be
covered in section 5.6.

Mercuri - Source code review is the appropriate place to
check for malicious code. This must be covered here as well
as in Section 5.6 (which refers to section 6.6 throughout).
NC - The suggested wording seems to refer to low level
modules within a functional modules/subsystem. This
section is discussing new or changed functional modules
interaction with the other system functional modules or
subsystem components.

Mercuri - The comment pertains to any new or changed
functional modules that interact with other modules or
components, and still needs to be addressed in terms of
                Reference Information                                 Reference Information




C- The comment contains a valid objection. The wording          How is code generated by commercial
has been changed as follows: i. Excluding code generated        code generators identified?? The
by commercial code generators, is written in small and easily   remainder of the correction is
identifiable modules that are constructed to be grouped         acceptable, but the commercial code
according to functional                                         generation clause is questionable and its
                                                                exemption should be removed.
       IEEE P1583
LOT COMMENT SUBMISSION FORM

               Date: 9-30-03 P1583 Draft 5.0 August 2003
                     Document:
                Commenter Clause/ Subclause        Paragraph        Type of
                and Number                         Figure/ Table   comment
                                                                   (General/
       #                                                           Technical/
                                                                    Editorial)
           1
               Sklein-054             5       All                      T




           2   RGH 032              5.2.1.1   item a                   G



           3   MercuriD50 - 047     5.2.1.1   c.                    General
               (formerly mercuri-
               112)
           4   MercuriD50 - 072     5.2.1.2   paragraph             General
               (formerly mercuri-
               166)




           5 Corry-034              5.2.1.2   Add sentence             T



           6   Jhulshof-004         5.2.1.2                            T
           7   vcw-09               5.2.1.2                            T
 8   PPLX-016             5.2.1.2         5.2.1.2 DRE              T and E
                                          System
                                          Standards




 9   selker-1005              5.2.1.2.3   #1                          t




10   selker-1006              5.2.1.2.5   b                           t

11   selker-1007              5.2.1.2.5   c                           t

12   selker-1008              5.2.1.2.5   d                           t

13   RGH 033                   5.2.2      par. 12                    G/T



14   MercuriD50 - 048          5.2.2      first paragraph          General
     (formerly mercuri-
     114)


15   MercuriD50 - 049          5.2.2      next to last             General
     (formerly mercuri-                   paragraph
     115)


16 Corry-035                   5.2.2      b.2                         T



17   Dill-25                   5.2.2      Last paragraph on page     T
18   Dill-45                      5.2.2                               T



19   PPLX-017             5.2.2            5.2.2 Accuracy             T
                                           Requirements




20   Alice - 017               5.2.2.2                                G



21   Aragon - 11               5.2.2.2        First sentence          T




22   RGH 034                   5.2.2.2                                T


23   RGH 035                               First setence of each.
                           5.2.2.2 & 5.2.2.3                          T

24   MercuriD50 - 050          5.2.2.3     paragraph                General
     (formerly mercuri-
     116)
25   PPLX-018             5.2.3.1          5.2.3.1 Common             E
                                           Standards
26   Dill-26                  5.2.3.1   item a               T


27   PPLX-019             5.2.3.2




                                        5.2.3.2 DRE
                                        Systems
                                        Standards            G
28   MercuriD50 - 073         5.2.3.2   b.                 General
     (formerly mercuri-
     167)



29   Aragon - 08              5.2.3.2        new item d.     T




30   Dill-27                  5.2.3.2   last sentence        T




31   vcw-10                   5.2.3.2                        T




32   MercuriD50 - 051         5.2.3.3   a.                 General
     (formerly mercuri-
     117)
33   PPLX-020           5.2.5           5.2.5
                                        Reliability




                                                              E
34 MercuriD50 - 052             5.2.5   Last sentence       General
   (formerly mercuri-
   119)
35 Lipsio-31                    5.2.5   Last line       T
36 schneidewind - 003      5.2.5      Pg. 43                   T




37
     Sklein-046            5.2.5      Second paragraph         T




38
     Sklein-047            5.2.5      First paragraph          T




39 Corry-036               5.2.6      Last sentence            T




40   Lipsio-32            5.2.6.2                        T



41   Lipsio-33            5.2.6.2                        T



42   MercuriD50 - 053      5.2.7      section                General
     (formerly mercuri-
     122)


43   RGH 036              5.2.7.b.3                            T
44   MercuriD50 - 015      6.2      main section       General
     (formerly mercuri-
     039)



45   MercuriD50 - 016     6.2.1     Bulleted list      General
     (formerly mercuri-
     040)




46   MercuriD50 - 017     6.2.1     End of section     General
     (formerly mercuri-
     041)




47 Corry-087              6.2.1.1                         T
                                    4th para., 2nd sentence




48 schneidewind - 004     6.2.1.1   Pg. 86               T


49 Corry-088              6.2.2                          E


50 Corry-089              6.2.2.1                         T
                                    2nd para., 2nd sentence


51   MercuriD50 - 018     6.2.3     list               General
     (formerly mercuri-
     042)




52   RGH 089              6.2.4               c.         G
                         Comment



There needs to be a statement that the
system shall not automatically shut down or
time out during any official part of a
voting-related procedure, such as filling
out a ballot by a voter, except in
accordance with the requirements of voting
law and procedure. This is needed to make
clear that features such as implied by
O.ABORT_SESSION in the current draft of
C.4.1 are prohibited except as specifically
provided by law.
Does this really need to be enumerated?
Presumably, this precludes any form of
encoding the data, including compression and
encryption.
Report of votes cast must not be in proprietary format.



Computerized ballot casting systems that provide voter
verified ballots should not be required to necessarily maintain
an electronic copy or tally of the results.




While a voter-verified paper copy may be
optional, if a paper copy is created and
verified by the voter then it must be the
official or correct copy.
CVR can be stored in memory modules
"As an additional means of ensuring accuracy in DRE
systems, voting devices shall record and retain
redundant copies of the original CVR. This is a
requirement whether or not a paper copy of voter
selections is printed for voter verification." specifies
only one purpose for creating a paper copy of the CVR.
The wording of this section relates to our
earlier comment about the change of
definition of a DRE. We suggest the
following change:

As an additional means of ensuring accuracy
in systems in which votes are counted on the
voting device itself, voting devices shall
record and retain redundant copies of the
original CVR. This is a requirement whether
or not a paper copy of voter selections is
printed for voter verification. (A CVR is an
electronic record of all votes cast by the
voter, including undervotes.) There shall be
a pre-defined hierarchy to determine which
copy is to be deemed the "official" or
"correct" one, in the case redundant copies
do not agree in data content (due to
corruption or loss of information).


new assumtion




we do not know that voters can reliably
verify
we do not know that voters can reliably
verify
we do not know that voters can reliably
verify
Why two different rates? Only the lower rate
is important. This is equivalent to
requiring "a 10-digit calculator on which
the first 6 digits are correct."
Since even a single misplaced vote could determine the
outcome of the election, the last sentence must be changed.




There must be a way to differentiate between deliberate
undervotes and votes lost by the system.




No mention of optional paper copies with DRE
systems.


Are these accuracy requirements realistic for optical scan
systems? I've heard of error rates on the order of 1%. I
don't understand what it means to "achieve a target rate".
Accuracy requirements seem to apply to touch screens.
Testing 1/10,000,000 on a touch screen would seem to be
very difficult (and it is unlikely that that level of accuracy can
be achieved). marks are placed by, and later
when those
read by a computer, may be located anywhere
on the paper ballot. The text as drafted
could be improved without any loss of
generality and accuracy by dropping the
words “ballot position”.


How would this be know? Is there any
methodology to determine the "ERROR FREE"
requirement? If anyone simply states it's
"ERROR FREE" is that sufficient.
Rework of 4.3 comments (Corry and aragon-014). This
section in 4.3 referred to device-level redundancy as a
means of achieving the functional goal of stability. Corry
comment was to apply functional requirement to all voting
equipment, not only DRE. Aragon comment was to clarify
that functional requirement did not depend on design choice
of redundancy (requirement applies to module as a whole
regardless of internals). Resolution was to introduce a
design requirement, for DRE only (opposite of Corry),
specifying a particular type of redundancy, and then
reversing application of aragon-014 to the changed sense of
redundancy (data record level vs. device level). Thus it has
lost ground from 4.3 (relative to the comments made against
4.3), and has also lost the reference to redundancy at the
device level which appears to have been the original
"demonstrated error free data retention for
22 mos" - are you going to have 22+ month
certifications
Willl this require a 22 month certification
process?
Magnetic media can be altered.



The proposed draft maintains cautions about
electromagnetic radiation. Many touch
screens are made for commercial environments
and therefore meet FCC Part 15 Class A
requirements. It is generally not
admissible to allow TV sets or radio
broadcasts into a polling place. Therefore,
Class A requirements should be admissible.
This particular section does not rule out
Class A, but the electromagnetic radiation
language seems to set the stage for the more
expensive and less inclusive Class B
requirement.
What does it mean to "protect against a single point of
failure." Are you proposing redundant power supplies?
Redundant touch screens?!
The draft states:
In addition to the common standards, DRE
systems shall:
a. Maintain a record of each ballot cast as
a CVR using a process and storage location
that differs
from the main vote detection,
interpretation, processing, and reporting
path;

The problem in this section is similar to
the problem in 5.2.1.2.
Should allow voter verification to be applied here (This item is
similar but not the same as the one that may appear in David
Aragon's comment set.)




Retention of CVR: in 5,2,2,3 we required
that the medium can hold the vote for some
amount of time, but if it's a rewriteable
medium, it could change anyway. This
concern was brought to me regarding flash
memories, but is also applicable to
writeable media generally, including paper.
Clarification: Suppose a touch screen machine prints an
optical scan ballot that the voter then feeds into the optical
scanner in the usual way? Is it a DRE, or a paper based
system? I hope it is not required to keep redundant
electronic records of the votes.
        "These are requirements whether or not a paper
        copy of voter selections is printed for voter
        verification." limits the purpose of a paper CRV
        to voter verification.
The event prior to the error or failure may have caused it.
We question the requirement of an MTBF of
163 hours for which a failure is any outage
is a “Degradation of performance such that
the device is unable to perform its intended
function for longer than 10 seconds”. Such
a requirement runs counter to some modern
software philosophies that point out that
recovery is a viable strategy to ensure
overall system readiness. We call the
Working Committee’s attention to the work of
Professors Armando Fox and David Patterson.
In a recently published article in
Scientific American (Scientific American,
Volume 288 Number 6; June 2003 pp 54-61),
Professors Fox and Patterson write: “Rather
than trying to eliminate computer crashes –
probably an impossible task – our team
concentrates on designing systems that
recover rapidly when mishaps do occur.”    We
suggest looking at the problem not from a
MTBF point of view, but from a system
availability point of view as section
5.2.6.2 does. Outages can be caused by many
factors outside of the system designer’s
control. This includes alpha particles
found within the plastic packages on any
semi-conductor to stray electromagnetic
fields that penetrate reasonable shielding.

Looking at the effect on a voter, 10 seconds
of outage of 163 hours represents 10 seconds
Early voting districts may require longer than 163 hours of
MTBF.
An MTBF of 163 hours is ludicrously sloppy,
suggesting that in a jurisdiction with
50,000 machines where the polls are open for
12 hours, there will be more than one
failure per second.
1. An average like MTBF is not a good
measure of reliability for a mission
critical system. A specification like time
to next failure would be much more
meaningful to the voter!

2. If MTBF is to be used, there should be an
explanation of how 163 hours was determined.
suggested_remedy = 1. Specify that there
shall be no non-recoverable failure that
impairs the accuracy of voting during the
time the voting equipment is in operating at
a polling place on a given election day.
Recovery from failures shall include a paper
trail of votes cast.

3. Explain the basis of 163 hours.


The 163 hour MTBF specified implies a 9.2%
probability of failure per machine during an
election. This is unacceptably high.


The scenario omits the storage portion of
the system usage cycle and fails to specify
the equivalent of a maximum failure rate
during storage.


"mean" is undefined in sentence.




“Shall achieve at least nine-nine percent
availability” seems rather permissive and
would erode public confidence in the device
if not exceeded.
“Recommended …spare … components” suggests
that the machines be repairable and,
therefore, modifiable during an election.

No where does the standard reflect the accuracy and
reliability of the scanners used at the precincts to alert voters
using mark-sense ballots to their correctness in preparation.


consolidated vote data should be referenced
only for a given equipment type, not all
types in the precinct
Accuracy is not the same as integrity. The omission of any
material in this section on Data Integrity is a serious detail
that needs to be added.




Add additional material to clarify testing procedures




It is important to clarify the fact that automated testing is not
the same as user testing.




Needs reference for SPRT and is this Wald's
SPRT?




The parameters "a" and "b" have not been
defined.

Talking about Mean Time Between Failures
(MTBF) in this paragraph. Don't know what
MBTF is?
Lacks reference for probability calculated
using an exponential distribution and
sentence is incorrectly worded.
Alert must be provided for unavailability.




Vague
                   Proposed Change



Insert a statement in an appropriate place
that “The system shall not automatically
shut down or time out during any official
part of a voting-related procedure, such as
filling out a ballot by a voter, except in
accordance with the requirements of voting
law and procedure.”




Only state the obvious if it is non-obvious.



…be able to produce a vendor-independent, non-proprietary,
human-readable report of all votes cast.


DRE systems that provide voter verified ballots may also
maintain an electronic copy or tally of results. Should there
be any differences noted, the voter verified ballot set should
be considered to supercede any electronic copy of the cast
ballots.

If a voter-verified paper copy is made it
shall be deemed the "official" or "correct"
copy.

voting devices and or memory modules
Replace with "As an additional means of ensuring
accuracy in DRE systems, voting devices shall record
and retain redundant copies of the original CVR. This
is a requirement whether or not a paper copy of voter
selections is printed."
The annonymity and security of new machines
must be as good as the security in the
processes they are replacing: ie people
overseeing each others handling of paper
ballots
eliminate: for voter veriraication

eliminate: for voter veriraication

eliminate: for voter veriraication

If this is a matter of statistical analysis
of the testing results, it should be stated
as such.

There must be a zero-error rate, so as to not to affect the
outcome of any election. If a zero-error rate is not able to be
achieved, jurisdictions must be informed as to the actual
error rate, such that if an election falls within this range
between candidates, a run-off election must be required.
A mechanism for differentiating between deliberate
undervotes and votes lost by the system must be employed,
such that the error rate can be ascertained without doubt.


2) Independently from voting data storage,
recording voter selections of candidates and
contests into CVR storage [and on to
optional paper copies if so equipped.]
Explicitly address accuracy requirements for
human input devices.


The proposed new text:

a. For paper-based systems:
1) Scanning paper ballots to detect
selections for individual candidates and
contests;
2) Conversion of selections detected on
paper ballots into digital data;
Need some way to determine the DRE memeory
is error free. Unless DRE memory is in test
mode for 22 months prior to determine this
factor.
Replace with: "System memory devices used to retain control
programs and data shall have demonstrated error-free data
retention for a minimum period of two years. This
requirement shall apply to any memory module addressable
as a single memory device, even if it internally uses
redundant storage elements. This requirement on memory
devices is separate from the requirement of 5.2.1.2 and
5.2.3.2 (c) that specific types of data be stored in multiple
memory locations; each such location would be subject to
the requirement of this section."




replace with language saying the ITA deems
the memory capable of retaining data for 22
mos
Providing spec. data from the manf. proving
this capability would constitue compliance.
The use of magnetic media, or media that can be readily
altered should be avoided, or additional mechanisms and
controls put in place to ensure that election and ballot data is
not changed or destroyed.
Delete item



We suggest the following change:
In addition to the common standards, DRE
systems that record votes on the voting
station shall:
a. Maintain a record of each ballot cast as
a CVR using a process and storage location
that differs
from the main vote detection,
interpretation, processing, and reporting
path;

Add at end: "This requirement applies to retrieval of Cast
Vote Records even if the system also provides voter verified
ballots, although in this case, if there is any difference in
results, the voter verified ballot set shall supersede any tallies
generated from the electronically recorded data."

Add:
"d. Provide output of CVRs, audit data
records, VVARs and related election
information in a manner that allows
detection of any subsequent alteration or
degradation of their data, even if stored on
a removable medium."




        Replace with "These are requirements whether
        or not a paper copy of voter selections is
        printed."
...existing immediately prior to (and not the cause of) the
error or failure….
Allow one machine per polling place to
reboot once during an election. Assume a
reboot takes 2 minutes.
This number should be corrected as per the discussion in the
July meeting.
Discuss in the working meeting; suggest 109
or 1010seconds.
Increase the required MTBF to 1500 hours,
implying a 1% probability of failure during
an election and a 2% probability of failure
during equipment setup and readiness
testing.
Incorporate an average 1 year storage time
into the scenario and add a sentence after
the MTBF requirement sentence specifying a
probability of operation of 99% after one
year of storage (equivalent to a 1% per year
failure rate in storage).
Although a more quantitative basis for
assessing maintainability, such as the mean
[time] to repair the system is desirable,
the qualification of a system is conducted
before it is approved for sale and thus
before a broader base of maintenance
experience can be obtained.
Discuss in the working meeting.



Prohibit modification, including for repair,
of the device from the time the ballot is
configured through the retrieval of election
data.
Need to add wording to reflect the comment here in the
appropriate place.




add wording "for a specific equipment type"
Confer with group on addition of material on Data Integrity.




* The testing should involve a randomized data set, not just
the same ballot position or subset of ballot positions. *
Testing should confirm that there is nothing in the setup,
shutdown, or restart procedures that can cause a loss or
alteration of data.



This section specifies testing and error rates that may be
discovered using automated procedures. Error rates that
result from actual user testing can be considerably higher.
Threshholds for user error rates must also be established on
the basis of accepted industry usability standards.




[Wald's (?)] Probability Ratio Sequential
Test [see ???] using [a] binomial
distribution is recommended. In the case of
ballot position error rate, the calculation
for a specific device (and the processing
function that relies on that device) is
based on:
Define the parameters.


Use MTBF two places in this paragraph.


The probability ratio for this test is
derived from [an] exponential [ ]
distribution [see ???].
iii. Appropriate feedback and alert must be provided to voter
and election officials in the event of error that affects
availability or usability of voting device.




Needs better definition.
                  Proposed Resolution
            (by the Chair on each comment submitted)




NC goes to design

Mercuri - Goes to auditability, reliability and recountability
which are all performance.
NC - The responsibility for determining which record
superceeds the other is for EAC/States and case law to
decide.

Mercuri - The resolution indicates a lack of understanding of
the concept of voter verified balloting and needs to be
resolved by the working group.
NC 0 Not achievable by any known means. Error rates a

Mercuri - Since this is a performance standard, the error rate
must be disclosed so that jurisdictions can know when an
election must be recalled on the basis of insufficient data to
NC Requires winner.
determine thevoter to declare intent to undervote and that is a
voting jurisdiction issue not equipment issue.

Mercuri - The equipment must be able to ascertain the
difference between a deliberate undervote and a system
error that caused vote lossage.
NC goes to design

Mercuri - Unalterability of media for critical data should be a
performance requirement.
NC - The responsibility for determining which record
superceeds the other is for EAC/States and case law to
decide.

Mercuri - The resolution indicates a lack of understanding of
the concept of voter verified balloting and needs to be
resolved by the working group.




C added text"event causing the "

Mercuri - Comment was misunderstood, resolution is
incorrect.
???????????? Someone was researching that number.
Need to insert correct number.
NC wording not provided

Mercuri - 5.2.7 a. 3) Precinct based scanners should be
available to produce reliable and accurate feedback to voters
using mark-sense ballots as to their correctness in
preparation (including alerts to undervotes and overvotes).
C- Revised wording: Industry recognized third party class
libraries (e.g., Microsoft Foundational Class (MFC) C++
libraries)

Mercuri - Resolution makes no sense. This comment needs
to be addressed. There is still NO material in this section on
Data Integrity and this is a serious omission. on ballot
C- Revised wording: Software that operates
printers, vote counting and verification devices, memory
devices used for results storage and/or ballot activation and
other hardware typically installed at precinct, early voting or in-
person absentee

Mercuri - Resolution makes no sense with respect to
comment. Comment needs to be addressed to reflect
NC - The error rates in this section do not include human
error where votes are inadvertently not cast as intended. It
refers to machine introduced errors. It also does not assume
an automated process. For paper tabulaors, real ballots are
read.

Mercuri - Resolution is incorrect. For paper tabulators, a
predefined set of ballots are typically read. For DRE-style
machines, it is my understanding that testing is typically NOT
performed on the ballot casting device, only the ballot




NC - The suggested wording relates to the functional
requirement of availability (or failure) indication not to the
section subject, meeting availability standards. The concern
is already covered in the statement of 5.6.9.1.3d "d) A means
of identifying

Mercuri - Resolution makes no sense. There is no section
5.6.9.1.3d in the V5.0 draft. 5.6.8.3.1d pertains only to the
"failure of the main power supply" and not to the actual
unavailability of the system due to other equipment
Reference Information   Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

               Date: 9-30-03 P1583 Draft 5.0 August 2003
                     Document:
                Commenter Clause/ Subclause        Paragraph             Type of
                and Number                              Figure/ Table   comment
                                                                        (General/
       #                                                                Technical/
                                                                         Editorial)
           1 Corry-052                       5.4   First sentence           E




           2   RGH 072                                                   T
                                         Second paragraph above clause 5.4.1.
                                          5.4




           3 Corry-053                   5.4.1     Entire section           T




           4   PPLX-031             5.4.10         5.4.10 Safety




                                                                            G
           5 Corry-054                   5.4.4     a.                       E


           6 Corry-055                   5.4.4     Should be c.             E


           7   MercuriD50 - 063          5.4.4     c.                    General
               (formerly mercuri-
               141)
 8   Jhulshof-026           5.4.4   b                  T




 9   Jhulshof-027           5.4.4   last sentence      T


10 Corry-056                5.4.5                      E


11   PPLX-028       5.4.5




                                    5.4.5
                                    Environmental
                                    Control -
                                    Operating
                                    Environment        G
12   RGH 073                5.4.5    Humidity range    T




13   RGH 074                5.4.5       Temp. ranges   T




14   PPLX-029       5.4.6




                                    5.4.6
                                    Environmental
                                    Control -
                                    Transit and
                                    Storage            G
15 Corry-057                5.4.6   First sentence     T
16 Corry-058                      5.4.6                        T




17   PPLX-030             5.4.8


                                          5.4.8 Product
                                          Marking              G
18 Corry-059                      5.4.9   b)                   T




19 Corry-135                      6.4     Last sentence        E




20   RGH 108                      6.4.1    First paragraph     G
21   RGH 109                      6.4.1   Second paragraph     T



22 Corry-136                      6.4.3   First sentence       E




23   MercuriD50 - 020             6.4.3   End of section     General
     (formerly mercuri-
     045)




24 Corry-137                      6.4.3   Last three paragraphs T
25 Corry-138            6.4.4.1                           T
                                    1st para., 1st sentence




26 Corry-139            6.4.4.1                           T
                                    2nd para., 1st sentence




27   Lipsio-4B          6.4.4.1     Para. 2         T




28 New - 001            6.4.4.1                          T




29 schneidewind - 006   6.4.4.1     Pg. 100              T


30 Corry-140            6.4.4.1.2   Last sentence        E




31 Corry-141            6.4.4.1.6                        T




32 Corry-142            6.4.4.3.2   Step 5               E
33 Corry-143   6.4.4.4.2                            E



34 Corry-144   6.4.4.5.2                            E

35 Corry-145   6.4.4.5.2      Step 3                E


36   RGH 110   6.4.5.1     Steps 4, 9, 10 and 11.   T
                  Comment



Facilities must be secure.




COTS equipment will be entrusted with
counting votes but is exempted from this
standard with a "proven record of
performance"? OEMs of voting eqipment also
have "proven" track records but must still
test to this standard? This seems
unreasonable.
Security of storage areas and containers not
addressed.




All voting systems shall meet the
requirements for safety of IEC 60950-1.

In cases where an entire third party
specification is mentioned, there is a
strong risk that the specifications conflict
with each other. It is best to state
directly what the requirements are.

Incorrect use of abbreviations.


Third item not correctly labeled. Also add
that there is no need for backup power to
light the voting system either.
Length of time to retain contents of memory.
 Insert the drawing from fec 2002 where the
wheelchair voter approaches the voting
station paralllel


 Insert the drawing from fec 2002 where the
wheelchair voter approaches the voting
station paralllel
Typographical error on operating
temperature. 441° to 104° Fahrenheit should
be 44° to 104° Fahrenheit

The proposed language is:
“All voting equipment shall be capable of
operation in temperatures ranging from +5°C
to +40°C degrees
(441° [sic] to 104° Fahrenheit) and relative
humidity from 5% to 85%, non-condensing.

This is a change from the VSS Operating
Environment. Many vendors have designed
their systems to meet VSS rules. This
change places an undue burden on those
vendors. In any event, 441° is a typo.

Same situation as with the temp range only
that in the old standard no humidity range
was called out. Beyond that the range is set
at 5-85% RH non-condensing. One our our CF
manf.'s makes some of the most rugged
industrial CF's on the market and they only
spec a range of 8-85% RH non-condensing.
This seems a little unreasonable.
Temp range went from 50F-95F to 41F-104F. We
already meet 2002 standards, are we to be
required to re-test all environmentals but
the temp range increased by eighteen degrees
total? This hardly seems reasonable for such
a small change.
The proposed language quotes MIL STD 810D.
We have heard, but do not vouch for the
accuracy, that these MIL Standards have
either been superseded or are difficult to
acquire. Since these standards affect both
design and certification testing, they
should be directly quoted here.
Incorporating a requirement by reference
creates risk that the incorporated
requirement conflicts with the specifically
enumerated requirement.
Calls for testing physical shock but no
standards for shock are specified.
Except for a footnote about electrostatic
discharge there are no standards about
altitude or air shipping requirements, e.g.,
must the equipment be air shipped in a
pressurized aircraft, must the shipping
container be vented or not, what altitude
must the equipment be certified to work at
(a lot of electronic equipment starts having
problems above 3 km or 10,000 ft above sea
level, a common elevation in Colorado)

This requirement is at odds with FEC 2002
VSS Section 2.3.1.3.1 Common Standards. As
we understand it, that section states that
no logo be allowed on any product in the
polling place.
Nothing about workmanship that introduces
hazards when equipment is in use.



Incomplete sentence.




This is very vague.
Current (2002) standard call out a lab test
range of 68F-75F but this document makes no
clear definition of what "standard',
"ambient" and "nominal" are or should be.
See comment 370 from v. 4.3. Wording was
changed at my request but I think the word
"encouraged" goes a bit too far. Recommend
the use of test fixtures be "permitted"
rather than "encouraged."
Pristine data sets do not reflect real voting situations. Add
paragraph at end.




See comment 371 from v. 4.3. The revision of
the first paragraph help but I think it
makes the last three paragraphs of this
section even more superfluous and an attempt
at micromanagement. My recommendation, as
before, is to delete these three paragraphs.
Sentence includes physical shock tests but
there are no specifications for shock except
a bench test. See comment 254 on v. 4.3 and
response. Issue is still unresolved. It is
still my recommendation that shock and drop
standards be incorporated. These are common
causes of failure and we can certainly
expect voting equipment to receive rough
handling.
Systems that are simply cobbled together
(kluge might be a better description) from
COTS components must not be exempted from
environmental testing. I've had too many
problems with little doohickies hung on some
piece of otherwise great equipment that
caused problems when fielded.
COTS hardware must have been tested to the
rigor required of non-COTS components; if
the supplier has not done this, then COTS
hardware must be treated like any other
component.


The mere idea of a voting system running on
an inherently vulnerable operating system
such as Windows that need not be evaluated
because it is already is certified at some
EAL-x level is absurd. We are also about to
have ATMs running on Windows software, which
will open up huge cans of worms. I am
STRONGLY OPPOSED to multipurpose systems.
Long ago someone proposed using ATMs on
election day, because people are familiar
with them. This is a HORRENDOUS IDEA. Of
course, I am also strongly opposed to voting
systems with essentially ZERO accountability
that your vote is correctly recorded and
counted, as is the case with the existing
certified DREs.
Why exempt COTS hardware from environmental
testing?

Make the last sentence a separate paragraph.




How many times can the system be retested
before the system is deemed to have failed
the environmental tests? I would suggest a
total of three failures during the complete
process with two consecutive failures
sufficient to reject the voting system.
Reference is made to Figures 514.3-2 adn
514.3-3 but couldn't find them.
Use metric units consistently. Degrees
Celsius not Fahrenheit. If English units are
used they should follow SI units in
parentheses.
Units problem again. Degrees C rather than
degrees F.
Stabilization required for minimum of 4
hours, not an exact period.

See comment for #2.
                   Proposed Change



The environmental requirements for voting
systems include [secure] shelter, space,
furnishings and fixtures, supplied energy,
environmental control, and external
telecommunications services.
Either require COTS equipment to comply to
the same standards as all other voting
equipment or remove the paragraph
altogether.



All precinct count systems shall be designed
for storage and operation in [an] enclosed
[secure] facility ordinarily used as a
[secure, locked] warehouse or polling place,
with prominent instructions as to any
special storage [or security] requirements.
[For example, voting systems kept in
unlocked storage containers will require a
higher level of secure storage facility than
those voting systems with a securely locked
storage container.]




Describe the requirements directly.
a. Systems shall operate with the electrical
supply ordinarily found in polling places
(120[VAC/60Hz]);
[c.] The backup power capability is not
required to provide lighting of the voting
area [or voting system].
…should retain the contents of all memories intact, until the
end of data collection or recovery efforts.
systems shall be capable of operating on
back up power, such that no voting data is
lost or corrupted. When power is inturrupted
or cuized the system shall retain…..

last sentenc should read: The back up power
is required to provide lighting of the
voting area
Change to read (44° to 104° Fahrenheit)




Fix typo and use FEC 2002 VSS requirements.
Maintain what the current standard calls out
or grandfather equipment that already meets
2002 standards.




Leave standard at 50F-95F or grandfatehr in
equipment that already meets 2002 standards.




Describe the requirements directly.
See comment 254 and notes in previous review
about adding shock and vibration standards.
Add shock and vibration standards.
Add standards for operation at elevation and
for non-operating air shipment.




Resolve difference, if any, with FEC 2002
VSS logo requirements
b) Ensure that components provided by
external suppliers are free from damage or
defect that could make them unsatisfactory
[or hazardous when used] for their intended
purpose.
The testing authority shall review the
modification(s) to determine what, if any,
tests must be run to confirm a unit’s
continued compliance. If an engineering
evaluation of the change(s) is (are) not
clear whether a retest is required, the test
shall be performed [again].
Needs clearer definition.
Clarification.



The use of test fixtures or ancillary devices to facilitate
volume hardware qualification testing is [permitted].


There must also be an additional acceptance metric based
on real voter data input. This could be related to a usability
metric, or created by asking humans to replicate a set of
ballot choices. Results must then be compared to the
equipment fault rate in order to determine acceptance
threshholds.


Delete last three paragraphs of 6.4.3.
Add shock and acceleration tests here and to
section 5.4.6.




Delete first sentence of second paragraph.




Change paragraph to “COTS systems or
components must be documented by their
suppliers to have been tested to at least
the same rigor as required of voting devices
as specified hereinbelow; else, the said
COTS components shall be tested in a like
manner to any other component.”




Require environmental testing of COTS
hardware.

New para.: When preparation for storage is
required, the equipment shall be prepared
using any protective enclosures or internal
restraints that the vendor specifies for
storage.
Add new paragraph: During the environmental
testing sequence the system cannot fail more
than three times for any combination of
reasons. If the system fails any test twice
in a row it shall be rejected.

Add figures 514.3-2 and 514.3-3 and renumber
them consistent with text of these
standards.
Use degree C rather than degrees F. Use en
character for minus sign for negative
numbers rather than hyphen.

Use SI units consistently.

Allow the chamber temperature to stabilize.
Maintain this temperature for a [minimum] of
4 hours after stabilization.
See proposed change for #2.
                 Proposed Resolution
           (by the Chair on each comment submitted)




NC - Suggested wording too vague. I agree some time
specification should be added to cover units that use battery
backup RAM for non-volatile memory. However, need to
define a concrete time period. (24 hours?, 2 weeks?)

Mercuri - Use time similar to other sections -- 24 months?
C - Agreed that real human data entry is required to
supplement automated testing. The first paragraph has been
changed as follows: The use of test fixtures or ancillary
devices to facilitate volume hardware qualification testing is
encouraged. These

Mercuri - Change is incorrect -- should read ...."human
interaction will uncover anomalies"... not will NOT uncover
Reference Information   Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

                Date: 9-30-03 P1583 Draft 5.0 August 2003
                      Document:
                 Commenter Clause/ Subclause        Paragraph     Type of
                and Number                       Figure/ Table   comment
                                                                 (General/
       #                                                         Technical/
                                                                  Editorial)
           1    RGH 075                                             T
                             Last sentence "Electrical power increases of 10%….
                                    5.5.1



           2 Corry-060                5.5.2    a)                    E
           3 Corry-061                5.5.2    b) and c)             T

           4 Corry-062                5.5.3    c), d), and e)        T


           5 Corry-063                5.5.3    c)                    T

           6 Corry-064                5.5.3    d)                    T

           7 Corry-065                5.5.3    e)                    T

           8 Corry-067                5.5.4    Footnote 26           T




           9    RGH 076               5.5.4     Superscript 26       T



           10   RGH 077               5.5.4                          T

           11   PPLX-032      5.5.5




                                               5.5.5
                                               Electromagnetic
                                               Radiation             E
12 brook - 001              5.5.6                     T

13 Corry-070                5.5.7   f) and g)         T

14   Jhulshof-030           5.5.7   Second sentence   T



15   PPLX-033       5.5.8


                                    5.5.8 Magnetic
                                    Fields Immunity   T
                 Comment



This is a change from 7.5% in the old
standard. Again, we will be required to re-
certify equipment that already meets 2002
standards?
Incomplete sentence.
I have no idea what is meant by these two
transient requirements.
Given the distances between lines specified
0.5 kV lightning surges seems an extremely
small requirement.
Incomplete sentence.

Incomplete sentence.

Incomplete sentence.

Footnote is incomplete.




I really question leaving the option to
raise the air discharge to + 25kV to a
jurisdiction, are they really qualified to
make this type of decision?
What about preserving votes but not normal
operation
The draft states: Equipment covered by this
standard shall comply with the Rules and
Regulations of the Federal Communications
Commission, Part 15, Class B or the CISPR
22, Class B requirements for both radiated
and conducted emissions.

We note this is the same requirement quoted
in the FEC 2002 VSS. Most touch screens
that are commercially available meet Class
A, not Class B, requirements. We recommend
relaxing this requirement. We see no
compelling reason to force Class B.
The meaning of these standards escapes me.

100 KHz is wrong acc. IEC 61000-4-6



The draft uses a unit of measure for
magnetic fields that it has not defined.
While the reader can probably understand m
for meter, it is not clear what the unit A
for magnetic field is.
             Proposed Change



Maintain what the current standard calls out
or grandfather equipment that already meets
2002 standards.

a) 2 kV AC & DC [in] external power lines;
Define these two requirements in a clear,
concise fashion.
Check values and specify more realistic
values.

c) +.5 kV DC line to line [at distances]
>10m [between lines];
d) +.5 kV DC line to earth [at distances]
>10 m [from the voting system]; and
e) +1 kV I/O [signal]/control [at distances]
>30 m [from the voting system].
26. Some jurisdictions which often
experience high levels of [electrostatic
discharge] due to typically low humidity or
high altitude may wish to increase the level
to ±25 kV air discharge. Such conditions are
commonly found at elevations above 2 km
(6,500 feet) or in high desert regions.

Leave the standard at + 15 kV and drop the +
25 kV "option".


replace with comments




Allow FCC Part 15 Class A
from "10 GHz" to "2.5 GHZ"

Restate these specifications in clear,
concise terms that even I can understand.
Should be 150 KHZ acc to IEC 61000-4-6.
Note: FEC 2002 in 3.2.2.11 Conducted RF
immunity does not state any freuquecy range.




Define the unit for magnetic fields.
Specify that the measurement shall be RMS.
     Proposed Resolution
(by the Chair on each comment submitted)
Reference Information   Reference Information
       IEEE P1583
LOT COMMENT SUBMISSION FORM

                Date: 9-30-03 P1583 Draft 5.0 August 2003
                      Document:
                 Commenter Clause/ Subclause        Paragraph     Type of
                and Number                       Figure/ Table   comment
                                                                 (General/
       #                                                         Technical/
           1
                                                                  Editorial)
           2
           3
            4
            5
            6
            7
            8
            9
           10
           11
           12
           13
           14
           15
           16
           17
           18
           19
           20
           21
           22
           23
           24
           25
           26
           27
           28
           29
           30
           31
           32
           33
           34
           35
           36
           37
           38
           39
           40
           41
           42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
Comment
Proposed Change
     Proposed Resolution
(by the Chair on each comment submitted)
Reference Information   Reference Information
  IEEE P1583
BALLOT COMMENT SUBMISSION FORM

     Date: 9-30-03 P1583 Draft 5.0 August 2003
           Document:
      Commenter Clause/ Subclause        Paragraph     Type of
     and Number                       Figure/ Table   comment
                                                      (General/
 #                                                    Technical/
                                                       Editorial)
Comment
Proposed Change
     Proposed Resolution
(by the Chair on each comment submitted)
Reference Information   Reference Information
  IEEE P1583
BALLOT COMMENT SUBMISSION FORM

     Date: 9-30-03 P1583 Draft 5.0 August 2003
           Document:
      Commenter Clause/ Subclause        Paragraph     Type of
     and Number                       Figure/ Table   comment
                                                      (General/
 #                                                    Technical/
                                                       Editorial)
Comment
Proposed Change
     Proposed Resolution
(by the Chair on each comment submitted)
Reference Information   Reference Information

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:7
posted:7/31/2012
language:English
pages:1890