Comment
Proposed Change
"source code generated by COTS code Delete this clause development package and embedded in software modules for compilation or interpretation shall be provided in human readable form" Some newer programming tools do not necessary generate traditional source code as reference within this clause.
COTS may be properly installed and configured but still not meet requirements unless latest security patches are installed.
Last three sentences should be separate paragraph.
Notwithstanding the fact that system certifiers can rely upon the prior validations of the individual components of the system [ ] provided they are properly installed and configured [with the latest security patches], there must still be an evaluation of the integrated system to make certain that security holes have not been left or created during the integration process. Start new paragraph: [As] COTS products require updates due to a detected security breach or vulnerability [the] voting system vendor must provide a method to assess the impact of COTS updates on the voting system, as well as a method for providing notice and distribution of updates to purchasers[, testing facilities, and election officials and boards]. Where COTS products are known to be inherently risky ([e.g.,] memory leaks in the C++ language), vendors must adequately describe the control methods they have employed to ensure these risks have been mitigated.
Systems that are simply cobbled together (kluge might be a better description) from COTS components must not be exempted from environmental testing. I've had too many problems with little doohickies hung on some piece of otherwise great equipment that caused problems when fielded.
Delete first sentence of second paragraph.
COTS - "These devices and software are exempted from certain portions of the qualification testing process so long as such products are not modified in any manner for use in the voting system." If COTS hardware or software is in the trusted subset, it must be treated exactly like software or hardware designed by the vendor.
Delete sentence. I do not believe that is appropriate in a reference section defining COTS. Specify that the COTS exclusion only applies to system components outside the trusted subset.
Explanation about exemption is unnecessary, and may become inconsistent if we add change requirements on COTS The treatment of COTS products contradicts section 5.1.2.2, “Elements of Security Outside of Vendor Control”.
Delete last sentence of definition. Change “COTS product may” to “COTS products shall”. Mandate compliance with section 4.3.11 (“Previously developed or purchased software”) of IEEE Std 1228-1994, “IEEE Standard for Software Safety Plans”.
There is implied a lack of testing in “COTS products require updates due to a detected security breach or vulnerability”; nothing that requires an update should pass testing.
Mandate that testing preclude any security breach or vulnerability; mandate compliance with section 4.3.11 (“Previously developed or purchased software”) of IEEE Std 12281994, “IEEE Standard for Software Safety Plans”. Mandate COTS be subject to the specifications of IEEE Std 1008™-1987 (R1993), “IEEE Standard for Software Unit Testing”. Add reference to IEEE Std 982.1™1988, “IEEE Standard Dictionary of Measures to Produce Reliable Software”. “The voting system vendor must provide a Bring into conformance with Annex D (“V&V method to assess the impact of COTS updates of reusable software“) of IEEE Std 1012on the voting system, as well as a method 1998, “IEEE Standard for Software for providing notice and distribution of Verification and Validation”, e.g., updates to purchasers” is inconsistent with “Reusable software (in part or whole) IEEE Std 1012-1998. includes software from software libraries, custom software developed for other applications, legacy software, or commercial-off-the-shelf (COTS) software. The V&V tasks of Table 1 are applied to reusable software just as they are applied to newly developed software. However, the inputs for these tasks may not be available for reusable software, reducing visibility into the software products and processes.“ Memory leaks are the result of using C++ Eliminate “(ex. memory leaks in the C++ language inappropriately; they are not a language)” risk of a COTS C++ compiler. Industry standard COTS compiler and runtime Require all tools, including compilers and interpreter both is not defined and assumes interpreters, to be validated and verified that, contrary to reality, something is in the same manner as application software. fail-safe and fool-proof by virtue of being in common use.
“COTS software is not required to be inspected…” is contrary to such other mission-critical methodologies as those used by the FDA and FAA, and contradicts what is specified in section 5.1.3.3.2
Eliminate the section, or, better yet, reverse its sense.
There is implied a lack of testing in “COTS products require updates due to a detected security breach or vulnerability”; nothing that requires an update should pass testing.
Mandate that testing preclude any security breach or vulnerability; mandate compliance with section 4.3.11 (“Previously developed or purchased software”) of IEEE Std 12281994, “IEEE Standard for Software Safety Plans”. Mandate COTS be subject to the specifications of IEEE Std 1008™-1987 (R1993), “IEEE Standard for Software Unit Testing”. Add reference to IEEE Std 982.1™1988, “IEEE Standard Dictionary of Measures to Produce Reliable Software”.
There is implied a lack of testing in “the most recent version of the COTS product incorporating all security patches” ”; nothing that requires an update should pass testing.
Bring into conformance with Annex D (“V&V of reusable software“) of IEEE Std 10121998, “IEEE Standard for Software Verification and Validation”, e.g., “Reusable software (in part or whole) includes software from software libraries, custom software developed for other applications, legacy software, or commercial-off-the-shelf (COTS) software. The V&V tasks of Table 1 are applied to reusable software just as they are applied to newly developed software. However, the inputs for these tasks may not be available for reusable software, reducing visibility into the software products and processes.“
COTS hardware must have been tested to the rigor required of non-COTS components; if the supplier has not done this, then COTS hardware must be treated like any other component.
Change paragraph to “COTS systems or components must be documented by their suppliers to have been tested to at least the same rigor as required of voting devices as specified hereinbelow; else, the said COTS components shall be tested in a like manner to any other component.”
“Unmodified, general purpose COTS nonvoting software ...is not subject to code examination...is not subject to the full code review and testing” is contrary to such other mission-critical methodologies as those used by the FDA and FAA, and contradicts what is specified in section 5.1.3.3.2. Second sentence is not part of the definition. Whether or not my later comments on COTS are accepted, “These devices and software are exempted from certain portions of the qualification testing process so long as such products are not modified in any manner for use in the voting system” does not belong in the definition. It is unclear if “vendors” means “COTS vendors” or “voting equipment vendors” in “vendors must adequately describe the control methods they have employed to ensure these risks have been mitigated.” COTS software was already covered in 5.1.1.
Eliminate the sections; ensure compliance with section 4.3.11 (“Previously developed or purchased software”) of IEEE Std 12281994, “IEEE Standard for Software Safety Plans”.
Delete the second sentence.
Change vendors” to “COTS vendors” or “voting equipment vendors”.
Eliminate “and software” from the first paragraph and eliminate item “a”.
“The software used by voting systems is selected by the vendor” appears to mean “COTS is selected”; else, it contradicts the subsequent sentence. Change the opening words from “The software” to “The COTS software”.
It needs to be specified how updates to software are going to be supplied and performed.
The decision by the FEC to exempt COTS products from inspection has created a serious security flaw. It should not be imperative that the IEEE standard continue to reflect this inappropriate practice. All exemptions for COTS product review should be removed from this standard.
* Documentation describing how an update is to be certified and performed, should there be a declared or discovered defect in the voting system, software, hardware, or firmware, or any COTS products used in or in the development of the system that could compromise its operation as an election device. Remove all exemptions for COTS product review from this standard on the grounds that such pose a serious security flaw. COTS products shall be presented in their entirety for open review in the same way that vendor software is examined.
Concerns addressing use of COTS products need to be added.
COTS products, especially software libraries, are a vulnerable attack point and must be subject to risks assessment prior to use in voting products. Configuration management should include vendor updates and alerts when flaws are detected that could compromise election operations or cast ballot data integrity. Object code modules should be provided such that compiled versions of programs can be compared.
Provision is made in the standard for update for COTS products releases, but there is no such provision for updating or decertifying non-COTS voting system components if such have been revealed to be insecure.
System changes that have resulted from identification of insecure voting system components must be propagated to all systems currently deployed. (This might be more appropriate in the configuration management section, or a different section under maintenance.)
There is a change of gears just past the middle of the paragraph.
Paragraph break with the sentence beginning "COTS products require updates…"
Memory leaks in C++ is not an example of an More appropriate would be "security inherent risk in COTS products. vulnerabilities in Microsoft products".
COTS equipment will be entrusted with counting votes but is exempted from this standard with a "proven record of performance"? OEMs of voting eqipment also have "proven" track records but must still test to this standard? This seems unreasonable.
Either require COTS equipment to comply to the same standards as all other voting equipment or remove the paragraph altogether.
"…COTS software …must be the most recent Remove this clause. verion of the COTS product …" The most recent version is not always stable enough to deploy and may not be compatible with the other aspects of the application. The vendor must have the latitude to employ the COTS versions and upgrades at the appropriate time.
COTS Hardware and software should not be exempted from qualification testing. This exemption should not be included in Definitions. The exemption is not a definition. Why specify that COTS software must be designed in a modular or object oriented fashion and not inspect it for compliance?
Eliminate the exemption.
Either eliminate the requirement or inspect for compliance.
Why exempt COTS hardware from environmental Require environmental testing of COTS testing? hardware. COTS software must work in conjunction with Eliminate the exemption of COTS software the voting application software. Therefore, from the testing requirement. it should be subjected to the same rigor of testing as the application software.
This is a far too vague and does nothing to Replace sentence with the following: address the security issues. "Underlying products, such as operating systems, database systems, firewalls, network devices, web browsers, smart cards, biometric devices, general purpose application components, libraries, and hardware platforms, that are crucial to the correct and secure operation of the entire system must be thoroughly tested. This includes COTS systems. In addition, there must be a line by line code review of ALL software that interacts with the voting system in any fashion. This is required because of the potential risk of malicious code."
There is no way to adequately test against all possible bugs and malicious code in COTS.
Add the requirement that all COTS used in any voting system must be open source.
Unmodified COTS must be evaluated at the source code level to protect against the threats identified in 5.3.2.1 (A).
COTS must meet the requirements of 5.1.3.1 COTS virus detection programs are not available for all operating systems.
Unmodified COTS are not exempt from evaluation to preclude the threats identified in 5.3.2.1 (A).
Delete “Unmodified third-party software is not subject to code examination; however,” and replace it with “All third party software shall be subject to source code an d other examination to preclude the presence of trap doors, hard-coded passwords, vulnerabilities and other nondeliberate errors, deliberate errors allowing the introduction of malicious code, and malicious code of any kind, especially malicious code intended to trigger upon use of the software in voting systems.” In the second sentence, after “security requirements defined in” insert “Section 5.1.3.1 and”. In the second sentence, replace the comma after “security patches” with “and”. Replace “and must be tested” by “. In complying with the requirement of 5.1.3.1, the vendor must document how the COTS has been defended against the threats identified in 5.1.2.3 (A-1), (A-3), (B-1) and (B-2), such as by testing”. Delete the second sentence of the definition.
Voter verified paper needs to be mandatory under certain circumstances
COTS evaluated should include compilers, libraries, and any other software tools used in system development and capable of introducing backdoors or other malicious code.
Add to the section created under comment SK4 above: A voter verified paper audit trail is mandatory for any system in which any of the following conditions is found: 1. Either the system software or any COTS used as either a system component or development tool, including compilers, libraries, and other tools, is too complex to clearly and thoroughly evaluate at the source code level to ensure absence of backdoors and other malicious code or means of introducing malicious code. 2. All other security, accuracy, integrity, and availability requirements are not satisfied clearly, easily, and without any question or requirement for interpretation. 3. There are any reports or significant suspicions that similar technology may have failed to record all ballots exactly as cast. 4. - There is any question whatever about the ability of all using jurisdictiions to easily and completely satisfy all assumptions regarding supervision of machines and relevant personnel at all times machines are in use, regarding fully secure storage of machines between elections, and regarding other COTS to be evaluated shall include compilers, libraries, and any other software tools used in system development and capable of introducing backdoors or other malicious code.
The COTS products may also be subject to a security evaluation themselves; such evaluations can support the voting system evaluation process. COTS, whether modified or not must be tested at least to system level.
delete second space before "voting system"
I would drop the last sentence.
In discussing the definition of COTS, this section goes on to say, “These devices and software are exempted from certain portions of the qualification testing process so long as such products are not modified in any manner for use in the voting system.” In general it is not a good idea to discuss policy in a definition. In particular, doing so here raises the question, which portions of the testing process are “certain” portions from which testing is exempted. This section of the draft has this language: “However, COTS software is not required to be inspected for compliance with this requirement but must be the most recent version of the COTS product incorporating all security patches,” [emphasis added] This section may be ambiguous. Must the latest version always be incorporated or only the latest version of security patches? What if the security patch is not relevant to the particular operation. In any case, forcing the latest version of COTS software is a configuration control nightmare and will result in endless requalification. One interpretation of this section is that software written to run on Windows 2000 must be rewritten and requalified to run on Windows XP even if it runs perfectly well on Win2000. An even worse interpretation requires vendors to update hard disk controllers with new firmware and drivers every time a new software version is available. We don’t think this is intended nor desirable. Note the term “Module”. The term Module is used here as it is used in the FEC VSS and we believe this usage to be non-standard. A module should be a collection of related
Remove the text in quotes.
This section has several problems. The module usage should be changed to subrouting or function, remove the strict requirement of only one exit per subroutine or function. Change so the most recent version of COTS is not required.