Get documents about "COTS Comments
Document Sample
Commenter and Clause/ Paragraph Status: OK = Type Comment Proposed Change S. A. Klein Doug Landol
Number Subclause Figure/ Table resolved with no
dissent; -1 =
resolved with
only one
dissenting
opinion by
someone who did
not subsequently
object, -2 or ? =
unresolved
because of 2 or
>2 dissenters to
the majority
opinion.
Sklein-051 3 26 OK T Unmodified COTS Delete the second Same as change
are not exempt sentence of the accepted above
from evaluation definition.
to preclude the
threats
identified in
5.3.2.1 (A).
wfw -001 3 26 OK E COTS, whether I would drop the Same as change Agree
modified or not last sentence. accepted above
must be tested at
least to system
level.
Dill-7 3 Def #26 OK T Explanation about Delete last Accept change. Agree
exemption is sentence of
unnecessary, and definition.
may become
inconsistent if
we add change
requirements on
COTS
df1 3 No. 26 OK E COTS - "These Delete sentence. Same as change Agree
devices and I do not believe accepted above
software are that is
exempted from appropriate in a
certain portions reference section
of the defining COTS.
qualification
testing process
so long as such
products are not
modified in any
manner for use in
the voting
system."
schneidewind -3 Pg. 10 OK T COTS Hardware and Eliminate the Same as change Agree
002 Line 26 software should exemption. accepted above
Definition 26 not be exempted
from
qualification
testing.
This exemption
should not be
included in
Definitions. The
exemption is not
a definition.
PPLX-001 3 Section 3. OK E In discussing the Remove the text Same as change Agree
Definition # 26 definition of in quotes. accepted above
COTS, this
section goes on
to say, “These
devices and
software are
exempted from
certain portions
of the
qualification
testing process
so long as such
products are not
modified in any
manner for use in
the voting
system.” In
general it is not
a good idea to
discuss policy in
a definition. In
particular, doing
so here raises
the question,
which portions of
the testing
process are
“certain”
portions from
which testing is
Lipsio-6D 3.26 OK E Second sentence Delete the second Same as change Agree
is not part of sentence. accepted above
the definition.
Whether or not my
later comments on
COTS are
accepted, “These
devices and
software are
exempted from
certain portions
of the
qualification
testing process
so long as such
products are not
modified in any
manner for use in
the voting
system” does not
belong in the
definition.
MercuriD50 - 4.6 Add bullet at OK G It needs to be * Documentation Accept comment Agree
013 (formerly end specified how describing how an only for technical
mercuri-034) updates to update is to be procedures.
software are certified and Administrative
going to be performed, should certification is
supplied and there be a out of scope.
performed. declared or
discovered defect
in the voting
system, software,
hardware, or
firmware, or any
COTS products
used in or in the
development of
the system that
could compromise
its operation as
an election
device.
RGH 072 5.4 Second -1 T COTS equipment Either require Require same Agree. Code is
paragraph above will be entrusted COTS equipment to standards. either in the TSF
clause 5.4.1. with counting comply to the Require all COTS (relied upon to
votes but is same standards as to be certified as enforce the
exempted from all other voting complying with security policy)
this standard equipment or this standard. or it is not.
with a "proven remove the
record of paragraph
performance"? altogether.
OEMs of voting
eqipment also
have "proven"
track records but
must still test
to this standard?
This seems
Sklein-007 5.6 Para 5.6.1.1 ? T Unmodified COTS Delete Accept change as I don't agree with
must be evaluated “Unmodified third-part of change the requirement
at the source party software is accepted for source code
code level to not subject to elsewhere. review. Delete all
protect against code examination; mentions.
the threats however,” and Furthermore, COTS
identified in replace it with should be treated
5.3.2.1 (A). “All third party no differently
software shall be than any other
subject to source software.
code an d other
examination to
preclude the
presence of trap
doors, hard-coded
passwords,
vulnerabilities
and other non-
deliberate
errors,
deliberate errors
allowing the
introduction of
malicious code,
and malicious
code of any kind,
especially
malicious code
intended to
trigger upon use
of the software
MercuriD50 - 7.13 ? T Provision is made System changes Covered by other Systems are not
078 (new) in the standard that have changes or out of either secure or
for update for resulted from scope as insecure there are
COTS products identification of administrative degrees of
releases, but insecure voting certification. assurance along a
there is no such system components continum. It is my
provision for must be understanding that
updating or propagated to all each election
decertifying non- systems currently official will be
COTS voting deployed. (This empowered to make
system components might be more a judgement. In
if such have been appropriate in practice you will
revealed to be the configuration find that sometime
insecure. management in the future a
section, or a system about to be
different section deployed will have
under a flaw that
maintenance.) slightly increases
the risk of not
enforcing a
security policy in
some special
circumstances.
Whereas one
election official
may decide to not
accept the voting
system as secure,
another may choose
to perform a
remedial
VCW-02 5.1.1 2nd to last OK E The COTS products delete second Don't understand Agree
para may also be space before comment. Proposed
subject to a "voting system" change does not
security appear to match
evaluation comment.
themselves; such
evaluations can
support the
voting system
evaluation
process.
RGH 006 5.1.1 last paragraph OK E There is a change Paragraph break Same as change
of gears just with the sentence accepted
past the middle beginning "COTS elsewhere.
of the paragraph. products require
updates…"
RGH 007 5.1.1 last paragraph OK T Memory leaks in More appropriate Same comment as remove reference
C++ is not an would be change accepted of example
example of an "security above.
inherent risk in vulnerabilities
COTS products. in Microsoft
products".
Corry-023 5.1.1 p. 20, last OK E Last three Start new Accept change Accept proposed
para. sentences should paragraph: [As] regarding change
be separate COTS products organization of
paragraph. require updates paragraph. Note
due to a detected paragraph is to be
security breach modified as part
or vulnerability of other changes.
[the] voting
system vendor
must provide a
method to assess
the impact of
COTS updates on
the voting
system, as well
as a method for
providing notice
and distribution
of updates to
purchasers[,
testing
facilities, and
election
officials and
boards]. Where
COTS products are
known to be
inherently risky
([e.g.,] memory
leaks in the C++
language),
Corry-022 5.1.1 p.20, 3rd OK T COTS may be Notwithstanding Accept need to Accept proposed
para., 3rd properly the fact that install relevant change
sentence installed and system certifiers patches. However,
configured but can rely upon the conflicts with
still not meet prior validations other comments on
requirements of the individual instability and
unless latest components of the impacts of latest
security patches system [ ] versions and
are installed. provided they are patches.
properly
installed and
configured [with
the latest
security
patches], there
must still be an
evaluation of the
integrated system
to make certain
that security
holes have not
been left or
created during
the integration
process.
Lipsio-12 5.1.1 Para. 5 ? T The treatment of Change “COTS Would need to see The section in
COTS products product may” to cited IEEE question is a
contradicts “COTS products standard before discussion of how
section 5.1.2.2, shall”. Mandate addressing the CC process
“Elements of compliance with proposed change. could be used in
Security Outside section 4.3.11 However, relevant the overall
of Vendor (“Previously IEEE standards assessment of
Control”. developed or should generally voting machines.
purchased be applied where "may" seems fine.
software”) of applicable in this Leave as is.
IEEE Std 1228- standard because
1994, “IEEE they reflect
Standard for relevant IEEE
Software Safety expertise.
Plans”.
Lipsio-14 5.1.1 Para. 7 ? T There is implied Mandate that Would need to see Sounds reasonable
a lack of testing testing preclude cited IEEE (not familiar with
in “COTS products any security standard before the standards
require updates breach or addressing mentioned.)
due to a detected vulnerability; proposed change.
security breach mandate However, relevant
or compliance with IEEE standards
vulnerability”; section 4.3.11 should generally
nothing that (“Previously be applied where
requires an developed or applicable in this
update should purchased standard because
pass testing. software”) of they reflect
IEEE Std 1228- relevant IEEE
1994, “IEEE expertise.
Standard for
Software Safety
Plans”. Mandate
COTS be subject
to the
specifications of
IEEE Std 1008™-
1987 (R1993),
“IEEE Standard
for Software Unit
Testing”. Add
reference to IEEE
Std 982.1™-1988,
“IEEE Standard
Dictionary of
Measures to
Produce Reliable
Lipsio-15 5.1.1 Para. 7 ? T “The voting Bring into Would need to see Don't know.
system vendor conformance with cited IEEE
must provide a Annex D (“V&V of standard before
method to assess reusable addressing
the impact of software“) of proposed change.
COTS updates on IEEE Std 1012- However, relevant
the voting 1998, “IEEE IEEE standards
system, as well Standard for should generally
as a method for Software be applied where
providing notice Verification and applicable in this
and distribution Validation”, standard because
of updates to e.g., “Reusable they reflect
purchasers” is software (in part relevant IEEE
inconsistent with or whole) expertise.
IEEE Std 1012- includes software
1998. from software
libraries, custom
software
developed for
other
applications,
legacy software,
or commercial-off-
the-shelf (COTS)
software. The V&V
tasks of Table 1
are applied to
reusable software
just as they are
applied to newly
developed
Lipsio-16 5.1.1 Para. 7 OK T Memory leaks are Eliminate “(ex. Accept change. Agree
the result of memory leaks in
using C++ the C++
language language)”
inappropriately;
they are not a
risk of a COTS
C++ compiler.
Lipsio-7A 5.1.1 Para. 7 OK E It is unclear if Change vendors” Change to “COTS Agree
“vendors” means to “COTS vendors” and voting system
“COTS vendors” or or “voting vendors”
“voting equipment equipment
vendors” in vendors”.
“vendors must
adequately
describe the
control methods
they have
employed to
ensure these
risks have been
mitigated.”
Simons - 002 5.1.1 the sentence ? G This is a far too Replace sentence Accept change as I believe the
that reads, vague and does with the part of change suggested rewrite
"The security nothing to following: accepted goes too far. I do
countermeasures address the "Underlying elsewhere. not think that all
implemented by security issues. products, such as of the examples
an IT system operating listed carry the
typically use systems, database same burden for
functions of systems, enforcing the
the underlying firewalls, security policy.
products and network devices, For example, web
depend upon the web browsers, browers are just
correct smart cards, expected to
operation of biometric perform as
those products devices, general expected, whereas
and their purpose biometric devices
security application are a core part of
functions." components, the identification
libraries, and system, and
hardware firewalls are
platforms, that relied upon to
are crucial to protect the system
the correct and from modification
secure operation etc. Remove all
of the entire examples. Also the
system must be requirement for a
thoroughly code review is not
tested. This supported
includes COTS elsewhere and is
systems. In not currently
addition, there support with a
must be a line by risk argument.
Sklein-056 5.1.3 All ? T Voter verified Add to the Accept change (it Disagree. Sounds
paper needs to be section created is mine). like an issue for
mandatory under under comment SK- the election
certain 4 above: A voter official. I am not
circumstances verified paper comfortable
audit trail is suggesting such a
mandatory for any mandate. Many of
system in which the terms and
any of the phrases in the
following suggested rewrite
conditions is (addition) are too
found: 1. subjective (too
Either the system complex). Many
software or any requirements go
COTS used as beyond the
either a system assurance that is
component or needed.
development tool,
including
compilers,
libraries, and
other tools, is
too complex to
clearly and
thoroughly
evaluate at the
source code level
to ensure absence
of backdoors and
other malicious
code or means of
Simons - 017 5.1.3.4.2 the entire ? G There is no way Add the Both open source Do NOT require
section to adequately requirement that and closed source that these be open
test against all all COTS used in are COTS. Both source. I see no
possible bugs and any voting system should be required requirement no
malicious code in must be open to be fully advantage of such
COTS. source. examined at the a requirement.
source level, as Resolution: Delete
recommended in the entire section.
resolution to COTS should not be
other comments. treated any
Otherwise, most differently than
experts agree that any other kind of
open source and software. The
closed source vulnerabilities
security are mentioned in the
roughly section apply
equivalent. equally to an
operating system
that I develop
myself as they do
one I buy off the
shelf.
Lipsio-80 5.1.3.6.5 ? E COTS software was Eliminate “and Reject change. Agree
already covered software” from Section is
in 5.1.1. the first explicitly focused
paragraph and on networking.
eliminate item Other sections
“a”. cover software
generally. This
covers networking
software.
MercuriD50 - 5.6.1.1 Section ? G Concerns COTS products, Covered by other Already covered
064 (formerly addressing use of especially changes. elsewhere
mercuri-143) COTS products software
need to be added. libraries, are a
vulnerable attack
point and must be
subject to risks
assessment prior
to use in voting
products.
Configuration
management should
include vendor
updates and
alerts when flaws
are detected that
could compromise
election
operations or
cast ballot data
integrity.
Object code
modules should be
provided such
that compiled
versions of
programs can be
compared.
Alice - 001 5.6.1.1 ? T "source code Delete this Depends on Suggested change:
generated by COTS clause resolution of "Unmodified third-
code development comments regarding party software is
package and software tools not subject to
embedded in used in voting code examination.
software modules system Software that has
for compilation development. If been modified or
or interpretation the tools are generated since
shall be provided thoroughly verification
in human readable inspected at the (e.g., source code
form" Some source code level generated by COTS
newer programming and otherwise code development
tools do not certified for use package) shall be
necessary in voting system provided in human
generate development, readable form."
traditional accept the change. This same sentence
source code as Otherwise, leave is repeated in
reference within the language as section 6.6.2.
this clause. is. Please make it
consistent if
there are any
changes.
Lipsio-89 5.6.2 ? E “The software The word Agree
used by voting “selected” is
systems is partly incorrect.
selected by the Add “or developed”
vendor” appears after “selected”.
to mean “COTS is
selected”; else,
it contradicts
the subsequent
sentence. Change
the opening words
from “The
software” to “The
COTS software”.
Lipsio-3E 5.6.2.2 para. 1 ? T Industry standard Require all Accept change as I agree with the
COTS compiler and tools, including part of change principle that
runtime compilers and accepted something is not
interpreter both interpreters, to elsewhere. fool-proof just
is not defined be validated and because it is in
and assumes that, verified in the common use.
contrary to same manner as However, it is
reality, application impractical to
something is fail- software. require that
safe and fool- compilers and
proof by virtue interpreters are
of being in validated and
common use. verified.
Information
security has a
long history of
implicitly
trusting these
because of the
impractibility of
assessing them.
Reasonable
assurance can be
gained from
indirect review of
these intrepreters
and compilers
(i.e., testing the
resultant
executable
rigorously enough
Sklein-057 5.6.2.3 5.6.1.1 ? T COTS evaluated COTS to be Accept change as Disagree. These
should include evaluated shall part of change developmental and
compilers, include accepted support tools are
libraries, and compilers, elsewhere. not relied upon to
any other libraries, and enforce the
software tools any other security policies
used in system software tools of the system
development and used in system instead they
capable of development and merely need to
introducing capable of perform as
backdoors or introducing expected. These
other malicious backdoors or tools could be
code. other malicious tested to ensure
code. their performance
is as expected,
but this is
typically a high
assurance activity
and seems beyond
what is needed
here.
PPLX-035 5.6.2.3 5.6.2.3 OK T & This section of This section has Require only Delete entire
Software E the draft has several problems. relevant version section
Modularity and this language: The module usage and patches per
Programming “However, COTS should be changed other accepted
software is not to subrouting or comments. Don't
required to be function, remove know what to do
inspected for the strict about part of
compliance with requirement of comment regarding
this requirement only one exit per module definition.
but must be the subroutine or
most recent function. Change
version of the so the most
COTS product recent version of
incorporating all COTS is not
security required.
patches,”
[emphasis added]
This section may
be ambiguous.
Must the latest
version always be
incorporated or
only the latest
version of
security patches?
What if the
security patch is
not relevant to
the particular
operation.
Sklein-044 5.6.2.3 First paragraph OK T COTS must meet In the second Accept change.
the requirements sentence, after
of 5.1.3.1 “security
requirements
defined in”
insert “Section
5.1.3.1 and”.
Sklein-045 5.6.2.3 First paragraph OK T COTS virus In the second Accept change (it
detection sentence, replace is mine).
programs are not the comma after
available for all “security
operating patches” with
systems. “and”. Replace
“and must be
tested” by “. In
complying with
the requirement
of 5.1.3.1, the
vendor must
document how the
COTS has been
defended against
the threats
identified in
5.1.2.3 (A-1), (A-
3), (B-1) and (B-
2), such as by
testing”.
Lipsio-43 5.6.2.3 Para. 1 OK T “COTS software is Eliminate the Accept change I agree that COTS
not required to section, or, (reverse sense) as software should
be inspected…” is better yet, part of change not be exempt from
contrary to such reverse its accepted review. If it is
other mission- sense. elsewhere. part of the
critical enforcement of the
methodologies as security policy
those used by the (the TCB or TSF)
FDA and FAA, and then it needs to
contradicts what be reviewed. See
is specified in the suggested
section 5.1.3.3.2 changes in row 13.
Lipsio-44 5.6.2.3 Para. 1 ? T There is implied Mandate that Would need to see I don't understand
a lack of testing testing preclude cited IEEE the statement
in “COTS products any security standard before "Nothing that
require updates breach or addressing requires updating
due to a detected vulnerability; proposed change. should pass
security breach mandate However, relevant testing." Any part
or compliance with IEEE standards of the system may
vulnerability”; section 4.3.11 should generally require an update
nothing that (“Previously be applied where at any time.
requires an developed or applicable in this Instead the
update should purchased standard because concept of
pass testing. software”) of they reflect emerging threat
IEEE Std 1228- relevant IEEE analysis must be
1994, “IEEE expertise. adopted. When a
Standard for system is deployed
Software Safety it must be shown
Plans”. Mandate to be resistant to
COTS be subject all "known"
to the vulnerabilities.
specifications of The
IEEE Std 1008™- vulnerabilities
1987 (R1993), that are known
“IEEE Standard change all the
for Software Unit time. I suggest
Testing”. Add incorporating the
reference to IEEE following language
Std 982.1™-1988, to address this
“IEEE Standard concern. "System
Dictionary of deployment must
Measures to include a review
Produce Reliable of the system
Lipsio-45 5.6.2.3 Para. 1 ? T There is implied Bring into Would need to see See above
a lack of testing conformance with cited IEEE
in “the most Annex D (“V&V of standard before
recent version of reusable addressing
the COTS product software“) of proposed change.
incorporating all IEEE Std 1012- However, relevant
security patches” 1998, “IEEE IEEE standards
”; nothing that Standard for should generally
requires an Software be applied where
update should Verification and applicable in this
pass testing. Validation”, standard because
e.g., “Reusable they reflect
software (in part relevant IEEE
or whole) expertise.
includes software
from software
libraries, custom
software
developed for
other
applications,
legacy software,
or commercial-off-
the-shelf (COTS)
software. The V&V
tasks of Table 1
are applied to
reusable software
just as they are
applied to newly
developed
schneidewind -5.6.2.3 Pg. 70 ? T Why specify that Either eliminate Inspect for Delete section
005 COTS software the requirement compliance. 5.6.2.3. I don't
must be designed or inspect for see the need for
in a modular or compliance. this requirement.
object oriented If there is such a
fashion and not need COTS should
inspect it for compliance? not be exempted,
but such a need
should be
determined based
on the percieved
risks. The Common
Criteria PP can
provide the
rationale for
against such a
requirement.
RGH 117 5.6.2.3 OK T "…COTS software Remove this Accept comment. Agree. I see no
…must be the most clause. Only relevant need to require
recent verion of security patches the most recent
the COTS product should be required version. Better
…" The most to be applied. yet. "If security
recent version is patches are
not always stable available for the
enough to deploy COTS products
and may not be these must be
compatible with reviewed for
the other aspects possible
of the inclusion."
application. (Sometimes patches
The vendor must make things worse,
have the latitude sometimes they do
to employ the not apply in
COTS versions and specific
upgrades at the environments,
appropriate time. sometimes the
threat is minor.)
Corry-139 6.4.4.1 2nd para., 1st OK T Systems that are Delete first Accept comment in Suggested rewrite:
sentence simply cobbled sentence of concept. Unclear "Systems that
together (kluge second paragraph. how to implement. employ the use of
might be a better Exemption from COTS hardware
description) from environmental whose
COTS components testing of COTS is configuration has
must not be purportedly based not been modified
exempted from on the idea that in any manner and
environmental the COTS has does not violate
testing. I've had already undergone the evaluated
too many problems testing. Might be configuration
with little better to exempt requirements of
doohickies hung COTS that are its previous
on some piece of certified as testing are not
otherwise great complying with subjected to this
equipment that standard from segment of
caused problems individual unit hardware testing."
when fielded. testing.
Lipsio-4B 6.4.4.1 Para. 2 OK T COTS hardware Change paragraph Accept as part of The suggested text
must have been to “COTS systems change accepted could be
tested to the or components elsewhere. interpreted to
rigor required of must be mean that the
non-COTS documented by supplier
components; if their suppliers themselves are
the supplier has to have been objective enough
not done this, tested to at to test the
then COTS least the same products. I
hardware must be rigor as required rewrote this to
treated like any of voting devices remove such a
other component. as specified possible
hereinbelow; confusion.
else, the said Suggested rewrite:
COTS components "Documentation
shall be tested that COTS systems
in a like manner or components have
to any other been tested to at
component.” least the same
rigor as required
of voting devices
as specified
hereinbelow must
be supplied by
their supplier;
else, the said
COTS components
shall be tested in
a like manner to
any other
component.”
schneidewind -6.4.4.1 Pg. 100 OK T Why exempt COTS Require Same change as See above (row 11)
006 hardware from environmental accepted
environmental testing of COTS elsewhere.
testing? hardware.
Lipsio-4D 6.6.2 para. 3 & 4 -1 T “Unmodified, Eliminate the Would need to see See above
general purpose sections; ensure cited IEEE
COTS non-voting compliance with standard before
software ...is section 4.3.11 addressing
not subject to (“Previously proposed change.
code developed or However, relevant
examination...is purchased IEEE standards
not subject to software”) of should generally
the full code IEEE Std 1228- be applied where
review and 1994, “IEEE applicable in this
testing” is Standard for standard because
contrary to such Software Safety they reflect
other mission- Plans”. relevant IEEE
critical expertise.
methodologies as
those used by the
FDA and FAA, and
contradicts what
is specified in
section
5.1.3.3.2.
MercuriD50 - 6.6.2 Paragraphs 2-4 -1 G The decision by Remove all Accept and change I agree that COTS
022 (formerly the FEC to exempt exemptions for where appropriate. software should
mercuri-048) COTS products COTS product Some changes not be exempt from
from inspection review from this identified in review. If it is
has created a standard on the other equivalent part of the
serious security grounds that such comments. Others enforcement of the
flaw. It should pose a serious to be determined. security policy
not be imperative security flaw. (the TCB or TSF)
that the IEEE COTS products then it needs to
standard continue shall be be reviewed. See
to reflect this presented in the suggested
inappropriate their entirety changes in row 13.
practice. All for open review
exemptions for in the same way
COTS product that vendor
review should be software is
removed from this examined.
standard.
schneidewind -6.6.2 Pg. 107 -1 T COTS software Eliminate the Accept change as Agree
007 must work in exemption of COTS part of change
conjunction with software from the accepted
the voting testing elsewhere.
application requirement.
software.
Therefore, it
should be
subjected to the same
rigor of testing as the
application software.
Dill-35 6.6.2 -1 T If COTS hardware Specify that the Trusted subset Suggested rewrite:
or software is in COTS exclusion needs to be "Unmodified,
the trusted only applies to defined before general purpose
subset, it must system components this comment can COTS software
be treated outside the be addressed [that is not
exactly like trusted subset. relied upon to
software or enforce the
hardware designed security policies
by the vendor. of the system] is
not subject to the
detailed
examinations
specified in this
section."
Suggested Rewrite:
"Unmodified COTS
software [that is
not relied upon to
enforce the
security policies
of the system] is
not subject to
code examination."
Phil Scruggs David Dill Bob Oliver Vince Lipsio R. Mercuri
Accept proposed Accept. Eliminate the (see my email on Accept proposed
change definition and add definition of change
in "Abbreviations COTS)
and Acronyms" the
text: "(subsystems
or components;
software,
electronic,
mechanical, et
cetera)" - Lipsio
email #3
Accept proposed I don't know. Eliminate the (see my email on Same as Sklein-051
change definition and add definition of
in "Abbreviations COTS)
and Acronyms" the
text: "(subsystems
or components;
software,
electronic,
mechanical, et
cetera)" - Lipsio
email #3
Accept proposed Accept. Eliminate the (see my email on Same as Sklein-051
change definition and add definition of
in "Abbreviations COTS)
and Acronyms" the
text: "(subsystems
or components;
software,
electronic,
mechanical, et
cetera)" - Lipsio
email #3
Accept proposed The principle Eliminate the (see my email on Same as Sklein-051
change should be that definition and add definition of
COTS products in "Abbreviations COTS)
should be and Acronyms" the
evaluated to the text: "(subsystems
same level of or components;
assurance as non- software,
COTS components. electronic,
Documentation of mechanical, et
this evaluation cetera)" - Lipsio
shall be available email #3
to the ITA, which
shall review it to
ensure that the
evaluation was
adequate. The
COTS product must
be identical to
the product that
was reviewed.
Accept proposed COTS should not be Eliminate the (see my email on Same as Sklein-051
change exempted, but definition and add definition of
comparable levels in "Abbreviations COTS)
of evaluation with and Acronyms" the
a non-COTS text: "(subsystems
components in the or components;
same role should software,
be accepted if electronic,
they are mechanical, et
documented and cetera)" - Lipsio
verified. email #3
Accept proposed Accept. Eliminate the (see my email on Same as Sklein-051
change definition and add definition of
in "Abbreviations COTS)
and Acronyms" the
text: "(subsystems
or components;
software,
electronic,
mechanical, et
cetera)" - Lipsio
email #3
Accept proposed Accept. Eliminate the (My comment) Same as Sklein-051
change definition and add
in "Abbreviations
and Acronyms" the
text: "(subsystems
or components;
software,
electronic,
mechanical, et
cetera)" - Lipsio
email #3
Accept proposed This needs more Accept the Agree; but should Given recent
change discussion. All proposal. add verbiage to revelations
updates should be the effect that regarding the
evaluated by should such a unauthorized use
whatever defect be found, of uncertified
certification the device shall code, it is
authority be decertified imperative that
evaluated the and the V&V this be clarified
original COTS procedures shall in the standard.
system. be amended to Even though this
flag the found was originally my
defect, and full comment, I now
regression agree with Vince's
testing shall be stronger
performed before requirements of
the corrected decertification
device be again and evaluation as
certified. to whether an
update could
compromise
operations prior
to
recertification.
Require COTS to Accept. NC - unmodified Agree, but should Require COTS to
meet standards. COTS components add a note that meet standards.
that have what
documented results distinguishes
that meet or COTS from other
exceed components is
environmental that the testing
condition testing of COTS
need not be components is
retested. performed by the
suppliers thereof
and proof of that
testing is
supplied with the
COTS components,
while for
Do not accept the Accept. NC - This proposed Agree, but need Any use of COTS
change. It is not change is not to insert “, that could impact
practical or, in practical and among other security or
most cases, would put many things,” between functionality of
feasible to vendors in a poor “presence” and the voting product
perform a line-by- competitive “of”. must be thoroughly
line review. position. examined.
Operating systems, Alternative
drivers, etc. may certification
not be available (such as Common
for line by line Criteria EAL4)
analysis. that is
independently
recognized could
be acceptable.
No to implement Accept. Accept proposed Language needs to Refer to Raba
this change - change. be stronger; report as to why
Should be left at otherwise, this is necessary
the discression of accept. Any (they describe
the election demonstrated lack exploits that
officials. of security or could be used by
integrity in and Blaster if the OS
component, COTS is not updated,
or otherwise, of for example).
the voting device
shall result in
decertification,
the remedying
whereof shall
require that the
V&V procedures
shall be amended
to flag the found
defect, and full
regression
testing shall be
performed before
the corrected
device be again
certified.
Accept proposed Accept. Accept proposed Sure. Turn over Agree (This is
change change. to the editing just an editing
committee with change. ;-)
the suggestion
that they excise
all redundant
spaces within
sentences.
Accept proposed Don't know Accept proposed Agree. Agree to splitting
change change. into 2 paragraphs,
but the 2nd
paragraph should
start with the
sentence:
"Guidance on how
to securely
configure COTS
products..."
Should not Reword. It is Accept proposed Use my language Agree that memory
reference a inappropriate to change. found in Lipsio- leaks is not a
specific COTS mention a 16. Methinks it good example, but
vendor's particular inappropriate to why pick on
vulnerability. company. cite a specific Microsoft? LINUX
Remove referenced COTS vendor and isn't immune.
example. not necessarily Need a more
correct to reckon generic, but good,
a memory leak as example.
a “security”
vulnerability.
Accept proposed Accept. Accept proposed Agree Last 4 sentences
change change. should be new
paragraph (as RGH
006)
Accept proposed Accept. NC - No suggested Agree Need to reiterate
change wording proposed. need for updates
with patches (see
Mercuri D50-013)
but also must
check patches and
effect on security
prior to
installation.
No comment Accept. NC - "may" allows (My comment) Accept proposed
the case where change
existing data can
help the
evaluation
process. Bring in
the essence of the
referenced specs
if they apply.
Many people,
myself included,
are not intimate
with the
references.
Accept proposed Accept. Change to "COTS (My comment) Accept proposed
change products may change
require.." Bring
in the essence of
the referenced
specs if they
apply. Many
people, myself
included, are not
intimate with the
references.
No comment Accept. NC - Bring in the (My comment) Would need to see
essence of the cited IEEE
referenced specs standard.
if they apply
Many people,
myself included,
are not intimate
with the
references.
Accept proposed Some languages are Accept proposed (My comment) See RGH 007
change much less prone to change.
memory errors than
others (e.g, Java
vs. C). Reword
original comment
to refer to memory
errors, which are
much more likely
to cause serious
harm, and not to
imply that it is a
compiler error
rather than
programmer error.
Accept proposed Accept. Change vendors” to (My comment) Change to "COTS
change “voting equipment and voting system
vendors”. vendors"
Do not accept Accept. The first sentence Assuming that the Interaction with
change. It takes is reasonable and scope is COTS components
something that is can be accepted. explicitly may not be readily
too vague then A determination of limited to what understood until
caries the what is "crucial is internal t the such assessment is
argument onto to the corrct and voting device, I performed. The
multiple tangents. secure operation" agree; for change should
is key here. For external devices reflect that all
example, voter connected to the COTS products must
verifiable paper voting device, be assessed in
ballots may proof that these order to determine
obviate some or devices can not their involvement
all software affect the with the security
security concerns. functioning of functions, and
The last sentence the voting device those that are
of the proposed should be identified as
change is not sufficient. having critical
practical and impact should be
would put many examined in
vendors in a poor detail. (Note: I
competitive would actually
position. prefer an
Operating systems, alternative,
drivers, etc. may independent,
not be available official, security
for line by line certification,
analysis. such as an
appropriate level
common criteria
evaluation for the
COTS products,
Do not accept I agree in Accept proposed Out of scope to The system audit
change. This principle with change with the mandate VVPT section (5.1.3-4)
should be a county this, but this is following here. However, here should be
requirement not a another example of modification: please note that reworded to
standard. a high-level Replace "A voter the stringency of reflect the fact
policy question verified paper V&V would not, in that auditability
that should be audit trail..." my opinion, be must pertain to
discussed before with "A secure nearly so great the ballots as
line-by-line voter verifiable in a system with well -- anonymity
editing. We need approach..." an independent of the ballots
to grapple with audit trail. As should not exempt
the question of things stand auditing at the
“What is an where no such most critical
adequate level of audit trail point, that of the
security?”, and exists, I am collection of the
“How do we inclined to test ballots. The COTS
identify the the systems to sections in this
components of a the standards of, part of the
system that are at least, RTCA/DO- standard 5.1.3.6.4
critical to 178B, ("Software and 5.1.3.6.5 need
achieve that level Considerations in to be augmented so
of security?” I Airborne Systems that auditability
strongly question and Equipment is possible
whether an Certification"), throughout. If
adequate level of while with the this is explained
security can be presence of an well, it should
achieved by independent audit not be necessary
anything short of trail, robustness to specify a
a voter verified is much less particular
paper trail. critical, and the implementation.
main concern
Agree with S. Open source should NC - This proposal Don’t know: Agree Open source does
Klein comment. only be required is clearly anti- in principle that not provide any
for security competitive. open source significant
critical should be advantage in terms
components, COTS required but if of security to
or custom. we require all closed source, in
tools to be fact, it might
certified, to even provide a
have undergone false sense of
formal V&V, I security. I
don’t see know disagree that an
that we can open source
mandate this. requirement
Disagree with the resolves this
implication that issue. But it
open source will does need
provide rewriting as per
adequately sKlein-056.
testing; rather,
it will improve
the adequacy of
testing.
Agree with SAK Accept. Accept the (My comment) Networking
proposed change. components may
contain embedded
software modules
that also need to
be examined, as
these can be
critical to proper
operation. Reject
proposed change.
No not implement Accept. Accept the Proposed Change: Need to ensure
change - comment proposed change. COTS products, that all of this
already addressed. especially is covered
software elsewhere in the
libraries, are a latest draft of
vulnerable attack the standard.
point and must be
subject to risks
assessment prior
to use in voting
products.
My comment: Agree
Proposed Change
(continued):
Configuration
management should
include vendor
updates and
alerts when flaws
are detected that
could compromise
election
operations or
cast ballot data
integrity.
My comment: Agree
with the
following
changes:
1) Change
“should” to
Agree with DJL Suggest we replace NC - Modified code No. Rather, Some code may be
with a requirement must be capable of require to be generated through
that all files and evaluation. provided in human other methods --
tools used to readable form the table-driven
generate the COTS code written by a packages, for
object code or human, which example. In such
executable by would be the cases, what must
supplied. The input to a COTS be provided are
build process code development the original files
should be executed package that and the programs
at the ITA. The produces “source used to generate
tools must either code”, and not code from them.
be approved by the necessarily the This needs to be
ITA, or the code produced by explained here,
generated files the package. If not exempted.
must be checked the input to a
for consistency COTS code
with the files development
from which they package is not
were derived at “traditional
each step. source code”, it
still must be
provided in a
human readable
form, even is
that means a
screen shot or
whatever.
Accept proposed Accept proposed (My comment) It is unclear what
change change. is meant --
whether it is the
COTS software or
custom. Need
clarification
(perhaps by
looking at earlier
versions of this
section, or the
FEC document?).
Accept proposed Accept. NC - This proposed (My comment) All tools must be
change change is not subject to
practical and validation, but
would put many even this does not
vendors in a poor provide full
competitive assurances (see
position. Tools Ken Thompson
may not be "Reflections on
available for line Trusting Trust"
by line analysis. paper). Here
again, it might be
best to seek
alternative
security (like
common criteria)
certification,
since that might
be more stringent
than what the
voting system
evaluators are
capable of
providing. The
reason for this
should be
explained here as
well.
Do not accept Accept. Accept proposed Agree. Change is
change. Not change. necessary and
practical or should be
necessary. accepted.
Delete entire The module notion Accept proposed Proposed Change I agree that it is
section. is outdated. change. (partial): This not necessary that
Perhaps this section has the latest version
should be talking several problems. of compilers be
about classes. The module usage used, only that
I'm not convinced should be changed the version that
that the single to subroutine or is used be
exit rule is worth function. The appropriately
preserving. relationship of security
these terms is qualified. I
not yet agree that the
standardized. definition of
My comment: Not "module" may need
correct; to be made more
ANSI/IEEE Std clear, although I
610.12-1990, don't think this
"IEEE Standard should be done by
Glossary of the COTS group.
Software Regardless, it may
Engineering be difficult (or
Terminology" impossible) to
defines module require COTS
thusly: products to
module. (1) A conform to the
program unit that voting system
is discrete and standard's
identifiable with definition of what
respect to a module is,
compiling, anyway.
combining with
other units, and
Accept proposed Accept. Accept proposed Agree. Accept proposed
change change. change
Accept proposed Accept. If a Accept proposed Agree. Accept proposed
change virus as a post- change. change
facto modification
of object code, we
should insist
that all
intermediate and
object files be
regenerated at the
ITA and that they
must be bit-for-
bit identical with
supplied files.
Accept proposed Reverse sense. Replace with (My comment) Accept proposed
change "Underlying change
products, such as
operating systems,
database systems,
firewalls, network
devices, web
browsers, smart
cards, biometric
devices, general
purpose
application
components,
libraries, and
hardware
platforms, that
are crucial to the
correct and secure
operation of the
entire system must
be thoroughly
tested."
No comment Accept. NC - Bring in the (My comment) Need to see
essence of the standard.
referenced specs
if they apply
Many people,
myself included,
are not intimate
with the
references.
No comment Accept. NC - Bring in the (My comment) Need to see
essence of the standard.
referenced specs
if they apply
Many people,
myself included,
are not intimate
with the
references.
Remove Accept. Eliminate the Agree. Note that Inspect for
requirement. requirement. the mandated compliance (as per
design simplifies other comments).
testing, but I
don’t see that it
should be a hard
requirement,
especially for
where it is found
necessary to use
assembly
language. But if
so mandated, then
of course, it
needs to be
tested.
independent audit
trail, robustness
is much less
critical, and the
main concern
would be that the
device is
reliable, not
crashing or
malfunctioning so
that it is an
annoyance to use
or be in charge
of, which is to
say less
Accept proposed Accept. Accept proposed Agree. Consider Agree, most recent
change change. FAA and FDA version need not
criteria which be required, only
allow whatever that the version
version of COTS used be secure,
is certified for etc.
the purpose. The
price of
repeating V&V
every time a COTS
software
component is
revised is
prohibitive.
Agree with DJL Accept. Accept proposed Rewrite. Would Kluge COTS must
change. accept wording also be subject to
proposed by Doug inspection.
Landol.
Accept proposed Further modify to Accept proposed (My comment) Documentation by
change include that COTS change. suppliers is
components be insufficient, must
unchanged from require formal
certified certification
versions, and be (such as common
buildable at the criteria) or other
ITA from original recognized
human-generated independent
input files. evaluation.
Accept proposed Accept. Accept proposed Agree unless, of Accept proposed
change change. course, the COTS change as in Corry-
vendor supplies 139
test data showing
compliance and it
can not be
demonstrated that
the manner
wherein the
hardware is used
would invalidate
that testing in
context.
Accept proposed Accept. NC - Bring in the (My comment) Must not exempt
change essence of the COTS from
referenced specs examination. See
if they apply MercuriD50-022
Many people, (below).
myself included,
are not intimate
with the
references.
All COTS must be This is a high- NC - This proposed Agree. Cross- Accept change.
reviewed, however, level issue that change is not reference my
modified COTS more needs more practical and comment for RGH
so. Also, the discussion. would put many 072.
interface to and Different aspects vendors in a poor
from the COTS must of the system need competitive
be examined. vastly different position.
levels of Operating systems,
scrutiny, drivers, etc. may
depending on not be available
whether they are for line by line
in the trusted analysis.
subset. COTS
components are
really not
different from
custom components
in this respect.
Accept proposed Accept. NC - see previous Agree. Must not exempt
change comments COTS from
examination. See
MercuriD50-022
(above).
Agree with DJL Accept, subject to Accept proposed Disagree. COTS Trusted subset is
clarification of change. testing not well-defined.
“trusted subset” (performed by the Change as above
or equivalent COTS supplier) (MercuriD50-022).
concept. We need shall be to the
some high-level same criteria and
discussion of this standards as it
topic. Line-by- would be tested
line editing is to were it
not going to designed by a
resolve it. voting equipment
vendor.
