This article was posted at Wired.com's Threat Level Blog and is reposted with permission of the author.
Following three months of investigation,
California's secretary of state has released a report examining why a
voting system made by Premier Election Solutions (formerly known as
Diebold Election Systems) lost about 200 ballots in Humboldt County
during the November presidential election.
But the most startling information in the state's 13-page report (.pdf) is not about why the system lost votes, which Threat Level previously covered in detail,
but that some versions of Diebold's vote tabulation system, known as
the Global Election Management System (GEMS), include a button that
allows someone to delete audit logs from the system.
Auditing logs are required under the federal voting system
guidelines, which are used to test and qualify voting systems for use
in elections. The logs record changes and other events that occur on
voting systems to ensure the integrity of elections and help determine
what occurred in a system when something goes wrong.
"Deleting a log is something that you would only do in
de-commissioning a system you're no longer using or perhaps in a
testing scenario," says Princeton University computer scientist Ed
Felten, who has studied voting systems extensively. "But in normal
operation, the log should always be kept."
Yet the Diebold system in Humboldt County, which uses version
1.18.19 of GEMS, has a button labeled "clear," that "permits deletion
of certain audit logs that contain – or should contain – records that
would be essential to reconstruct operator actions during the vote
tallying process," according to the California report.
The button is positioned next to the "print" and "save as" buttons
(see image above), making it easy for an election official to click on
it by mistake and erase crucial logs.
In fact, the report says, this occurred recently in a California
county when an official, while attempting to print out a copy of a
so-called "poster log," inadvertently deleted it instead.
system provides no warning to the operator that clicking on the button
will result in permanent deletion of records in the log, nor does it
require the operator to confirm the action before executing it.
Apparently Premier/Diebold was aware that having a "clear" button on
its system was a bad idea. According to California's report, one of the
system's developers wrote in an e-mail in 2001: “[a]dding a 'clear'
button is easy, but there are too many reasons why doing that is a bad
idea.” Yet the company included the button in its system anyway.
The button was removed from version 1.18.20 of the software and
following, but Premier/Diebold never went back to jurisdictions using
previous versions to upgrade them, and version 1.18.19 is still used in
three California counties as well as in other states. It's unclear how
many previous versions of the software had the button, or why it was
included in the first place.
According to the report:
report states that the inclusion of the button violated the federal
voting system standards under which the Premier/Diebold system was
qualified to be used in elections. The standards require that voting
system software automatically creates and permanently retains
electronic audit logs of important system events that occur on the
machine.Premier/Diebold did not respond to a request for comment.
The “Clear” buttons . . .
allow inadvertent or malicious destruction of critical audit trail
records in all GEMS version 1.18.19 jurisdictions, risking the accuracy
and integrity of elections conducted using this voting system. Five
years after the company recognized the need to remove the “Clear”
buttons from the GEMS audit log screens, not only Humboldt, San Luis
Obispo and Santa Barbara Counties in California but jurisdictions in
other parts of the country, including several counties in Texas and
Florida, continue to use GEMS version 1.18.19. . . .
The "clear" button isn't the only problem with the auditing log in the Premier/Diebold system. I wrote previously about other issues with the logs
-- for example, they don't record significant events that occur in the
tabulation system, such as when someone deletes votes from the software.
The California report states that the "clear" button, along with
other problems with the auditing logs as well as the software flaw that
caused the system to lose votes in Humboldt County (see below for more
information on that flaw), should have been red flags to the testing
laboratories that certified the system and should have been sufficient
to "fail" the system and prevent it from being used in any federal
As the report points out, under the voting system standards (VSS)
"each of the errors and deficiencies in the GEMS version 1.18.19
software described in this report standing alone would warrant a
finding by an Independent Testing Authority (ITA) of 'Total Failure'
(indicated by a score of 1.0) had the flaw been detected. Under the
1990 VSS, a finding of 'Total Failure' required failure of the voting
"Presumably some organization, some lab, looked at this system and
decided they thought it complies with the standard," says Felten. "And,
obviously, they were wrong. Any state that uses GEMS should be looking
at this seriously."
The findings raise questions about the auditing logs on voting
systems made by other vendors and about what states that use the
Premier/Diebold system will do now that they know their voting software
does not create an adequate audit trail to ensure the integrity of an
California's secretary of state will be holding a public hearing on March 17
(.pdf) to discuss the report and whether version 1.18.19 of GEMS should
be decertified in the state. The state can't order counties not to use
the software, but decertifying this software version will force
counties to upgrade to different versions.
As for addressing the fundamental problems with the auditing logs in
all versions of the GEMS software, a spokeswoman for the secretary of
state's office said only that the state sent the report to the federal
Election Assistance Commission to post on its web site and communicate
the issue to election officials in other jurisdictions.
A spokeswoman for the EAC told Threat Level that the commission has
no authority to address problems with voting systems that were tested
and qualified before it assumed responsibility for voting machines,
even if those machines are in violation of the voting system standards.
In 2002, Congress gave the EAC oversight responsibility for the testing
and qualification of voting systems, but the commission has yet to
shepherd a voting system through testing and certification. Prior to
the EAC assuming responsibility for voting systems, the National
Association of State Election Directors voluntarily assumed the
oversight task and it was under NASED's watch that all of the voting
systems currently in use in the country were tested and certified.
"There's no regulatory action that we could take," says EAC
spokeswoman Jeannie Layson. "But certainly in the area of notification,
when the voting sytem reports come out, the commissioners make sure
that the test labs and independent reviewers who look at the test
reports are aware of all that information."
The lab that was responsible for testing and qualifying GEMS version
1.18.19 with the "clear" button is Colorado-based Ciber, Inc. In 2007,
the lab was barred from testing voting systems
for not following quality-control procedures and for failing to
document that it was conducting all tests. The EAC restored the lab's
accreditation to test voting systems last year.
Ciber did not respond to a call for comment about its examination of
the Premier/Diebold system and its approval of the "clear" button.
As mentioned above, the California report is the result of an
investigation into what occurred in Humboldt County during the
After county officials had already certified their election results to state officials, they discovered that the tabulation software they used to tally votes
had dropped 197 paper vote-by-mail ballots from the totals at one
precinct. The system did so without giving any warning or message to
election officials that it was doing so. Humboldt uses a central-count
optical-scan system made by Premier/Diebold.
The vendor acknowledged that a programming flaw in its software
caused the system to randomly and silently delete votes and disclosed
that it had known about the problem since October 2004 and provided
some election officials with a workaround, though Humboldt County
election director Carolyn Crnich had never been told of the problem.
The issue involved a programming error in version 1.18.19 of GEMS
that caused ballots to be randomly dropped from the system. The GEMS
software is used to tabulate votes on both touch-screen and
optical-scan machines, but the problem only occurred when the software
was used to tabulate votes scanned on a central-count optical-scan
system -- a high-speed optical scanner that is used in a county's
election headquarters, as opposed to precinct-based optical scanners
that are used at polling places.
The California secretary of state investigated and confirmed that a
flaw in the GEMS software can automatically delete the first batch of
ballots scanned into the system if officials delete a subsequent batch
-- something that occurs on a regular basis when officials make a
mistake during scanning. The system provided no notice to officials
whenever it deleted such ballots.
As the California report notes, the loss of votes in Humboldt County
could have been much greater and was limited only because election
officials had scanned only 197 ballots in their first batch.
Although Premier/Diebold knew about the problem for years, it didn't
notify the federal Election Assistance Commission or the National
Association of State Election Directors, which oversaw the testing and
certification of voting systems at the time the Premier/Diebold system
was certified, so that NASED could notify election officials around the
country about the problem.
Instead, the report says, the company sent a "vague email" to
election officials in the 11 California counties that, at the time,
were using that version of the GEMS software with a central count
optical-scan system. The e-mail, previously published here,
provided officials with a workaround for the problem. But the e-mail
never told officials why it was important for them to do the workaround
-- that is, it never explained that there was a programming flaw in the
software and that if they didn't do the workaround, the system would be
at risk of silently dropping votes.
An employee in Humboldt County received the e-mail, but never wrote
the workaround instructions into the office's election procedures and
never told Crnich about the issue before he left to go work in another
county. As a result, Humboldt was vulnerable to the flaw during the
The flaw was fixed in version 1.18.24 of the software in May 2005.
But until that occurred, Premier/Diebold continued to allow other
jurisdictions across the country to use at least five flawed versions
of the software and never explained the problem or the workaround in
its user documentation.
Diebold has said that no jurisdiction outside California used these
versions of GEMS with a central count optical-scan system and therefore
were not at risk from the flaw. California officials backed this
statement in their report.
But even when the flaw was fixed in version 1.18.24, the vendor allowed
California counties to continue to use the flawed software rather than
upgrade them to the fixed version. The company never informed state
officials about the problem with its system.
Secretary of State Debra Bowen has sponsored legislation that would require voting machine vendors to notify the state in writing
(.pdf) any time it discovers a problem with its voting system. The
vendor would have to notify the state, and any California jurisdiction
using the voting system, within five working days of discovering a flaw
in software or hardware.
The bill also requires vendors to disclose any flaws it already knows
about systems that are currently in use in the state. These reports
will then be submitted to the Election Assistance Commission so that
officials in other states will know about them as well. The bill
provides for civil penalties of $10,000 per violation against vendors
for undisclosed flaws or for making unauthorized changes to a voting
Kate Folmar, spokeswoman for the secretary of state's office, said
Bowen hopes that the bill, if passed, "could become a model for other
states for dealing with similar anomalies and problems that pop up with
their voting systems."
Comment on This Article
You must login to leave comments...
Other Visitors Comments
You must login to see comments...