The image http://www.votetrustusa.org/images/votetrust-small2.jpg cannot be displayed, because it contains errors.

 

   
National Issues

Voting Systems Batch Test Results – Reliability PDF  | Print |  Email
By John Gideon and Howard Stanislevic   
March 15, 2006

In the recently released paper, "DRE Reliability: Failure by Design?" by Howard Stanislevic, we learn that the 2002 (and 1990) voting systems standards require all voting systems to have a reliability, or Mean Time Between Failures (MTBF), of 163 hours, or a 9.2% failure rate in a 15-hour election day. We also learn that this is a woefully inadequate standard especially when compared to everyday items like an incandescent light bulb, which has an MTBF of 1,000 hours, a standard PC, which has an MTBF of 30,000 hours, and even New York City’s aged mechanical lever voting machines.

With this in mind we looked at the results from the recently completed "batch testing" of voting systems manufactured by Diebold, Hart Intercivic, and Sequoia Voting Systems, and put the information from those tests, provided by the California Secretary of State’s office, into the MTBF formulae.


In order to provide results as fairly and as accurately as possible, we had to make some decisions based on the data in the volume test reports and protocols. First, we only used errors that were a result of machine failures and not human errors. The FEC/EAC standard says that a failure is only a failure if it cannot be fixed within 10 seconds. Paper jams in optical scan machines are difficult to evaluate so we made the assumption that the paper jams reported were longer than 10 seconds duration.

Other assumptions were also necessary. We assumed a 15-hour election day. This is based on pre-election testing, Logic and Accuracy Testing (L&A Test), poll opening, election day, and poll closing and is consistent with the voting system standards. We also used a hypothetical county in our formulae with 1,000 machines, and instead of just calculating MTBF, we also calculated the number of failures per thousand ballots cast. The reason for using the latter metric is because the volume tests used 20,000 ballots to test only 50 optical scan voting machines, but only 11,000 ballots to test 100 Direct Recording Electronic (DRE) voting machines.

We also assumed that all testing was conducted over a 6 hour time span, per the California DRE volume testing protocol, except for the Diebold TSx batch test from July, 2005, which the report states was done in 5.33 hours.

It should also be noted that, according to the Sequoia test reports, ballot “fatigue” was probably a factor in some of the optical-scan failures, because the same ballots were scanned at least seven times each during the test (twice to verify the test decks, and five more times to perform the scanner tests). Employing only 10 test decks to test 50 scanners calls into question the accuracy of the optical-scan reliability measurement.

Here are the results of the batch tests when viewed from the perspective of the required reliability standards and our other metrics:

Diebold Accuvote TSX/AVPM July 2005 Test
(34 errors related to the machines)
MTBF = 15 hours
3.2 failures per 1,000 ballots cast
All machines expected to fail in an election

Diebold Accuvote TSX/AVPM Sept. 2005 Test
(10 errors related to the machines)
MTBF = 60 hours
0.9 failures per 1,000 ballots cast
250 machines per thousand expected to fail in an election

Hart eScan -
(59 errors related to the machines)
MTBF = 5.1 hours
3 failures per 1,000 ballots cast
All machines expected to fail in an election

Hart eSlate -
(24 errors related to the machines)
MTBF = 25 hours
2.2 failures per 1,000 ballots cast
600 machines per thousand expected to fail in an election

Sequoia Edge -
(6 errors related to the machines)
MTBF = 100 hours
0.5 failures per 1,000 ballots cast
150 machines per thousand expected to fail in an election

Sequoia Edge II -
(2 errors related to the machines)
MTBF = 300 hours
0.2 failures per 1,000 ballots cast
50 machines per thousand expected to fail in an election

Sequoia Insight (OS) -
(29 errors related to the machines)
MTBF = 10.3 hours
1.5 failures per 1,000 ballots cast
All machines expected to fail in an election

Sequoia Insight Plus (OS) -
(28 errors related to the machines)
MTBF = 10.7 hours
1.4 failures per 1,000 ballots cast
All machines expected to fail in an election

These results are stunning. Even the best of the DRE systems tested is only 30% as reliable as an incandescent light bulb and this is the only machine to meet the federal Reliability standard. Also stunning is that all of these systems except the Diebold TSx tested in July, 2005 and the Hart eScan were recommended for certification by the Staff of the Secretary of State Office of Voting Systems Technology Assessment. Problems with the Diebold TSx were corrected, according to Diebold and the still unacceptable MTBF of 60 hours is a result of that correction.

While this study shows that none of the tested voting systems are desirable for reliability reasons, it also shows that some Direct Recording Electronic machines may be more reliable than Optical-scan voting systems if the scanners are not tested with pristine ballots. However, we should not forget that every failure of a DRE may result in disenfranchised voters or lost and irretrievable votes. A like failure of an optical-scan voting system only means that the paper ballots may have to be scanned later or hand counted.

Eddie Ajamian contributed to this article.
Comment on This Article
You must login to leave comments...
Other Visitors Comments
You must login to see comments...
< Prev   Next >
National Pages
Federal Government
Federal Legislation
Help America Vote Act (HAVA)
Election Assistance Commission (EAC)
Federal Election Commission
Department of Justice - Voting Section
Non-Government Institutions
NASS
NASED
Independent Testing Authority
The Election Center
Carter Baker Commission
Topics
General
Voting System Standards
Electoral College
Accessibility
Open Source Voting System Software
Proposed Legislation
Voting Rights
Campaign Finance
Overseas/Military Voting
Canada
Electronic Verification
: mosShowVIMenu( $params ); break; } ?>