California Secretary of State - Voting Systems Review

About Elections Division

Election Results and Election Dates

Register to Vote

Voter Registration Statistics

Voter Information

Initiatives

Voter Education

Political Parties

Candidate Information

Resources

Site Map

Top-to-Bottom Review

Secretary of State Debra Bowen conducted a top-to-bottom review in 2007 of many of the voting systems certified for use in California. The review, led by computer scientists from the University of California, was designed to restore the public's confidence in the integrity of the electoral process and to ensure that California voters cast their ballots on machines that are secure, accurate, reliable, and accessible. Following the top-to-bottom review, on August 3, 2007, Secretary Bowen strengthened the security requirements and use conditions for certain systems. The following documents detail Secretary Bowen's decisions and the independent experts' findings in the review.

Withdrawal of Approval and Reapproval Decisions Issued by Secretary of State Debra Bowen Premier Election Solutions (formerly Diebold Election Systems)

Withdrawal of Approval/Conditional Reapproval - October 25, 2007 Revision (.pdf, 9,903KB)
Withdrawal of Approval/Conditional Reapproval - October 25, 2007 Redline Version (.pdf, 64KB)
Post-Election Manual Tally Requirements - October 25, 2007 (.pdf, 56KB)
Post-Election Manual Tally Suggested Instructions - January 30, 2008 (.pdf, 1.3MB)

Hart InterCivic

Withdrawal of Approval/Conditional Reapproval - December 6, 2007 Revision (.pdf, 5,886KB)
Withdrawal of Approval/Conditional Reapproval - December 6, 2007 Red Line Version (.pdf, 5,886KB)
Post-Election Manual Tally Requirements - October 25, 2007 (.pdf, 56KB)
Post-Election Manual Tally Suggested Instructions - January 30, 2008 (.pdf, 1.3MB)

Sequoia Voting Systems

Withdrawal of Approval/Conditional Reapproval - October 25, 2007 Revision (.pdf, 10,356KB)
Withdrawal of Approval/Conditional Reapproval - October 25, 2007 Redline Version (.pdf, 67KB)
Post-Election Manual Tally Requirements - October 25, 2007 (.pdf, 56KB)
Post-Election Manual Tally Suggested Instructions - January 30, 2008 (.pdf, 1.3MB)
Letter to Riverside County - October 2, 2007 (.pdf, 114KB)

Election Systems and Software

ES&S Inkavote Plus Conditional Approval - January 2, 2008 (.pdf, 12.4MB)
InkaVote Plus Source Code Report - October 2, 2007 (.pdf, 172KB)
InkaVote Plus Red Team Report - October 2, 2007 (.pdf, 123KB)
InkaVote Plus Voting System Accessibility Review - January 2, 2008 (.pdf, 697KB)
InkaVote Plus - November 26, 2007 Public Hearing Notice (.pdf, 48KB)
InkaVote Plus - November 26, 2007 Hearing Transcript (.pdf, 169KB)
Rescission and Withdrawal of Approval - August 3, 2007 (.pdf, 303KB)
Post-Election Manual Tally Requirements - October 25, 2007 (.pdf, 56KB)
Post-Election Manual Tally Suggested Instructions - January 30, 2008 (.pdf, 1.3MB)

UC Final Reports

Following are the final reports from the University of California scientists detailing their findings from the top-to-bottom review. The red, source code, and documentation review team reports are separated by voting system. The accessibility report contains findings on all of the voting systems that were reviewed.

UC Source Code Team Reports

UC Principal Investigator David Wagner's Statement on Protection of Security-Sensitive Information (.pdf, 13.9KB)
Premier Election Solutions (formerly Diebold) (.pdf, 561KB)
Hart InterCivic (.pdf, 573KB)
Sequoia Voting Systems (.pdf, 831KB)

UC Red Team Reports

Overview by UC Principal Investigator Matt Bishop (.pdf, 303KB)
Red Team Test Protocol (.pdf, 134KB)
Premier Election Solutions (formerly Diebold) (.pdf, 498KB)
Hart InterCivic (.pdf, 376KB)
Sequoia Voting Systems (.pdf, 108KB)

UC Documentation Review Reports

Premier Election Solutions, Inc. (formerly Diebold) (.pdf, 402KB)
Hart InterCivic (.pdf, 1,028KB)
Sequoia Voting Systems (.pdf, 368KB)

UC Accessibility Report

Accessibility Report (.pdf, 1.07MB)
Accessibility Report (HTML)

Public Hearing

Webcast of the July 30, 2007, Public Hearing Transcript (.pdf, 48KB)
Public Notice (.pdf, 58KB)
Agenda (.pdf, 48KB)
Written Public Comment and Testimony

Press Releases

Video of Secretary of State Bowen's August 3, 2007, Announcement
August 3, 2007 (.pdf, 152KB)
July 27, 2007 (.pdf, 152KB)
June 21, 2007 (.pdf, 138KB)
May 10, 2007 (.pdf, 165KB)
May 9, 2007 (.pdf, 144KB)

Testing Security Plans

Secretary of State Testing Security Plan (.pdf, 51KB)
Source Code Review Security Plan (.pdf, 78KB)
Red Team Security Plan (.pdf, 128KB)
Documentation Review Security Plan (.pdf, 68KB)

Additional Information

Following are a number of documents related to the review, including a "Frequently Asked Questions" document to help people understand how the review was designed.

One-Page Summary - May 9, 2007 (.pdf, 148KB)
Frequently Asked Questions - Revised August 15, 2007 (.pdf, 242KB)
Frequently Asked Questions - Revised July 2, 2007 (.pdf 232KB)
Frequently Asked Questions - Revised June 13, 2007 (.pdf, 232KB)
Frequently Asked Questions - Revised May 11, 2007 (.pdf, 257KB)
Frequently Asked Questions - Revised May 9, 2007 (.pdf, 248KB)
Contract Between Secretary of State and University of California Regents (.pdf, 186KB)

Public Observation Guidelines (.pdf, 56KB)

Professional Qualifications of the Top-to-Bottom Review Teams

Documentation Review Team Members
The reviewers were responsible for analyzing voting system security, accessibility, usability, reliability, accuracy and protection of ballot secrecy based on relevant documentation. The reviewers had access to documents such as reports from Independent Testing Authorities (ITAs), reports and data from state certification testing, and documentation related to how the systems are designed to be used in an actual election. The reviewers were split into three teams of two or three members to review each voting system subject to the top-to-bottom review.

David Wagner (.pdf, 455KB) Principal Investigator
Aaron Burstein (.pdf, 57KB)
Nathan Good (.pdf, 311KB)
Joseph Hall (.pdf, 229KB)
Candice Hoke (.pdf, 404KB)
Dave Kettyle (.pdf, 167KB)
Deirdre Mulligan (.pdf, 460KB)
Laura Quilter (.pdf, 55KB)
Tom Ryan (.pdf, 125KB)

Source Code Review Team Members
Source code is the computer language that effectively controls how electronic voting systems operate. The source code reviewers were split into three teams of six or seven members to review each voting system subject to the top-to-bottom review. Here is a more detailed description of source code review.

David Wagner (.pdf, 455KB) Principal Investigator

Matt Blaze (.pdf, 72KB)
Joseph Calandrino (.pdf, 62KB)
Arel Cordero (.pdf, 52KB)
Sophie Engle (.pdf, 145KB)
Ariel Feldman (.pdf, 47KB)
J. Alex Halderman (.pdf, 60KB)
Srinivas Inguva (.pdf, 174KB)
Chris Karlof (.pdf, 70KB)
Eric Rescorla (.pdf, 78KB)
Naveen Sastry (.pdf, 138KB)
Hovav Shacham (.pdf, 164KB)
Micah Sherr (.pdf, 108KB)
Till Stegers (.pdf, 62KB)
Dan Wallach (.pdf, 634KB)
Ka-Ping Yee (.pdf, 173KB)
Harlan Yu (.pdf, 54KB)
William Zeller (.pdf, 65KB)

Red Team Members
Red team members were responsible for testing the functions and performance of the voting systems and identifying security or accuracy vulnerabilities . The red team members were split into three teams to review each voting system subject to the top-to-bottom review. Here is an explanation of red team testing in general and the preliminary protocols that were used in the red team testing.

Matthew Bishop (.pdf, 934KB) Principal Investigator

Robert Abbott (.pdf 203KB)
Elliot Proebstel (.pdf 124KB)
Sujeet Shenoi (.pdf 37KB)
Davide Balzarotti (.pdf 66KB)
Greg Banks (.pdf 12KB)
Marco Cova (.pdf 61KB)
Mark Davis (.pdf 33KB)
Viktoria Felmetsger (.pdf 59KB)
Richard Kemmerer (.pdf 49KB)
William Robertson (.pdf 69KB)
Jacob Stauffer (.pdf 32KB)
Fredrik Valeur (.pdf 74KB)
Giovanni Vigna (.pdf 111KB)

Accessibility Team Members
The accessibility reviewers examined the voting systems subject to the top-to-bottom review to determine whether they were accessible to voters with disabilities and voters with alternative language needs.

Noel Runyan (.pdf 13KB)
James Tobias (.pdf 20KB)

Draft Criteria

The draft criteria for the top-to-bottom review of voting systems certified for use in California were released for public comment on March 22, 2007. The final criteria for the review can be found in the Contract Between Secretary of State and University of California Regents.

Public Comment

Counties

California Association of Clerks and Election Officials (CACEO)
California State Association of Counties (CSAC)
Butte
Contra Costa
Kern
Los Angeles
Madera
Mariposa
Mendocino
San Joaquin
San Luis Obispo
San Mateo
Santa Clara
Santa Cruz
Shasta
Solano
Sonoma
Yolo
Yuba

Voting System Vendors

Avante
Premier Election Solutions (formerly Diebold)
Equalivote and VotePAD
ES&S
Hart InterCivic
Sequoia Voting Systems

Organizations

Black Box Voting
California Council of the Blind
California Foundation for Independent Living Centers
Coloradoans for Voting Integrity
Open Voting Consortium
Protection and Advocacy, Inc.
Secure Accurate Elections
Townsend and Associates
VerifiedVoting.org
Wellstone Democratic Renewal Club

Individuals

Comments from Individuals
All personal information has been removed from comments submitted by individuals.