Erroneous reports are circulating that a virus caused a problem in the scanners used in the NY-23 Congressional race. The reports, based on an inaccurate article published in the Gouverneur Times, are incorrect. There was no virus in the NY-23 machines. How do I know? Well, in the first place, the Dominion ImageCast scanners in question run the Linux operating system, which is nearly immune to viruses due to its inherent ability to lock out programs that lack explicit permission to run, unlike the highly vulnerable Windows operating system. Second, the State Board of Elections gave an account of the problem at their public meeting on November 10, and which I confirmed in a phone conversation with staff earlier this week. Here’s what really happened:
Let’s be clear. While no votes were lost due the ability to independently count the paper ballots, a problem did occur that affected certain machines around the state. The issue was a bug in the Dominion source code that caused the machine to hang while creating ballot images for certain vote combinations in multiple candidate elections (the ImageCast, like the other scanner used in New York, the ES&S DS200, creates digital images of each ballot which can be reviewed after the election). So if, for example, a “vote for three candidates out of five” race was voted in a certain way, the scanner would hang. This is one reason why the defect affected some, but not all machines with ballots containing this type of race, because only certain combinations of votes caused the memory problem. But here’s the thing – the problem was discovered before the election.
New York State has decent requirements for pre-election testing of machines, and they came through. The defect described above was discovered a week before the election during pre-election testing, just as it should! Indeed, it was putting the machines through their pre-election paces when the defect was discovered. The problem was determined to be in the ImageCast source code, which may not be modified without retesting and certification prior to use. But it was possible to modify the ballot configuration file to contain less ancillary text, freeing up a bit more memory and preventing the crash. The Board of Elections approved the configuration file change, the 10 counties which had races susceptible to the bug were identified, and the changes made to the files in plenty of time for the election. Everything worked the way it should have, except for one thing – the process of identifying machines which needed the fix missed some of them, so the modified configuration file was not installed, and, as expected, these machines hung.
In my opinion, it’s a Bad news-Good news-Bad news-Good news situation:
1) The bad news is that the scanner source code had a bug that Dominion should have found before they delivered machines to New York for certification testing. I spent a long time doing software engineering, and in my opinion a robust in-house test plan absolutely should have turned up a simple memory buffer overflow like this long before the code was delivered to the state, especially when New York’s ballot requirements are known and supposedly addressed by the vendor. Dominion gets an ‘F’ for blowing an easy one in their first roll-out of their new scanner.
2) The good news is that pre-election testing found the bug before the election, a reasonable fix was found and approved, and it was applied to the machines that needed it before the election – well, almost.
3) The next bit of bad news is of course, that not everyone who needed the fix had it applied. I’ve had conflicting accounts of who was responsible for determining this. I’ve heard of at least three – the State Board, the local county election officials, and Dominion. In my opinion, it’s all of the above. Here we have a serious procedural problem that needs to be addressed. When a problem is found prior to an election, what are the proper procedures for accurately determining who is affected? This is one good lesson the State Board should take from the pilot.
4) Finally, the good news – because New York votes on paper, everybody’s vote was counted. When the scanner stopped working, the ballots were removed and counted, so no votes were lost. Paper ballots, a software independent record of the vote, proved their great value in their very first outing in the Empire State. Compare this to lever machines, where counters on the back would get stuck and wouldn’t turn when the vote is cast, something that occurred with far more frequency than most New Yorkers realize. When the counters on the back of a lever machines froze, a machine bug typically not discovered until the polls close, those votes were lost forever. More than a few lever machine elections had the incorrect candidate declared the victor as a result. When the scanner freezes, everyone knows about it, the machine is removed from service, and the paper ballots of those who have voted already and of those who will vote later in the day are sure to be counted.
I’ll take paper any day.