Things I Know I Know

From an article in the Chicago Sun-Times about libraries and the PATRIOT ACT:

A Department of Justice spokesman said actions by libraries to warn patrons or to regularly discard certain records are legal and don't violate the Patriot Act. But such steps are "an unfortunate waste of their time," Justice Department spokesman Mark Corallo said.

The Patriot Act is used only to gain information about terrorists or foreign spies, so libraries don't need to take steps to protect their patrons' privacy, Corallo insisted.

From Crypto-Gram Newsletter and Bruce Schneier:

Assume a simple database -- name and a single code indicating "innocent" or "guilty." When a policeman encounters someone, he looks that person up in the database, and then arrests him if the database says "guilty."

Example 1: Assume the database is 100% accurate. If that is the case, there won't be any false arrests because of bad data. It works perfectly.

Example 2: Assume a 0.0001% error rate: one error in a million. (An error is defined as a person having an "innocent" code when he is guilty, or a "guilty" code when he is innocent.) Furthermore, assume that one in 10,000 people are guilty. In this case, for every 100 guilty people the database correctly identifies it will mistakenly identify one innocent person as guilty (because of an error). And the number of guilty people erroneously listed as innocent is tiny: one in a million.

Example 3: Assume a 1% error rate -- one in a hundred -- and the same one in 10,000 ratio of guilty people. The results are very different. For every 100 guilty people the database correctly identifies, it will mistakenly identify 10,000 innocent people as guilty. The number of guilty people erroneously listed as innocent is larger, but still very small: one in 100.

The differences between examples 2 and 3 are striking. In example 2, one person is erroneously arrested for every 100 people correctly arrested. In example 3, one person is correctly arrested for every 100 people erroneously arrested. The increase in error rate makes the database all but useless as a system for figuring out how to arrest. And this is despite the fact that, in both cases, almost no guilty people get away because of a database error.

The reason for this phenomenon is that the number of guilty people is a very small percentage of the population. If one in ten people were guilty, then a 0.0001% error rate would mistakenly arrest one innocent for every 100,000 guilty, and a 1% error rate would arrest approximately one innocent for every guilty. And if the number of guilty people is even less than one in ten thousand, then the problem of arresting innocents magnifies even more as the database has more errors.

...via Shifted Librarian and BoingBoing