These Algorithms Were Meant to Improve the Human Condition: They Failed

High-tech computer systems can reflect the misjudgments of the people who designed them—but this idea gets obscured because the systems’ algorithms don’t have any agency.

In order to understand what’s wrong with governments relying too heavily on data and algorithms, let’s go back to the forests of 18th-century Prussia.

As James Scott wrote in his book Seeing Like a State, governments in the former German state primarily viewed forests as a way to extract revenue. So when they did surveys of forests, all they counted was the number of trees that could be cut down and turned into commercial wood products.

With the countable target of salable wood in hand, the state reconfigured forests in the region to optimize for that metric. Underbrush was cleared. Profitable monocultures of identical trees were planted simultaneously—often in straight, orderly lines that made them ...