THE LIMITATIONS OF OPAQUE LEARNING MACHINES Judea Pearl Judea Pearl is a professor of computer science and director of the Cognitive Systems Laboratory at UCLA. His most recent book, co-authored with Dana Mackenzie, is The Book of Why: The New Science of Cause and Effect. As a former physicist, I was extremely interested in cybernetics. Though it did not utilize the full power of Turing Machines, it was highly transparent, perhaps because it was founded on classical control theory and information theory. We are losing this transparency now, with the deep-learning style of machine learning. It is fundamentally a curve-fitting exercise that adjusts weights in intermediate layers of a long input-output chain. I find many users who say that it “works well and we don’t know why.” Once you unleash it on large data, deep learning has its own dynamics, it does its own repair and its own optimization, and it gives you the right results most of the time. But when it doesn’t, you don’t have a clue about what went wrong and what should be fixed. In particular, you do not know if the fault is in the program, in the method, or because things have changed in the environment. We should be aiming at a different kind of transparency. Some argue that transparency is not really needed. We don’t understand the neural architecture of the human brain, yet it runs well, so we forgive our meager understanding and use human helpers to great advantage. In the same way, they argue, why not unleash deep-learning systems and create intelligence without understanding how they work? I buy this argument to some extent. I personally don’t like opacity, so I won’t spend my time on deep learning, but I know that it has a place in the makeup of intelligence. I know that non-transparent systems can do marvelous jobs, and our brain is proof of that marvel. But this argument has its limitation. The reason we can forgive our meager understanding of how human brains work is because our brains work the s