Peering inside an AI’s brain will help us trust its decisions
Oi, AI – what do you think you’re looking at? Understanding why machine learning algorithms can be tricked into seeing things that aren’t there is becoming more important with the advent of things like driverless cars. Now we can glimpse inside the mind of a machine thanks to a test that reveals which parts of an image an AI is looking at.
Artificial intelligence doesn’t make decisions in the same way that humans do. Even the best image recognition algorithms can be tricked into seeing a robin or cheetah in images that are just white noise, for example.
It’s a big problem, says Chris Grimm at Brown University in Providence, Rhode Island. If we don’t understand why these systems make silly mistakes, we should think twice about trusting them with our lives in things like driverless cars, he says.
So Grimm and his colleagues created a system that analyses an AI to show which part of an image it is focusing on when it decides what the image is depicting. Similarly, for a document-sorting algorithm, the system highlights which words the algorithm used to decide which category a particular document should belong to.
Peek inside
It’s really useful to be able to look at an AI and find out how it’s learning, says Dumitru Erhan, a researcher at Google. Grimm’s tool provides a handy way for a human to double-check that an algorithm is coming up with the right answer for the right reasons, he says.
To create his attention-mapping tool, Grimm wrapped the second AI around the one he wanted to test. This “wrapper AI” replaced part of an image with white noise to see if that made a difference to the original software’s decision.
If replacing part of an image changed the decision, then that area of the image was likely to be an important area for decision-making. The same applied to words. If changing a word in a document makes an AI classify a document differently, it suggests that word was key to the AI’s decision.
Grimm tested his technique on an AI trained to sort images into one of 10 categories, including planes, birds, deer, and horses. His system mapped where the AI was looking when it made its categorisation. The results suggested that the AI had taught itself to break down objects into different elements and then search for each of those elements in an image to confirm its decision.
Read more at New Scientist