A camera and its software labeled Bill Swearingen as a person. Then it suddenly wasn’t sure — all because of some weird pattern that he held up to disguise himself.
I watched it happen from my seat at annual hacker convention Defcon. Swearingen, a longtime cybersecurity professional and founder of the Kansas City security community SecKC, stood onstage in front of a live camera feed as a person-detection system analyzed him. On the giant screen behind him, the software’s confidence score cleared 0.75, the threshold it needed to declare that, yes, there was a human being in the frame.
Then Swearingen raised a flat panel covered in a bizarre black-and-white pattern. The score started falling. It slipped below the threshold, eventually landing at 0.21.
“No person detected,” the screen announced in bright green letters.
It felt like a low-budget magic trick. Swearingen was still standing there, plainly visible to everyone in the room. The software was still receiving the camera image, but it no longer detected a person above the configured confidence threshold.
Swearingen has spent the past year searching for patterns that can confuse the computer-vision systems used to identify people. His project is called noRecognition, and its end goal is to create clothing that makes the wearer harder for AI surveillance systems to detect. It’s a fascinating project, but it’s still a work in progress.
The camera wasn’t trying to identify him
We tend to call this kind of technology “facial recognition,” but surveillance systems can involve several separate layers of AI-based detection.
A person detector asks whether a human body is in the frame. A face detector finds and isolates a face. Facial recognition then compares that face with a database and asks whether it knows who the person is.
The demonstration I saw targeted the first step. The system didn’t mistake Swearingen for somebody else or decide the room was empty. It simply stopped reporting a person detection above the threshold set for the demonstration.
If you break the first link, the rest of the surveillance chain may never get started. If a camera fails to detect a person, it may never crop out that person’s face, send it to an identity database, then track them across a series of images.
An AI detector doesn’t “see” a person in the way we do. It generates thousands of guesses about what might be in an image, assigns a score to them and discards anything that fails to clear a chosen confidence threshold.
Swearingen was still there. The detector’s math just wasn’t sure he was a person.
How the patterns are made
Swearingen didn’t sit down and draw the patterns himself. He built a program to create them.
Security researchers normally use programs called fuzzers to bombard software with strange inputs in an attempt to break it. Swearingen’s fuzzer does something similar with computer vision. It creates a pattern, digitally places it on an image of a computer-generated person and then shows the altered image to several AI models.
If a pattern causes a major change, like making the person’s detection box disappear or sharply lowering the model’s confidence score, then the fuzzer flags it for more testing.
The most successful designs are altered and combined to produce new patterns, much like selectively breeding specific traits in animals. Swearingen’s software repeats that process automatically, allowing it to work through far more possibilities than a person could design and test by hand.
He currently tests each pattern against 11 models: five that detect people, four that find faces and two that try to recognize whose face it is. In many cases, he tested a public model that works similarly to proprietary systems, not the actual software being used by companies in the wild. Most are publicly available models that researchers can run themselves.
Swearingen said the project had run 31.7 million tests as of June. About 534,600 triggered one of his system’s anomaly rules. That could mean a detector found fewer people or faces than it did in the original image, invented extra ones, produced a much lower confidence score or returned the wrong identity. It doesn’t mean he found 534,600 patterns that could hide someone from a camera.
Of those results, 85 met the project’s definition of “extreme,” which means the pattern defeated at least one person detector and at least one face detector in the same test.
The tests also showed that covering more of the body isn’t always more effective. Torso patterns had the greatest effect on person detectors, while patterns closer to the head mattered more to systems looking for faces. In one comparison, Swearingen said a small collar pattern worked better than a much larger print across the torso.
Even the winning patterns can lose
Some of Swearingen’s results looked promising until he tested the designs on different people.
One pattern lowered the detector’s confidence for all four people whose images were used while it was being developed. But when Swearingen applied that same design to images of eight new people, it didn’t work once.
Counted together, the pattern had worked on four of 12 people. Swearingen withdrew the resulting 33% performance claim because every success involved people whose images had been used to develop the pattern. It hadn’t worked on anyone new.
Another pattern, designed to fool a different detector, worked on all eight new people. That contrast shows how much the results can change between detectors — what works with one person or detection model may fail with another.
Even a successful pattern may have a short shelf life. A camera company could retrain its software to recognize it. Swearingen compared the patterns to software “zero-days” vulnerabilities — they take advantage of a weakness that may disappear once the system is updated.
The shirt doesn’t exist yet, and the tech isn’t field-tested
On one of Swearingen’s slides, in large type, were the words “Still unproven: The worn garment.”
During the demo, Swearingen held the pattern in front of his body on a rigid panel. It wasn’t printed on clothing, which would be a much harder test. Ink on a flat board stays exactly where the software expects it to be. Fabric drapes, folds and stretches. Sunlight, shadows, wrinkles, viewing angles and distance all change what reaches the camera.
Swearingen said most of his 31.7 million tests were digital, with patterns pasted onto images and scored by computer-vision models. The panel demonstration was a limited physical test using a model he said came from a fielded Flock Safety unit. Until a pattern works while printed on fabric, worn by a moving person and viewed through deployed hardware, he isn’t claiming that the clothing itself works.
The noRecognition website is promoting a Kickstarter-backed run of T-shirts, hoodies and neckwear, with each pattern scored against 11 models before it ships. Swearingen said the money would help pay for test fabrics, a dye-sublimation printer and additional cameras so he can see whether the patterns still work once they’re printed and worn.
So could a pattern on your clothing fool facial recognition?
Maybe. But the real test begins when he prints it on clothing and steps back in front of the camera.
See full bio
Read the full article here
