Interactive
Where should “Not sure” begin?
Drag two lines and see how my wildlife app decides when to say “Not sure.” Every number comes from its real test photos.
My wildlife app looks at a photo and names the animal, from a list of 40 local species. It also gives the photo a score for how closely it matches something on that list. Higher means a closer match.
The two mistakes
- False positive: the app gives a name when it should have said “Not sure.” This usually happens when the photo shows something that isn’t on the list, and the app names the closest match anyway.
- False negative: the app says “Not sure” about a photo it would have named correctly.
The goal is as few of both as possible. They pull against each other: making the app more careful cuts false positives but adds false negatives, and the other way around.
Reading the chart
Each dot is one test photo, placed by its score. The top row is 632 photos of animals on the list. The bottom row is 131 photos of things that aren’t, so every one of those should end in “Not sure.” A hollow dot means the photo ended in “Not sure.” Hover over or tap a dot to see what it is.
The lower line
Every photo that scores below this line gets “Not sure.”
- Move it right: more off-list photos fall below it, so fewer false positives. But more on-list photos fall below it too, so more false negatives.
- Move it left: the opposite. Fewer false negatives, more false positives.
The upper line
Every photo that scores at or above this line gets its answer with no further check. Photos between the two lines go to a second AI. If it agrees with the answer, the answer stands. If it doesn’t, the photo gets “Not sure.”
- Move it left: fewer photos get checked. Off-list photos in that range no longer get caught, so more false positives.
- Move it right: more photos get checked. Very few off-list photos score that high, so there is little left to catch. The second AI sometimes says no to a correct answer, so this mostly adds false negatives.
Below the chart, “Off-list photos caught” tracks false positives (the ones that slipped through), and “Listed photos sent to ‘Not sure’” tracks false negatives.
- Confident answers
- Listed photos sent to “Not sure”
- Off-list photos caught
- Photos the second AI checks
What the app’s lines do
- The lower line alone catches 88.5% of off-list photos, with no false negatives. But 15 off-list photos still slip through as false positives. They score between about 68 and 72, the same range as many correct answers, so no single line can separate them.
- Adding the check between the lines catches 93.1% of off-list photos, leaving 9 false positives. The cost is 12 on-list photos (1.9%) sent to “Not sure.”
- Moving the lower line up to 70 instead, with no check, catches 95.4%. But it sends 41 on-list photos (6.5%) to “Not sure.”
- Checking everything above the lower line catches no extra off-list photos and adds 27 more false negatives.
Checking only the overlap, where the score can’t tell the two rows apart, removes most false positives for the fewest false negatives.
How the app’s lines were chosen
Not with these photos. The lines were set using a separate group of 289 photos. That keeps the numbers here honest: they show how the app does on photos it has never seen. Setting the lines with the same photos you test on makes the results look better than they really are.
The test photos come from iNaturalist. Nothing runs when you move a line: the page replays saved results. For more on the second AI, see Giving my smartest AI a veto, not a vote.