AI versus a giraffe with no spots

aiweirdness:

AI versus a giraffe with no spots

On July 31, 2023, a giraffe with no spots was born at Brights Zoo in Tennessee.

Image recognition algorithms are trained on a variety of images from around the internet, and/or on a few standard image datasets. But there likely haven’t been any spotless giraffes in their training data, since the last one to be born was probably in 1972 in Tokyo. How do they do when faced with photos of the spotless giraffe?

Here’s Multi-Modal In-Context Learning:

Image is of a brown baby giraffe with no spots. Series of questions and answers: Describe this image.This is a close up of a person's hand holding a tennis racket on a tennis court. What color is the giraffe?brown and white or tan with brown spots or tan with white spots. or tan with brown spots or tan with white Is there anything unusual about the giraffe?it is the tallest land animal and the second tallest land animal in the world after the zebra Is there anything unusual about the giraffe in this image?Answer: the giraffe in this image is standing on its hind legs. or it is standing on its hind legsALT

And InstructBLIP, which was more eloquent but also added lots of spurious detail.

Image is of a plain brown baby giraffe with no spots, standing on gravel in front of a zoo fence. Conversation with InstructBLIP: Me: Describe this image, noting specific details. Bot: The image features a young giraffe standing at the edge of a fenced-in area near a gravel surface. The giraffe appears to be in a playful and curious mood as it stares at something outside the fence. There are two chairs placed in the background, on opposite sides of the image. Additionally, a person stands in a grey shirt, located somewhat further into the frame, near the right side of the giraffe. A small pink umbrella lies on the ground, partially hidden between the giraffe and the chair. Me: Is there anything unusual about the giraffe? Bot: The giraffe is unusually large, as it is only a baby and yet appears to be standing in a fenced area. This is a rare sight and an opportunity to observe a young giraffe up close. It is also unusual that such a wild animal like a giraffe is alive and on display in a controlled environment, behind a fence. Observing the giraffe in this setting may allow for better understanding of the species' behavior and development.ALT

More examples at AiWeirdness.com

Are these crummy image recognition models? Not unusually so. As far as I can tell with a brief poke around, MMICL and InstructBLIP are modern models (as of Aug 2023), fairly high up on the leaderboards of models answering questions about images. Their demonstration pages (and InstructBLIP’s paper) are full of examples of the models providing complete and sensible-looking answers about images.

Then why are they so bad at Giraffe With No Spots?

I can think of three main factors here:

  1. AI does best on images it’s seen before. We know AI is good at memorizing stuff; it might even be that some of the images in the examples and benchmarks are in the training datasets these algorithms used. Giraffe With No Spots may be especially difficult not only because the giraffe is unusual, but because it’s new to the internet.
  2. AI tends to sand away the unusual. It’s trained to answer with the most likely answer to your question, which is not necessarily the most correct answer.
  3. The papers and demonstration sites are showcasing their best work.Whereas I am zeroing in on their worst work, because it’s entertaining and because it’s a cautionary tale about putting too much faith in AI image recognition.