Learning Transferable Visual Models From Natural Language Supervision

Before CLIP, a vision model learned a fixed list of labels. A model trained on ImageNet knew 1,000 categories, and adding a 1,001st meant collecting labeled photos and training again. CLIP (Contrastive Language-Image Pre-training), from Alec Radford, Jong Wook Kim and colleagues at OpenAI, replaced the label list with language. The authors gathered 400 million image and text pairs from the public internet, searched with 500,000 queries built from words common in Wikipedia, and trained two encoders, one for images and one for text, so that matching pairs land close together in a shared embedding space.

The training task is a matching game. Take a batch of N images and their N captions. Compute every image-caption similarity, an N × N grid, and train both encoders so each image scores highest with its own caption, and each caption with its own image. The real pairs sit on the diagonal; every other cell is a negative. The batch size was 32,768, so every image competed against 32,767 wrong captions per step.

Classification then needs no training at all. Write each class name into a sentence such as "A photo of a dog.", embed the sentences with the text encoder, and assign each image to the nearest one. That is what "zero-shot" means here: the classifier is built from words, and the target dataset's training labels are never used. On ImageNet the best model reached 76.2% top-1, the accuracy of the original fully supervised ResNet-50, where the previous zero-shot method managed 11.5%.

CLIP and its descendants sit inside most production image search, multimodal retrieval and visual tagging systems built since 2021. When a customer asks for "find every photo of a damaged pallet" or "tag our product catalog", a contrastive image-text model is usually the first thing to try, because it needs a list of words instead of a labeled dataset.

You can prototype a classifier in an afternoon. Write the customer's categories as short descriptions, embed them, and score a sample of their images. That gives a baseline before anyone labels anything, and it tells you which categories are easy and which need data.

The wording of the classes is part of the model. Changing "crane" to "a photo of a crane, a type of bird" changes the result. The paper got almost 5 points on ImageNet from prompt templates and ensembling alone. Treat class descriptions as configuration that is tested and versioned.

Zero-shot is a starting point, and a demo can flatter it. CLIP was near chance on some tasks, such as distances in driving photos, and weak on fine-grained ones like aircraft variants. Score it on the customer's own images before promising anything.

The same embeddings power search. Text-to-image and image-to-image retrieval use the shared space directly, so one model serves both classification and search, stored in an ordinary vector database.

It carries the web's biases. The authors found harmful associations in face classification probes, and showed that adding one class to the label list changed who was misclassified. A deployment that classifies people needs its own audit.