How Is ChatGPT's Behavior Changing over Time?

When this paper was written, OpenAI's API offered two snapshots of each model: GPT-4 and GPT-3.5 as of March 2023 and as of June 2023. Products calling "gpt-4" were moved from one to the other. The authors asked a practical question: does the same prompt still produce the same behavior?

They ran both snapshots, at temperature 0.1 with the default system prompt, on eight tasks: deciding whether a number is prime (1,000 questions), counting happy numbers in an interval (500), answering sensitive questions (100), answering opinion survey questions (1,506), running LangChain's ReAct agent on HotpotQA multi-hop questions (7,405), writing code for recent LeetCode easy problems (50), the USMLE medical exam (340), and ARC visual puzzles given as number grids (467). For each they tracked a main score, response length, and the mismatch rate: how often the two snapshots gave different final answers.

Behavior moved on almost every task. GPT-4's prime accuracy fell from 84.0% to 51.1% and its happy-number accuracy from 83.6% to 35.2%, while GPT-3.5 improved on both. GPT-4 answered 21% of sensitive questions in March and 5% in June, and 97.6% of opinion questions in March and about 22% in June. The share of GPT-4's code that ran as returned fell from 52% to 10%, because June wrapped it in markdown fences. GPT-4 got better at the LangChain agent task, from 1.2% to 37.8% exact match.

The authors trace much of this to instruction following. In June, GPT-4 skipped the reasoning steps it was asked for, ignored formatting requests, and on a set of simple single instructions dropped from 99.5% to 0.5% on putting a yes or no answer in square brackets.

Pin the model version and treat changes as releases. The paper's core lesson is that a model name is not a contract. Use dated snapshot identifiers where the provider offers them, and treat a move to a new snapshot like a dependency upgrade: run the regression suite before switching.

Keep a regression suite small enough to run on every change. The cheapest drift alarm here is the mismatch rate: run a fixed set of prompts on old and new versions and count changed answers. GPT-4's USMLE accuracy moved 4.5 points, but 12.2% of its answers changed, which is the better early signal.

Compare every score with a constant answer. On a balanced yes-or-no task, a model that always says "no" scores 50%. June GPT-4's 51.1% on primes is that model. Before reading a drop as lost capability, check whether the new version has collapsed onto one answer.

Parse defensively. The code-generation drop came from markdown fences around otherwise correct code, and the LangChain failure from output the agent's parser rejected. Strip fences, validate structure, and alert on parse failures separately from wrong answers.

Size the suite to the change you need to detect. A drop from 84% to 51% shows up with about 30 items. A drop from 80% to 75% needs about 1,100 per snapshot with a simple two-sample test. Fifty code problems give a 95% interval of about 14 points.