WeatherWearBench

A text-only benchmark that tests whether a language model gives clothing advice for the hours a person spends outside. It has 765 tasks built from 72 real hourly forecasts in 16 cities. Each task gives the user's outdoor plans, an optional log of how they felt on earlier days, and a closet of about 30 items.

Read the forecast and the plan, then return JSON: the items to wear or take, the feels-like temperature it dressed for, and flags for rain and sun. Items are named by opaque ids, so the model has to pick from the user's own closet.

By code, with fixed rules for warmth, removable layers, season errors and protection against rain and sun. No language model or person judges the answers.

A person who asks at 07:00 about a lunch walk needs clothes for noon, not for the cold morning. A rule that dresses for the feels-like temperature at the time of the request makes gross errors on 20.3% of tasks; the five language models we tested make them on 0.2–6.3%.

On single answers, partly. One fixed outfit per closet that states the middle of the outdoor range and the right flags reaches a strict pass rate of 74.3%, against 77.7% for the best model, on days not used to choose the outfit. That is why the main results come from paired checks.

Two tasks that differ in one thing, for example warmer outdoor hours with the same 24-hour mean, or a user who often feels cold. The main measure looks only at pairs in which the old outfit no longer fits the new weather. With the same 24-hour mean, the models' new answer passes and fits the new hours in 58.5–93.5% of these pairs, against 5.7–21.8% for another answer to the old prompt.

Five low-priced language models (Claude Haiku 5.5, GPT-6 Luna, Gemma 4 31B, Gemini 3.1 Flash Lite and GPT-5 nano), one decision model asked through an adapter (Jev 1.13), and 8 rule baselines: 18,090 calls through OpenRouter, for USD 7.10.

Its clothes follow the outdoor hours: in the same-mean check its outfit moves the expected way 56.9 points more often than in two repeats of one prompt, against 54.9 for the best model. Its stated temperature is within 1.5 °C of the outdoor range in only 53.5% of answers, and it sets the sun flag on most low-UV days, so its strict pass rate is 37.8%. Code asks it the questions and writes its answer, so it is reported apart from the language models.

No. The benchmark checks the total warmth and the protection of what the model lists against its rules. It does not measure real comfort, and the rules have not been checked with people yet.

Yes, once the release is out. The code, the data, the prompts and the raw responses will be published on this page, with one command that rebuilds every table of the paper. Any model on OpenRouter can be added, and the runner skips calls that already have an answer, so adding a model does not repeat the others.