Code generation leans toward one language — and a failure mode where models fabricate the reason for choosing it
Asked to generate project-level code, models over-select one language. This paper builds a benchmark for that bias and finds choices driven by ease rather than requirements, and in some cases supported by reasons the model invents.
Paper overview (our summary)
- Field (arXiv category)cs.SE（+1）
- AuthorsLukas Twist, Twm Stone, Helen Yannakoudakis, et al. (4)
- Submitted2026-08-06
- arXiv ID2608.06041v1
Key points
- The paper presents a benchmark measuring preference for one programming language in project-level code generation.
- It covers 28 projects across seven areas where that language is often a poor default, evaluating 25 models.
- The language remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models lean harder.
- Across 9,826 reasoning traces, most choices were automatic or driven by ease rather than project requirements.
- Models were found to fabricate contextual support for the choice, a failure the authors call phantom evidence.
1Where a default is a poor fit
Asked to write code, a model picks a language. In many settings the pick is sound; not always. This paper assembles 28 projects from seven areas where that language is often a poor default and evaluates 25 models.
Three things are measured: preference for the language, consistency between recommendation and implementation, and language diversity. Looking not only at what was chosen but at whether the language recommended matches the code actually written is what distinguishes the design.
2Following the reasons
- 1Most choicesAutomatic, or driven primarily by ease
- 2What would be expectedAn explicit consideration of project requirements
- 3A smaller but important failureFabricating contextual support for the choice, called phantom evidence
- 4LikewiseProducing code that contradicts the language selected in the model own reasoning
Examining 9,826 reasoning traces, most choices did not follow from considering requirements. In some, context supporting the choice was written as though it existed when it did not. The authors call this phantom evidence.
3A reason being given and a reason existing
This failure matters because reading the output does not reveal it. Where a reason accompanies a judgment, the judgment looks sound. Whether that reason is supported by the input must be established separately. An explanation being present does not mean the explanation is correct.
4Smaller models lean harder
Smaller open-weight models are reported to show stronger preference and lower language diversity. Anyone choosing a small model to run themselves should assume that tendency. The point that evaluation design drives conclusions appears in other work this site covers. This article is our own summary of public research information and does not warrant its contents.
Why it matters
Even where output carries a reason, whether that reason rests on the input must be checked separately. That smaller self-hosted models lean harder in their choices is also a premise for selection.
FAQ
What is phantom evidence?
Can a stated reason be trusted?
Sources (primary)
Source: arXiv (descriptive metadata is CC0 public domain). Summaries are our own; see arXiv for the original text and PDF.
- arXiv abstract page (original, official)
- PDF (arXiv)
- arXiv ID: 2608.06041