Firehose

Large language models (LLMs) can produce highly variable judgments of novel research ideas, with small changes in the evaluation setup resulting in dramatically different verdicts. This instability is particularly pronounced when evaluating AI-generated ideas, where the choice of prompt, reasoning budget, and model can significantly impact the outcome, and more search or reasoning does not necessarily produce more consistent judgments. AI summary