Will an LLM agent complete >50% of the lab tasks on the Factorio Learning Environment benchmark in 2025?
Will an LLM agent complete >50% of the lab tasks on the Factorio Learning Environment benchmark in 2025?
4
Ṁ852026
29%
chance
1D
1W
1M
ALL
Large Language Models (LLMs) are rapidly saturating existing benchmarks, necessitating new open-ended evaluations. We introduce the Factorio Learning Environment (FLE), based on the game of Factorio, that tests agents in long-term planning, program synthesis, and resource optimization.
https://jackhopkins.github.io/factorio-learning-environment/
As of the time of this market's creation, Claude 3.5 Sonnet tops the leaderboard with a success rate of 21.9%. This market resolves YES if the leaderboard displays an entry with >50% success before the end of 2025, NO otherwise. Claims of higher success rates won't count for resolution unless they're displayed on the leaderboard. Autonomous systems that are not based on an LLM agent framework also won't count for resolution.
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00What is this?
What is Manifold?
Manifold is the world's largest social prediction market.
Get accurate real-time odds on politics, tech, sports, and more.
Win cash prizes for your predictions on our sweepstakes markets! Always free to play. No purchase necessary.
Are our predictions accurate?
Yes! Manifold is very well calibrated, with forecasts on average within 4 percentage points of the true probability. Our probabilities are created by users buying and selling shares of a market.
In the 2022 US midterm elections, we outperformed all other prediction market platforms and were in line with FiveThirtyEight’s performance. Many people who don't like trading still use Manifold to get reliable news.
How do I win cash prizes?
Manifold offers two market types: play money and sweepstakes.
All questions include a play money market which uses mana and can't be cashed out.
and can't be cashed out.Selected markets will have a sweepstakes toggle. These require sweepcash to participate and winners can withdraw sweepcash as a cash prize. You can filter for sweepstakes markets on the browse page.
to participate and winners can withdraw sweepcash as a cash prize. You can filter for sweepstakes markets on the browse page.Redeem your sweepcash won from markets at 1.00 → $1.00, minus a 5% fee.
Learn more.1.00Related questions
What is this?
What is Manifold?
Manifold is the world's largest social prediction market.
Get accurate real-time odds on politics, tech, sports, and more.
Win cash prizes for your predictions on our sweepstakes markets! Always free to play. No purchase necessary.
Are our predictions accurate?
Yes! Manifold is very well calibrated, with forecasts on average within 4 percentage points of the true probability. Our probabilities are created by users buying and selling shares of a market.
In the 2022 US midterm elections, we outperformed all other prediction market platforms and were in line with FiveThirtyEight’s performance. Many people who don't like trading still use Manifold to get reliable news.
How do I win cash prizes?
Manifold offers two market types: play money and sweepstakes.
All questions include a play money market which uses mana and can't be cashed out.
and can't be cashed out.Selected markets will have a sweepstakes toggle. These require sweepcash to participate and winners can withdraw sweepcash as a cash prize. You can filter for sweepstakes markets on the browse page.
to participate and winners can withdraw sweepcash as a cash prize. You can filter for sweepstakes markets on the browse page.Redeem your sweepcash won from markets at 1.00 → $1.00, minus a 5% fee.
Learn more.1.00Related questions
Will there be any simple text-based task that most humans can solve, but top LLMs can't? By the end of 2026
66% chance
LLM Factorio by mid 2025
11% chance
Will there be an LLM which scores above what a human can do in 2 hours on METR's eval suite before 2026?
82% chance
By the end of June 2025, will closed-source LLMs increase access to pandemic agents?
63% chance
Will LLMs be able to formally verify non-trivial programs by the end of 2025?
31% chance
What will be true of OpenAI's best LLM by EOY 2025?
Will LLMs be better than typical white-collar workers on all computer tasks before 2026?
18% chance
Will RL work for LLMs "spill over" to the rest of RL by 2026?
33% chance
Will the best public LLM at the end of 2025 solve more than 5 of the first 10 Project Euler problems published in 2026?
78% chance
Will one of the major LLMs be capable of continual lifelong learning (learning from inference runs) by EOY 2025?
26% chance