It's soooooo slow though. And the results in real world day-to-day usage are rarely better than Sonnet 3.5 new, which are nearly instantaneous. (I ask both the same questions, and use them on a nearly daily basis)

Option of Oct 12th2024 should be resolved.

@Adamacki I can’t seem to find a way to partially resolve the market

Apparently o1's AIME score was pass@10000, not pass@1. Criteria should be updates accordingly

@JaundicedBaboon I updated the benchmark to the pass@1 score