@mods can this resolve yes? According to LMSYS, the March 2024 version of GPT-4 is currently ranked 40th and is outclassed by Gemini 1.5 8b in 31st and Gemma 2 9b in 25.

@JonathanMilligan https://manifold.markets/BoltonBailey/will-we-have-an-opensource-model-be?r=Sm9uYXRoYW5NaWxsaWdhbg

@JonathanMilligan Closing while I try and understand.

@JonathanMilligan Wait, why should LMSYS count? That isn't a benchmark, right?

@NathanpmYoung Yeah LMSYS is a benchmark of how people rank the output of LLM models.

@JonathanMilligan Currently it seems to be the best benchmark of real “in the wild” use.

@JonathanMilligan Okay but it isn't a "benchmark" see "GPT-4's benchmark results as of its release in march 2023." Right?

@mods what do you think?

@NathanpmYoung If you look at the original paper where they introduce LLM arena they refer to it as a “benchmark” https://lmsys.org/blog/2023-05-03-arena/ and on the homepage they also refer to it as an eval benchmark. https://lmarena.ai/

Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings | LMSYS Org

<p>We present Chatbot Arena, a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner. In t...

Sebastian Bubeck (Microsoft research) recently said that he thinks a 13 billion parameter model could do this. There've been reports of others working at similar sized models for similar performance.

@firstuserhere Source? Interesting speculation there

@CarterHinsley https://www.youtube.com/watch?v=K0XZ_ShxWkI