Immediately after switching the page, it will work with CSR.
Please reload your browser to see how it works.

no_multitudes 1 daysReload

Are there examples of the outputs the LLMs under test generated? I couldn't find any detailed ones in the paper or code.

The result here seems to be "Our Judge LLM gave another LLM a 21% grade for some code it generated", which is ... not qualitatively meaningful at all to me.

smusamashah 1 daysReload

    We evaluate several frontier models on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, achieves an average replication score of 21.0%.

tetris11 6 hoursReload

Sounds like a good initiative, but not one that should be under the ownership of a for-profit company with a massive stake in the race.

amelius 1 daysReload

One thing I'd be interested in is a UI for reading papers with AI assistance.

timabdulla 1 daysReload

What were the human PhDs able to do after more than 48 hours of effort? Presumably given that these are top-level PhDs, the replication success rate would be close to 100%?

smusamashah 1 daysReload

    We evaluate several frontier models on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, achieves an average replication score of 21.0%.

no_multitudes 1 daysReload

Are there examples of the outputs the LLMs under test generated? I couldn't find any detailed ones in the paper or code.

The result here seems to be "Our Judge LLM gave another LLM a 21% grade for some code it generated", which is ... not qualitatively meaningful at all to me.