I’m writing an undergraduate thesis comparing real and play money prediction markets at the moment, for which Polymarket and Manifold are my data sources respectively. Their relative accuracy is one of a few questions I plan to investigate.

The data: paired price time series of markets with identical resolution criteria. Polymarket’s price is the mid of the best bid and ask, Manifold’s the AMM price. Topics span sports futures, politics, econ, crypto prices, awards, and whatever other pairs I could find. Shooting for a sample size of at least 150.

I’ll probably use the prices one week before resolution, at least to resolve this market. I’ll bound Polymarket’s prices between 0.01 and 0.99 for a fair test. I’ll restrict the analysis to a-priori plausibly independent markets (which throws out a lot of politics markets). There’s a fairly big range of liquidity/number of traders in the markets.

The test: permutation test on difference in log scores. This means each market’s forecast is given the score ln(p) if it happened, ln(1-p) if it didn’t. Here, higher log score = more accurate. Then I’ll take the sum of differences in log scores across Polymarket-Manifold pairs. This is the test statistic.

If there were no systematic difference in accuracy, then the sign of each difference in log-scores should be random. This lets us generate a distribution of test statistics if Polymarket and Manifold were equally accurate - assign a random sign to the empirical log-score differences, compute the test statistic, then repeat (say) 10,000 times. If the true test statistic is greater than 95% of these values, we can reject the hypothesis of equal accuracy at 0.05 significance.

This market resolves YES iff this procedure shows Polymarket is more accurate than Manifold at p<0.05.

I anticipate I’ll have done this test some time in the next 1-3 months. But could be next week, whenever I get around to it given my other courses etc. I won’t trade in this market.