The Nonlinear Library EA - Higher-Order Forecasts by Ozzie Gooen
May 25, 2024
05:42
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Higher-Order Forecasts, published by Ozzie Gooen on May 25, 2024 on The Effective Altruism Forum.
Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.
The core idea is straightforward:
Nth-order forecasts are forecasts about (N-1)th order forecasts.
Examples
Here are some examples:
0-Order Forecasting (i.e., the ground truth)
Biden won the 2020 U.S. presidential election
The US GDP in 2023 was $27 trillion
1st-Order Forecasting (i.e., regular forecasting)
What is the chance that Trump will win the 2024 U.S. presidential election?
What will be the GDP of the US in 2024?
2nd-Order Forecasting
How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?
If the question "What is the chance that a Republican will win the 2028 Presidential Election?" was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?"
3rd-Order Forecasting
How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?], be correlated, from now until 2024?
How valuable were all the forecasts for the question, ['How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?']
As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.
Benefits
Some potential benefits of higher-order forecasting include:
1. Identify Overconfidence
Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
"How overconfident is [forecast/forecaster] X"
2. Prioritize Questions
Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
"How valuable is the information from forecasting question X?"
3. Surface Relationships
Surface key drivers and correlations between events by letting forecasters predict how different questions relate to each other.
"How correlated will the forecasts for questions X and Y be over [time period]?"
4. Faster Information Aggregation
Enable faster aggregation of information by allowing forecasts on future forecast values, which may update more frequently than the underlying events.
"What will the forecast for question X be on [future date], conditional on [other forecasts or events]?"
5. Leverage Existing Infrastructure
Leverage the existing infrastructure and resolution processes of prediction platforms, which are already designed to handle large numbers of forecasting questions.
We've already seen some early examples of higher-order forecasts on platforms like Manifold Markets. For example, with the recent questions:
Will there be a Manifold bot that makes profitable bets on random 1-month markets by December 2024? (Ṁ3,000 subsidy!)
Manifold Top Traders Leaderboard Ranking Prediction (2024)
If Manifold begins allowing real-money withdrawals, will its accuracy improve?
Is Manifold's P(Doom) by 2050 currently between 10% and 90%? [Resolves to Poll]
Will Manifold be more accurate than real-money markets in forecasting the 2024 election?
Challenges
Of course, there are also challenges and risks to cons...
Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.
The core idea is straightforward:
Nth-order forecasts are forecasts about (N-1)th order forecasts.
Examples
Here are some examples:
0-Order Forecasting (i.e., the ground truth)
Biden won the 2020 U.S. presidential election
The US GDP in 2023 was $27 trillion
1st-Order Forecasting (i.e., regular forecasting)
What is the chance that Trump will win the 2024 U.S. presidential election?
What will be the GDP of the US in 2024?
2nd-Order Forecasting
How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?
If the question "What is the chance that a Republican will win the 2028 Presidential Election?" was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?"
3rd-Order Forecasting
How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?], be correlated, from now until 2024?
How valuable were all the forecasts for the question, ['How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?']
As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.
Benefits
Some potential benefits of higher-order forecasting include:
1. Identify Overconfidence
Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
"How overconfident is [forecast/forecaster] X"
2. Prioritize Questions
Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
"How valuable is the information from forecasting question X?"
3. Surface Relationships
Surface key drivers and correlations between events by letting forecasters predict how different questions relate to each other.
"How correlated will the forecasts for questions X and Y be over [time period]?"
4. Faster Information Aggregation
Enable faster aggregation of information by allowing forecasts on future forecast values, which may update more frequently than the underlying events.
"What will the forecast for question X be on [future date], conditional on [other forecasts or events]?"
5. Leverage Existing Infrastructure
Leverage the existing infrastructure and resolution processes of prediction platforms, which are already designed to handle large numbers of forecasting questions.
We've already seen some early examples of higher-order forecasts on platforms like Manifold Markets. For example, with the recent questions:
Will there be a Manifold bot that makes profitable bets on random 1-month markets by December 2024? (Ṁ3,000 subsidy!)
Manifold Top Traders Leaderboard Ranking Prediction (2024)
If Manifold begins allowing real-money withdrawals, will its accuracy improve?
Is Manifold's P(Doom) by 2050 currently between 10% and 90%? [Resolves to Poll]
Will Manifold be more accurate than real-money markets in forecasting the 2024 election?
Challenges
Of course, there are also challenges and risks to cons...