User:IssaRice/Scoring rule

how can we formalize the idea of a rule for scoring predictions?

first pass: statements and probabilities

we can start with a list $s_{1},\ldots ,s_{n}$ of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities $p_{1},\ldots ,p_{n}$ where $p_{j}$ is the probability someone assigns to $s_{j}$ being true. now, reality evaluates each statement, giving us a yes/no answer $r(s_{j})\in \{0,1\}$. our probabilities are scored against this response from reality. so a scoring rule S can be some function of $p_{1},\ldots ,p_{n},r(s_{1}),\ldots ,r(s_{n})$. so the type can be $S:[0,1]^{n}\times \{0,1\}^{n}\to \mathbf {R}$.

if we are an ordinary statistician [1], we might pick a rule like $\sum _{j=1}^{n}(p_{j}-r(s_{j}))^{2}$. (this is actually almost the brier score)

second pass: events

in probability theory, we are used to dealing with events and random variables. in the previous section, we naively stated scoring rules in terms of statements and probabilities. but we might try now to phrase things in terms of events.

instead of statements $s_{1},\ldots ,s_{n}$, we could have a list of events $A_{1},\ldots ,A_{n}$. here, $A_{j}$ is an event expressing the fact that $s_{j}$ is true. then $p_{j}=P(A_{j})$, where P is the probability measure which encodes our knowledge of what events are likely. $r(s_{j})$ is the outcome in some possible world, so $r(s_{j})=1_{A_{j}}(\omega )$.