User:IssaRice/Scoring rule

how can we formalize the idea of a rule for scoring predictions?

first pass

we can start with a list $s_{1},\ldots ,s_{n}$ of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities $p_{1},\ldots ,p_{n}$ where $p_{j}$ is the probability someone assigns to $s_{j}$ being true. now, reality evaluates each statement, giving us a yes/no answer $r(s_{j})\in \{0,1\}$. our probabilities are scored against this response from reality. so a scoring rule S can be some function of $p_{1},\ldots ,p_{n},r(s_{1}),\ldots ,r(s_{n})$. so the type can be $S:[0,1]^{n}\times \{0,1\}^{n}\to \mathbf {R}$.

if we are an ordinary statistician [1], we might pick a rule like $\sum _{j=1}^{n}(p_{j}-r(s_{j}))^{2}$. (this is actually almost the brier score)