User:IssaRice/Scoring rule

how can we formalize the idea of a rule for scoring predictions?

first pass

we can start with a list $s_1,\dots ,s_n$ of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities $p_1,\dots ,p_n$ where $p_j$ is the probability someone assigns to $s_j$ being true. now, reality evaluates each statement, giving us a yes/no answer $r(s_j)\in \{0,1\}$. our probabilities are scored against this response from reality. so a scoring rule S can be some function of $p_1,\dots ,p_n,r(s_1),\dots ,r(s_n)$. so the type can be $S:[0,1{]}^n\times \{0,1{\}}^n\to \mathbf{R}$.

if we are an ordinary statistician [1], we might pick a rule like $\sum _{j=1}^n(p_j-r(s_j){)}^2$. (this is actually almost the brier score)