User:IssaRice/Scoring rule: Difference between revisions
(Created page with "how can we formalize the idea of a rule for scoring predictions? ==first pass== we can start with a list <math>s_1,\ldots,s_n</math> of statements. each statement makes a ye...") |
No edit summary |
||
| Line 4: | Line 4: | ||
we can start with a list <math>s_1,\ldots,s_n</math> of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities <math>p_1,\ldots,p_n</math> where <math>p_j</math> is the probability someone assigns to <math>s_j</math> being true. now, reality evaluates each statement, giving us a yes/no answer <math>r(s_j) \in \{0,1\}</math>. our probabilities are scored against this response from reality. so a scoring rule S can be some function of <math>p_1,\ldots,p_n,r(s_1),\ldots,r(s_n)</math>. so the type can be <math>S : [0,1]^n \times \{0,1\}^n \to \mathbf R</math>. | we can start with a list <math>s_1,\ldots,s_n</math> of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities <math>p_1,\ldots,p_n</math> where <math>p_j</math> is the probability someone assigns to <math>s_j</math> being true. now, reality evaluates each statement, giving us a yes/no answer <math>r(s_j) \in \{0,1\}</math>. our probabilities are scored against this response from reality. so a scoring rule S can be some function of <math>p_1,\ldots,p_n,r(s_1),\ldots,r(s_n)</math>. so the type can be <math>S : [0,1]^n \times \{0,1\}^n \to \mathbf R</math>. | ||
if we are an ordinary statistician [https://www.readthesequences.com/A-Technical-Explanation-Of-Technical-Explanation], we might pick a rule like <math>\sum_{j=1}^n (p_j - r(s_j))^2</math>. (this is actually almost the brier score) | |||
==second pass== | |||
Revision as of 03:45, 8 January 2020
how can we formalize the idea of a rule for scoring predictions?
first pass
we can start with a list of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities where is the probability someone assigns to being true. now, reality evaluates each statement, giving us a yes/no answer . our probabilities are scored against this response from reality. so a scoring rule S can be some function of . so the type can be .
![{\displaystyle S:[0,1]^{n}\times \{0,1\}^{n}\to \mathbf {R} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/c3908f29e508a308eebeec71392f346c056b5427)
if we are an ordinary statistician [1], we might pick a rule like . (this is actually almost the brier score)
