User:IssaRice/Scoring rule: Difference between revisions
No edit summary |
No edit summary |
||
| Line 5: | Line 5: | ||
we can start with a list <math>s_1,\ldots,s_n</math> of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities <math>p_1,\ldots,p_n</math> where <math>p_j</math> is the probability someone assigns to <math>s_j</math> being true. now, reality evaluates each statement, giving us a yes/no answer <math>r(s_j) \in \{0,1\}</math>. our probabilities are scored against this response from reality. so a scoring rule S can be some function of <math>p_1,\ldots,p_n,r(s_1),\ldots,r(s_n)</math>. so the type can be <math>S : [0,1]^n \times \{0,1\}^n \to \mathbf R</math>. | we can start with a list <math>s_1,\ldots,s_n</math> of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities <math>p_1,\ldots,p_n</math> where <math>p_j</math> is the probability someone assigns to <math>s_j</math> being true. now, reality evaluates each statement, giving us a yes/no answer <math>r(s_j) \in \{0,1\}</math>. our probabilities are scored against this response from reality. so a scoring rule S can be some function of <math>p_1,\ldots,p_n,r(s_1),\ldots,r(s_n)</math>. so the type can be <math>S : [0,1]^n \times \{0,1\}^n \to \mathbf R</math>. | ||
if we are an ordinary statistician [https://www.readthesequences.com/A-Technical-Explanation-Of-Technical-Explanation], we might pick a rule like <math>\sum_{j=1}^n (p_j - r(s_j))^2</math>. (this is actually almost the brier score) | if we are an ordinary statistician [https://www.readthesequences.com/A-Technical-Explanation-Of-Technical-Explanation], we might pick a rule like <math display=inline>\sum_{j=1}^n (p_j - r(s_j))^2</math>. (this is actually almost the brier score) | ||
==second pass: events== | ==second pass: events== | ||
| Line 11: | Line 11: | ||
in probability theory, we are used to dealing with events and random variables. in the previous section, we naively stated scoring rules in terms of statements and probabilities. but we might try now to phrase things in terms of events. | in probability theory, we are used to dealing with events and random variables. in the previous section, we naively stated scoring rules in terms of statements and probabilities. but we might try now to phrase things in terms of events. | ||
instead of statements <math>s_1, \ldots, s_n</math>, we could have a list of events <math>A_1, \ldots, A_n</math>. here, <math>A_j</math> is an event expressing the fact that <math>s_j</math> is true. then <math>p_j = P(A_j)</math>, where P is the probability measure which encodes our knowledge of what events are likely. <math>r(s_j)</math> is the outcome in some possible world, so <math>r(s_j) = 1_{A_j}(\omega)</math>. | instead of statements <math>s_1, \ldots, s_n</math>, we could have a list of events <math>A_1, \ldots, A_n</math>. here, <math>A_j</math> is an event expressing the fact that <math>s_j</math> is true. then <math>p_j = P(A_j)</math>, where P is the probability measure which encodes our knowledge of what events are likely. <math>r(s_j)</math> is the outcome in some possible world, so <math>r(s_j) = 1_{A_j}(\omega)</math>. the idea here is we have some implicit sample space <math>\Omega</math> of all "possible worlds". then each <math>\omega \in \Omega</math> is a possible world. but this is exactly the idea expressed by our reality function <math>r</math> -- we could have had some other reality <math>r'</math> in which our same probabilities would perform differently. | ||
Revision as of 03:55, 8 January 2020
how can we formalize the idea of a rule for scoring predictions?
first pass: statements and probabilities
we can start with a list of statements. each statement makes a yes/no prediction about the future, like "the die will show 3 when rolled". then we have a list of probabilities where is the probability someone assigns to being true. now, reality evaluates each statement, giving us a yes/no answer . our probabilities are scored against this response from reality. so a scoring rule S can be some function of . so the type can be .
![{\displaystyle S:[0,1]^{n}\times \{0,1\}^{n}\to \mathbf {R} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/c3908f29e508a308eebeec71392f346c056b5427)
if we are an ordinary statistician [1], we might pick a rule like . (this is actually almost the brier score)
second pass: events
in probability theory, we are used to dealing with events and random variables. in the previous section, we naively stated scoring rules in terms of statements and probabilities. but we might try now to phrase things in terms of events.
instead of statements , we could have a list of events . here, is an event expressing the fact that is true. then , where P is the probability measure which encodes our knowledge of what events are likely. is the outcome in some possible world, so . the idea here is we have some implicit sample space of all "possible worlds". then each is a possible world. but this is exactly the idea expressed by our reality function -- we could have had some other reality in which our same probabilities would perform differently.
