User:IssaRice/Scoring rule - Revision history

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T05:04:28Z

proving logarithmic scoring is proper

← Older revision		Revision as of 05:04, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p,X)]</math>. to make sense of this, first just pay attention to the bare p's. we're looping through all possible p's we can submit to the scoring process, and we want to find the p that maximizes the score. but we can't ''just'' do this raw kind of maximization, because then we'd obviously get that the best probability is ~~the~~ assign 1 to something if it will happen, and 0 if it won't happen -- but we can't predict the future so well, so this is useless! so instead of maximizing the raw score, we want to maximize the expected score, given our current beliefs <math>\overline p</math>. so this maximization process spits out some probabilities that we should submit that will get us a high score, given our current beliefs. now, a score is proper if one way to do good is to just submit your actual beliefs.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p,X)]</math>. to make sense of this, first just pay attention to the bare p's. we're looping through all possible p's we can submit to the scoring process, and we want to find the p that maximizes the score. but we can't ''just'' do this raw kind of maximization, because then we'd obviously get that the best probability is to assign 1 to something if it will happen, and 0 if it won't happen -- but we can't predict the future so well, so this is useless! so instead of maximizing the raw score, we want to maximize the expected score, given our current beliefs <math>\overline p</math>. so this maximization process spits out some probabilities that we should submit that will get us a high score, given our current beliefs. now, a score is proper if one way to do good is to just submit your actual beliefs.

	for logarithmic scoring the expectation is <math>\sum_{x\in\{0,1\}} (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.		for logarithmic scoring the expectation is <math>\sum_{x\in\{0,1\}} (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T05:03:56Z

proving logarithmic scoring is proper

← Older revision		Revision as of 05:03, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p)]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p,X)]</math>. to make sense of this, first just pay attention to the bare p's. we're looping through all possible p's we can submit to the scoring process, and we want to find the p that maximizes the score. but we can't ''just'' do this raw kind of maximization, because then we'd obviously get that the best probability is the assign 1 to something if it will happen, and 0 if it won't happen -- but we can't predict the future so well, so this is useless! so instead of maximizing the raw score, we want to maximize the expected score, given our current beliefs <math>\overline p</math>. so this maximization process spits out some probabilities that we should submit that will get us a high score, given our current beliefs. now, a score is proper if one way to do good is to just submit your actual beliefs.

	for logarithmic scoring the expectation is <math>\sum_{x\in\{0,1\}} (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.		for logarithmic scoring the expectation is <math>\sum_{x\in\{0,1\}} (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:58:43Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:58, 8 January 2020
Line 33:		Line 33:
	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p)]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p)]</math>.

	for logarithmic scoring the expectation is <math>\~~sum_x~~ (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.		for logarithmic scoring the expectation is <math>\sum_{x\in\{0,1\}} (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:57:45Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:57, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S(p)]</math>.

			for logarithmic scoring the expectation is <math>\sum_x (x \log p + (1-x) \log (1-p))\cdot \overline p(x) = \overline p(1)\log p + (1-\overline p(1))\log(1-p)</math>. differentiating this, setting it to 0, and solving for p [is this even valid here? we need to check for convexity/increasing/whatever optimization stuff but idk how] gets us <math>p = \overline p(1)</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:54:14Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:54, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \mathrm{arg\,max}_p \mathbb E_{\overline p}[S]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline p = \mathrm{arg\,max}_p \mathbb E_{\overline p}[S]</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:54:03Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:54, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \~~operatorname~~{arg\,max}_p \mathbb E_{\overline p}[S]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \mathrm{arg\,max}_p \mathbb E_{\overline p}[S]</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:53:55Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:53, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \mathrm*{arg\,max}_p \mathbb E_{\overline p}[S]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \operatorname{arg\,max}_p \mathbb E_{\overline p}[S]</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:53:43Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:53, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \~~operatorname~~*{arg\,max}_p \mathbb E_{\overline p}[S]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \mathrm*{arg\,max}_p \mathbb E_{\overline p}[S]</math>.

IssaRice: /* proving logarithmic scoring is proper */

2020-01-08T04:53:25Z

proving logarithmic scoring is proper

← Older revision		Revision as of 04:53, 8 January 2020
Line 31:		Line 31:
	==proving logarithmic scoring is proper==		==proving logarithmic scoring is proper==

	from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \operatorname*{arg\,max}_p \~~mathbbE_~~{\overline p}[S]</math>.		from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \operatorname*{arg\,max}_p \mathbb E_{\overline p}[S]</math>.

IssaRice: /* third pass: non-binary predictions */

2020-01-08T04:53:13Z

third pass: non-binary predictions

← Older revision		Revision as of 04:53, 8 January 2020
Line 27:		Line 27:
	here we get to one of the confusions i have about the explanation of scoring rules in [https://www.readthesequences.com/A-Technical-Explanation-Of-Technical-Explanation]. there, we score each probability separately, i.e. <math>S' : [0,1] \to \mathbf R</math>. and we assume that <math>S(p_1,\ldots,p_k,i) = S'(p_i)</math>. i think it's better to write this as <math display=inline>\sum_{j=1}^k S'(p_j)[j=i]</math>, where <math>[j=i]</math> is 1 if j=i and 0 otherwise. this makes the dependence on all the probabilities clear.		here we get to one of the confusions i have about the explanation of scoring rules in [https://www.readthesequences.com/A-Technical-Explanation-Of-Technical-Explanation]. there, we score each probability separately, i.e. <math>S' : [0,1] \to \mathbf R</math>. and we assume that <math>S(p_1,\ldots,p_k,i) = S'(p_i)</math>. i think it's better to write this as <math display=inline>\sum_{j=1}^k S'(p_j)[j=i]</math>, where <math>[j=i]</math> is 1 if j=i and 0 otherwise. this makes the dependence on all the probabilities clear.

	how do we make sense of stuff like the invariance <math>S[P(A,B)] = S[P(A)] + S[P(B\|A)]</math> from Technical Explanation? we first need two partitions <math>A_1,\ldots,A_k</math> and <math>B_1, \ldots, B_m</math> of the sample space. these two partitions induce a third partition where each element of the new partition is the intersection of one element each from As and Bs, e.g. <math>A_1 \cap B_1, A_1\cap B_2, \ldots</math>. call this new partition <math>C_1, \ldots, C_r</math>. then we want <math>S(P(C_1), \ldots, P(C_r), (a,b)) = S(P(A_1), \ldots, P(A_k), a) + S(P(B_1 \| A_j), \ldots, P(B_m \| A_j), b)</math> for every <math>A_j</math>.		how do we make sense of stuff like the invariance <math>S[P(A,B)] = S[P(A)] + S[P(B\|A)]</math> from Technical Explanation? we first need two partitions <math>A_1,\ldots,A_k</math> and <math>B_1, \ldots, B_m</math> of the sample space. these two partitions induce a third partition where each element of the new partition is the intersection of one element each from As and Bs, e.g. <math>A_1 \cap B_1, A_1\cap B_2, \ldots</math>. call this new partition <math>C_1, \ldots, C_r</math>. then we want something like <math>S(P(C_1), \ldots, P(C_r), (a,b)) = S(P(A_1), \ldots, P(A_k), a) + S(P(B_1 \| A_j), \ldots, P(B_m \| A_j), b)</math> for every <math>A_j</math> (need better notation?).

			==proving logarithmic scoring is proper==

			from [https://arxiv.org/pdf/1808.07501.pdf], if <math>\overline p</math> is the actual beliefs of the forecaster, and <math>p</math> is the submitted probabilities, then to be proper we want <math>\overline \operatorname*{arg\,max}_p \mathbbE_{\overline p}[S]</math>.