[statssde]

I think the description could be misleading. In particular, the meaning of “distance” and the entries in the point score table.
Here is what I understand using the first example:
sample_id, prob_0, prob_1, prob_2, prob_3, prob_4, prob_5, prob_6, prob_7, prob_8, prob_9, prob_10, confidence
1,0,0,0.1,0.8,0.1,0,0,0,0,0,0,1
This example means:
sample_id=1, prob_0=0, prob_1=0, prob_2=0.1, prob_3=0.8, prob_4=0.1, prob_5=0, prob_6=0, prob_7=0, prob_8=0, prob_9=0, prob_10=0, confidence=1
The truth state is 3.
According to the score calculation example:
Score[observation_1] = 1*(0+0+0.1*0.5+0.8*1+0.1*0.5+0+0+0+0+0+0) = 0.9,
the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0.5, prob_3=1, prob_4=0.5, prob_5=0, prob_6=0, prob_7=0, prob_8=0, prob_9=0, prob_10=0.

The score calculation for observation_2 seems to miss one term, and my best guess that the correct formula is the following:
Score[observation_2] = 0.2*(0+0+0+0+0-0.1*0.5+0.1*0+0.4*0.5+0.4*1+0.1*0.5+0.1*0) = 0.12.
And the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0, prob_3=0, prob_4=0, prob_5=-0.5, prob_6=0, prob_7=0.5, prob_8=1, prob_9=0.5, prob_10=0.

So, the definition of distance between the true state and predicted state seems to be the number of states between the two, excluding the true state and the predicted state. E.g. if true state is 3, a predicted state of 2 or 4 has zero distance from the true state. and a predicted state of 1 or 5 has distance 1 from the true state. And from the score calculation examples, the “zero distance” ones get a point=0.5, which is not in the table. A distance = 1 prediction gets a point=0. A distance = 2 prediction gets a point=-0.5.

So, I think the table describing the sore system needs a correction. Distance=0 is not a correct prediction, and its score should have been 0.5. The correct prediction should be in its own distance slot, maybe “distance undefined”.

[Author]

Posts