- This topic has 3 replies, 3 voices, and was last updated 1 year, 9 months ago by
.
-
Posts
-
Hello:
I have a question on the score calculation. In the example of Score observation_1 calculation, I don’t understand where that coefficient 0.5 of the third term come from, I think the coefficient should it be 0? Because the true state is 3, the the third term is for state 2, its distance to state 3 is 1, so based on that table the coefficient (points) should be 0?
Thanks
Hello,
Thank you for using the interactive forum.
Sorry for our late response.
For your question, in the example, the correct label is 3. (just a reminder, the first entry is the sample id, the second entry is for label 0, the third entry is for label 1, etc.)
With the sample answer, label 3 has a probability of 0.8, which is multiplied with weight 1, while label 2 and label 4 have a probability of 0.1, which is multiplied with weight 0.5 (they are distance 1 away from the true answer).
Best,
PHM data Challenge team.
I think the description could be misleading. In particular, the meaning of “distance” and the entries in the point score table.
Here is what I understand using the first example:
sample_id, prob_0, prob_1, prob_2, prob_3, prob_4, prob_5, prob_6, prob_7, prob_8, prob_9, prob_10, confidence
1,0,0,0.1,0.8,0.1,0,0,0,0,0,0,1
This example means:
sample_id=1, prob_0=0, prob_1=0, prob_2=0.1, prob_3=0.8, prob_4=0.1, prob_5=0, prob_6=0, prob_7=0, prob_8=0, prob_9=0, prob_10=0, confidence=1
The truth state is 3.
According to the score calculation example:
Score[observation_1] = 1*(0+0+0.1*0.5+0.8*1+0.1*0.5+0+0+0+0+0+0) = 0.9,
the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0.5, prob_3=1, prob_4=0.5, prob_5=0, prob_6=0, prob_7=0, prob_8=0, prob_9=0, prob_10=0.The score calculation for observation_2 seems to miss one term, and my best guess that the correct formula is the following:
Score[observation_2] = 0.2*(0+0+0+0+0-0.1*0.5+0.1*0+0.4*0.5+0.4*1+0.1*0.5+0.1*0) = 0.12.
And the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0, prob_3=0, prob_4=0, prob_5=-0.5, prob_6=0, prob_7=0.5, prob_8=1, prob_9=0.5, prob_10=0.So, the definition of distance between the true state and predicted state seems to be the number of states between the two, excluding the true state and the predicted state. E.g. if true state is 3, a predicted state of 2 or 4 has zero distance from the true state. and a predicted state of 1 or 5 has distance 1 from the true state. And from the score calculation examples, the “zero distance” ones get a point=0.5, which is not in the table. A distance = 1 prediction gets a point=0. A distance = 2 prediction gets a point=-0.5.
So, I think the table describing the sore system needs a correction. Distance=0 is not a correct prediction, and its score should have been 0.5. The correct prediction should be in its own distance slot, maybe “distance undefined”.
Hello Statssde,
Thank you for the reply.
your calculation for 2 is correct, there is a missing 0. Thanks for pointing that out.
Also, in the weight table there is a missing 0.5 and misaligned for the rest.
Score[observation_2] = 0.2*(0+0+0+0+0-0.1*0.5+0.1*0+0.4*0.5+0.4*1+0.1*0.5+0.1*0) = 0.12.
And the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0, prob_3=0, prob_4=0, prob_5=-0.5, prob_6=0, prob_7=0.5, prob_8=1, prob_9=0.5, prob_10=0.Based on the assigned weight, 8-8=0 and had 0 distance, with a weight of 1. 7 and 9 has distance of 1 and a weight of 0.5. For distance 2, weight is 0, and for distance 3, weight is -0.5.
We will update the corresponding table.
Let us know if there is still ambiguity.Best,
Data Challenging Team.
- You must be logged in to reply to this topic.