Tagged: question on score calculation
 This topic has 3 replies, 3 voices, and was last updated 1 year, 3 months ago by yongzhiqu.

AuthorPosts

July 14, 2023 at 1:25 pm #2479kshen000998365Participant
Hello:
I have a question on the score calculation. In the example of Score observation_1 calculation, I don’t understand where that coefficient 0.5 of the third term come from, I think the coefficient should it be 0? Because the true state is 3, the the third term is for state 2, its distance to state 3 is 1, so based on that table the coefficient (points) should be 0?
Thanks
July 17, 2023 at 4:08 pm #2484yongzhiquKeymasterHello,
Thank you for using the interactive forum.
Sorry for our late response.
For your question, in the example, the correct label is 3. (just a reminder, the first entry is the sample id, the second entry is for label 0, the third entry is for label 1, etc.)
With the sample answer, label 3 has a probability of 0.8, which is multiplied with weight 1, while label 2 and label 4 have a probability of 0.1, which is multiplied with weight 0.5 (they are distance 1 away from the true answer).
Best,
PHM data Challenge team.
July 17, 2023 at 5:49 pm #2487statssdeParticipantI think the description could be misleading. In particular, the meaning of “distance” and the entries in the point score table.
Here is what I understand using the first example:
sample_id, prob_0, prob_1, prob_2, prob_3, prob_4, prob_5, prob_6, prob_7, prob_8, prob_9, prob_10, confidence
1,0,0,0.1,0.8,0.1,0,0,0,0,0,0,1
This example means:
sample_id=1, prob_0=0, prob_1=0, prob_2=0.1, prob_3=0.8, prob_4=0.1, prob_5=0, prob_6=0, prob_7=0, prob_8=0, prob_9=0, prob_10=0, confidence=1
The truth state is 3.
According to the score calculation example:
Score[observation_1] = 1*(0+0+0.1*0.5+0.8*1+0.1*0.5+0+0+0+0+0+0) = 0.9,
the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0.5, prob_3=1, prob_4=0.5, prob_5=0, prob_6=0, prob_7=0, prob_8=0, prob_9=0, prob_10=0.The score calculation for observation_2 seems to miss one term, and my best guess that the correct formula is the following:
Score[observation_2] = 0.2*(0+0+0+0+00.1*0.5+0.1*0+0.4*0.5+0.4*1+0.1*0.5+0.1*0) = 0.12.
And the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0, prob_3=0, prob_4=0, prob_5=0.5, prob_6=0, prob_7=0.5, prob_8=1, prob_9=0.5, prob_10=0.So, the definition of distance between the true state and predicted state seems to be the number of states between the two, excluding the true state and the predicted state. E.g. if true state is 3, a predicted state of 2 or 4 has zero distance from the true state. and a predicted state of 1 or 5 has distance 1 from the true state. And from the score calculation examples, the “zero distance” ones get a point=0.5, which is not in the table. A distance = 1 prediction gets a point=0. A distance = 2 prediction gets a point=0.5.
So, I think the table describing the sore system needs a correction. Distance=0 is not a correct prediction, and its score should have been 0.5. The correct prediction should be in its own distance slot, maybe “distance undefined”.
July 18, 2023 at 7:44 am #2489yongzhiquKeymasterHello Statssde,
Thank you for the reply.
your calculation for 2 is correct, there is a missing 0. Thanks for pointing that out.
Also, in the weight table there is a missing 0.5 and misaligned for the rest.
Score[observation_2] = 0.2*(0+0+0+0+00.1*0.5+0.1*0+0.4*0.5+0.4*1+0.1*0.5+0.1*0) = 0.12.
And the points corresponding to each prob_X is as follows:
prob_0=0, prob_1=0, prob_2=0, prob_3=0, prob_4=0, prob_5=0.5, prob_6=0, prob_7=0.5, prob_8=1, prob_9=0.5, prob_10=0.Based on the assigned weight, 88=0 and had 0 distance, with a weight of 1. 7 and 9 has distance of 1 and a weight of 0.5. For distance 2, weight is 0, and for distance 3, weight is 0.5.
We will update the corresponding table.
Let us know if there is still ambiguity.Best,
Data Challenging Team.

AuthorPosts
 You must be logged in to reply to this topic.