- This topic has 1 reply, 2 voices, and was last updated 2 weeks, 1 day ago by
.
-
Posts
-
Dear PHM North America 2026 Data Challenge Organizing Team,
While preparing our submissions, we noticed an ambiguity in the scoring criteria that we would like to have clarified.
According to the challenge description, predictions may be rescaled using an optimal monotonic transformation prior to evaluation, so that the shape of the trajectory—and not the absolute scale—is explicitly evaluated. The metric specified is the MSE between the predicted and ground-truth trajectories.
We tested two submissions that are identical in form: The second file corresponds exactly to the first, multiplied by a factor of 7 (the values in one file are normalized to the range 0–1, while those in the other are in the range up to 7). The curve profiles are therefore identical except for a purely linear scaling; the Pearson correlation coefficient per run is 1.0.
On the submission page, however, these two files yield significantly different error values (approx. 10.08 versus approx. 6.143). Mathematically, a pure scaling by a factor of 7 should be completely absorbed by any correctly applied optimal monotonic or affine rescaling, so that both submissions should yield the same MSE. The differing values suggest that the scaling is not (fully) accounted for in the displayed score.
Hence our questions:
- Is the optimal monotonic transformation mentioned in the description already applied to the daily scores during the test phase, or only during the final evaluation of the validation phase?
-
If it is applied: How is the transformation specifically defined (e.g., free scale/offset fit, isotonic regression, with or without a fixed anchor)? A more precise definition might explain why a global factor is not fully compensated for.
-
Should we submit our predictions within a specific order of magnitude, or is the absolute scale actually irrelevant for the final evaluation?
-
During validation, is only one minute per run provided as a single file, based on which the percentage wear is to be estimated?
We would greatly appreciate a brief response. Thank you very much for organizing the challenge.
Best regards,
Team: ReichWiensHi Team ReichWiens,
I’ve scored your data submitted personally, and they are indeed the MSE. It’s straight forward; no tricks, no transformation. Just MSE scoring. The code is below:
def score_submitted_result(df_true, df_pred): ''' Calculate the score for a single team's submission ''' # Extract the targets y_true = df_true.metric.values y_pred = df_pred.metric.values # MSE score score = mean_squared_error(y_true, y_pred) return scoreI think you might be interpreting the phrase “optimal monotonic transformation” different than intended. We have metrics that we’ve extracted to represent the damage. They are empirical and monotonic. You have to match our metrics. I think that answers 1, 2, and 3.
Q4: files that you have are indeed 1 minute intervals. But They damage is assessed ~6 hours. (The exact time is in the files). You can use that data as best as you see fit.
- You must be logged in to reply to this topic.