[tony00c]

Dear organizers,

Thank you for your efforts in organizing and operating the competition. We have a few questions regarding the evaluation and submission process.

Q1. Regarding the validation dataset, will the validation score be provided in a leaderboard format during the competition, or will it remain hidden until the final evaluation day?

Q2. Is the metric intended to be applied consistently across all datasets? In other words, is the same evaluation procedure used for the training, validation, and test image data?

Q3. By the way, after the competition ends on August 7, will the ground-truth image labels for the train data be disclosed? Or will they remain confidential?

Q4. At the end of the competition, will teams be given an opportunity to select their best submission for final evaluation, or will the most recent submission at the deadline automatically be used as the team’s final submission?

Thank you in advance for your clarification.

[thejbills]

Q1. Regarding the validation dataset, will the validation score be provided in a leaderboard format during the competition, or will it remain hidden until the final evaluation day?

They will remain hidden until the final evaluation. You can upload as many times as you like during the open period, but only your final submission will be scored.

Q2. Is the metric intended to be applied consistently across all datasets? In other words, is the same evaluation procedure used for the training, validation, and test image data?

It’s is the same metric.

Q3. By the way, after the competition ends on August 7, will the ground-truth image labels for the train data be disclosed? Or will they remain confidential?

That is too be determined. In the passed we’ve never released the ground-truth, but this data set is lent to us. We’re not sure what the owner will do, but we will likely make that announced at the PHM Conference.

Q4. At the end of the competition, will teams be given an opportunity to select their best submission for final evaluation, or will the most recent submission at the deadline automatically be used as the team’s final submission?

It’s the last uploaded submission that’s scored.

[Author]

Posts