2022 PHM Conference Data Challenge
Objective
This year’s data challenge addresses the problem of fault classification for a rock drill application under different individual configurations of the rock drill. The task is to develop a fault diagnosis/classification model using the provided pressure sensor data as input. The training data consists of data from various faults from five individual configurations, while the testing data for the online leaderboard is blind and is from one individual configuration of the rock drill. A final validation data set for the final scoring for the competition will be from two individual configurations from the rock drill and the labels will be blind to the contest participants. For both the testing data for the online leaderboard and the final validation data set, a reference condition from a no-fault health condition will also be provided.
The training data set contains data from 11 different fault classification categories, in which 10 are different failure modes and one class is from the healthy/no fault condition. The task is to train a model to classify the fault conditions using the training data, and to test this model on the testing data, in which the one submission per day can be used for submitting results to the online leaderboard. Validation is done with a validation data set that will be released for a one-time assessment at the end of the data challenge. Scoring of performance is done through this web interface.
The top three teams will be asked to submit their approach as a journal paper to the International Journal of Prognostics and Health Management (IJPHM) and present their approach and results at the 2022 PHM Society Conference. Top teams are also being recognized at the banquet and will receive a plaque.
PHM2022 Data Challenge Leaderboard
| PHM 2022 Data Challenge | |||
|---|---|---|---|
| Final Results | |||
| Rank | Submitting User | Team Name and Members | Accuracy |
| 1st Place | jinoh | Team name: Team-Nuri Organization: Seoul National University, Laboratory for System Health & Risk Management
| 100.00% |
| -- | linghuixuan | Disqualified from competition (unable to attend conference in person) | 99.97% |
| 2nd Place | wwaatt301 | Team name: SWING Organization: Seoul National University, Laboratory for System Health & Risk Management
| 99.77% |
| 3rd Place | jmontoya | Team name: MathWorks Organization: MathWorks
| 99.04% |
| 5 | ashitg | 98.09% | |
| 6 | igorgo | 97.80% | |
| 7 | jmpion | 97.38% | |
| 8 | mokeeffe7 | 93.61% | |
| 9 | asl79 | 92.76% | |
| 10 | osarenren | 91.66% | |
| 11 | minamitu | 89.95% | |
| 12 | martinvlh | 86.17% | |
| 13 | jeremy | 84.62% | |
| 14 | hsuch | 82.90% | |
| 15 | seoka | 82.04% | |
| 16 | jacobchesnes | 79.63% | |
| 17 | mtumcakir | 60.64% | |
| 18 | ebaumann | 49.71% | |
Teams
Collaboration is encouraged and teams may be comprised of one or more students and professionals. The teams judged to have the first, second, and third best scores will be awarded prizes, contingent upon:
- Having at least one member of the team register and attend the PHM 2022 Conference.
- Submitting a journal paper for the IJPHM journal.
- Presenting the analysis results and technique employed at a special session within the conference program.
The organizers of the competition reserve the right to both modify these rules and disqualify any team for any efforts it deems inconsistent with fair and open practices.
Data Challenge Registration
Teams may register here. Please note that the registration is a 2 steps process:
1) You must first log in with a PHM user account to complete the form. If you need to create one here –> Sign Up
2) Once logged in, you need to complete the Data Challenge Application form. Then, we will grant your user access to use the Data Challenge submission area.
Please note: In the spirit of fair competition, we allow only one account per team. Please do not register multiple times under different user names, under fictitious names, or using anonymous accounts. Competition organizers reserve the right to delete multiple entries from the same person (or team) and/or to disqualify those who are trying to “game” the system or using fictitious identities.
Relevant Dates
| Key PHM Data Challenge Dates | |
|---|---|
Competition Open – the following information will be posted:
|
June 3, 2022 |
| Final Validation Data Posted | July 29, 2022 |
| Competition Closed | August 12, 2022 (11:59:59 pm PDT) |
| Preliminary Winners Announced | August 19, 2022 |
| Conference Presentation Due, Winners Announced | September 13, 2022 |
| PHM Conference Dates | November 1 – November 4, 2022 |
Data Challenge Details
The details about this year’s data challenge can be found in this document. For each CSV file that is provided, the first column is the label for that cycle, in which a label of 1 represents the no-fault condition, a label of 2-11 correspond to different fault types. A label of 0, which is present for the testing data for the online leaderboard and the final validation data set, indicate an unknown/blind label, in which the objective is to predict the label for each cycle. The columns of data after the label represent the pressure signal data for that cycle.
Dataset
Training and Testing Dataset
The training and test datasets are available here: Training Data Set, Testing Data Set
Validation Dataset
The one-time validation data set is available here: Validation Data Set
NOTE: The official scoring for the competition is based solely on the validation data set.
Data Challenge Submission
For Testing Dataset:
Please upload your submission in the Data Challenge submission area. Note that you will only see the Submission area if you have previously applied using the application form and we have granted you access
Please ensure the filename is yourusername.txt or else the automated scorer will not be able to read your file and score your algorithm and the conference organizers will not be able to consider your submission as a valid submission. Please ensure that your results are in the format shown in this Data_Challenge_PHM2022_Example_Submission.txt file which contains only 3184 values in a column vector format.
Also, you only can submit (or overwrite) the yourusername.txt file for your submission as often as you’d like. The system will automatically score the most current submission twice daily, at noon and midnight US Pacific Daylight Time (0700 and 2100 UTC). You may continue to upload submission of testing data to the submission area after the validation data is available, and they will continue to be scored and displayed.
For Validation Dataset:
Please upload your submission in the Data Challenge Validation submission area. Note that you will only see the Submission area if you have previously applied using the application form and we have granted you access
Please ensure the filename is yourusername.txt or else the automated scorer will not be able to read your file and score your algorithm and the conference organizers will not be able to consider your submission as a valid submission. Since the validation data consists of two individual configurations (#7 and #8), please combine the results into a single file. The first 7935 rows should correspond to #7 and the last 8461 rows corresponding to #8, for a total of 16396 rows. A format checker has been implemented that displays (in no particular order, other than alphabetically) a table of all current entries and whether or not the system recognizes it as correctly formatted. This checker runs at 30 minutes after each hour.
Also, you only can submit (or overwrite) the yourusername.txt file for your validation submission as often as you’d like. However, only the most current submission at 23:59:59 US PDT on August 12, 2022 (August 13, 2022, 06:59:59 UTC), will be scored for the competition.