CHAOS Banner
Join

News & FAQ

You may find current updates and answers of frequently asked questions in this page.

NEWS

1) CHAOS data has a DOI number now. You can use it in your scientific work with citation.

https://doi.org/10.5281/zenodo.3362844

2) CHAOS is featured at the December issue of Computer Vision News Magazine!

https://www.rsipvision.com/ComputerVisionNews-2018December/16/

FAQ

0) Questions regarding the test data:

  • I need ground truths/references of the test data. How can I get it?
  • Can you privately share with me? I promise I will not share.
  • It is so important for our research. Can you make an exception?

Answer: Ground truths of the test data will never be shared publicly. Such requests will be ignored.

1) Is it possible to join and submit result to CHAOS?

Yes, online submission is opened. You may find further info from main page.

2) Why my "join request" is rejected or not accepted?

The challenge contains human data which need certain ethical permissions. Only educational and/or non-commercial usage is allowed. Therefore, we are trying to avoid anonymous users in order to keep a true record of interested participants. That is why using official e-mail account of the university/institution/company is strongly recommended. If the organizers cannot verify your e-mail address on a related web site, then the access to the data set is not granted.

3) Can I obtain the ground truths of the test data?

No. The ground truth of the test data will never be published. Only training sets include ground truths.

4) Is it possible to use CHAOS data in another academic study?

Yes, CHAOS data is shared under  Creative Commons Attribution Non Commercial Share Alike 4.0 International license. It is possible to use it with giving appropriate credit, provide a DOI link of the data, and indicate if changes were made.

5) Is it possible to download evaluation code?

Yes. You can download from https://github.com/emrekavur/CHAOS-evaluation for MATLAB, Python and Julia languages.

6) When can I learn result of my submission?

The evaluation is handled by grand-challenge.org's servers.  Results of automatic evaluation will be added to the Results page in a short time.

7) I want to submit my results for evaluation but I do not want them published on the leaderboard. Can you only send my scores to me via e-mail?

No, it is not possible. Each evaluated submission will be automatically published on the leaderbord regardless of score.

8) I do not like my score (or one of my score) and I want it to be cleared from the leaderboard. Is it possible?

Due to the rules, every evaluated submission will be automatically published, so it is not possible. Only if you think you have a strong reason to remove your result, you can contact us and explain the situation.

9) My DICE scores are very high however my final score is lower. How is it possible? Is the evaluation fair and correct? Why do you use many metrics? I saw many papers/works that only use DICE metric. I think DICE is enough, isn't it?

Segmentation of medical imaging is used for clinical operations. Hence, tolerance of error is fewer for clinical aspects. There is no single metric that evaluates 3D segmented data completely and fairly in terms of clinically acceptable results. That is why combination of different metrics is used for calculating the final score. This approach was used and is still being used in many respectable segmentation challenges.

10) Why is it necessary to prepare PDF file for each submission?

The main goals of challenges are to provide a unique dataset and up-to-date information about the problem. In CHAOS, we are trying to build a significant database about the abdomen organ segmentation to help the scientist all over the world over years. This database includes not only scores but also methods behind these scores. That is how the challenge can contribute to the literature and scientists can use this information for their scientific works.

11) Is it possible to re-use previously uploaded supplementary file?

It is possible if the algorithm of new submission is very similar with previous one. However, we strongly encourage the participants to prepare new file that indicates changes even they are minor. We would like to provide more information as much as possible to all visitors.

12) What if someone uses multiple usernames/nicknames?

The all results of them will be deleted from results page immediately when they are detected.

13) How can I see metric outputs and scores of each task/case.

The scoreboard shows mean position of all tasks by default. If you would like see scores of individual tasks, you may click "Additional Metrics" or "Show all metrics" button.

If you would like to examine scores set by set, just click on any score in results page. Detailed scores will be accessible in a new page. 

  Last update: 12/10/2019