News & FAQ

You may find current updates and answers of frequently asked questions in this page.


1) Test data and online submission are available now!. Please check our download page.

2) Results of CHAOS has been announced!

3) CHAOS is featured at the December issue of Computer Vision News Magazine!


1) When will the challenge start?

CHAOS challenge will start during the IEEE International Symposium on Biomedical Imaging (ISBI) on April 8-11, 2019 Venice, ITALY. The exact date, time and place of the challenge will be announced once determined by the ISBI organizing comitee.

2) How can I participate the challenge in ISBI?

First, the candidates need to register to the challenge from this website. Also, the participants will be required to register and join ISBI conference. They have to run their code at the challenge day.

3) Do I need to register ISBI to join the challenge?

Yes. According to ISBI rules, participants have to register ISBI conference and pay the registration fee.

4) Is there additional fee to join challenge?

No. The registration to the ISBI conference includes the challenges.

5) I have some results now. How can I submit?

Since the challenge will start at ISBI, the earliest time for the test data to be available is 11th of April 2019. Before that time, the submission of the results is not possible.

6) Why my "join request" is rejected or not accepted?

The challenge contains human data which need certain ethical permissions. Only educational and/or non-commercial usage is allowed. Therefore, ee are trying to avoid anonymous users in order to keep a true record of interested participants That is why using official e-mail account of the university/institution/company is strongly recommended. If the organizers cannot verify your e-mail address on a related web site, then the access to the data set is not granted.

7) Will the dataset available in only challenge day in ISBI?

No. A similar approach in previous grand challenges, such as SLIVER07 is preferred. The dataset and the challenge will be available after the challenge day. New participants may submit their results to the system for evaluation. Also previous participants will have the option to update their new results.The actual scores will be published in this website.

8) Can I obtain the ground truths of the test data?

No. The ground truth of the test data will never be published. Only training sets include ground truths.

9) I think, I found some problems in the published dataset. What should I do?

The ground truths are being generated by consensus of imaging specialists and radiologists. Please do not hesitate contact us if you think that you find any mistake about the data. All contributions are valuable and welcome.

10) Can I download the evaluation code? / Where can I download evaluation code?

The evaluation code will be released after the ISBI 2019 conference.

11) What are the differences between the five tasks?

1. Liver Segmentation (CT & MRI): This is also called "cross modality" and it is simply based on using a single system, which can segment liver from both CT and MRI. For instance, the training and test sets of a machine learning approach would have images from both modality without explicitly feeding the model with corresponding information. A unique study about this is reference below and this task is one of the most interesting tasks of the challenge (Keep in mind that any kind of ensemble or fusion of individual systems (i.e. two models, one working on CT and the other on MRI and one is selected by some decision criteria) would not be valid for this category. They can be evaluated as individual systems at Tasks 2 and 3.

Valindria, V. V., Pawlowski, N., Rajchl, M., Lavdas, I., Aboagye, E. O., Rockall, A. G., ... & Glocker, B. (2018, March). Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 547-556). IEEE.
2. Liver Segmentation (CT only): This is mostly a regular task of liver segmentation (such as SLIVER07). This task is easier than SLIVER07 as it only contains healthy livers aligned at the same direction and patient position. However, the challenging part is the enhanced vascular structures (portal phase) due to contrast injection. One of the biggest challenges in this case is the "maximum symmetric surface distance", which measures errors for surgical precision (the datasets are from transplantation donors, who will undergo a very complicated surgery). For instance, an inter-observer score was 95 with maxSSD while it was 97.4 without it. Here, the "score" refers to the average of Volume Overlap Error (or Dice), Mean SSD, RMS SSD, Max SSD and Relative Volume Error (PS. The final evaluation strategy will be announced at the following week)

3. Liver Segmentation (MRI only): Similar to "Task 2", this is also a regular task of liver segmentation, but include two different pulse sequences: T1-DUAL and T2-SPIR. Moreover, T1-DUAL has two forms (in and out phase). The developed system should work on both sequences without explicit knowledge about the pulse sequence. In this task, ensemble or fusion of individual systems (i.e. two models, one working on T1-DUAL and the other on  T2-SPIR ) are allowed. However, there might be a penalty at the scoring.

4. Segmentation of abdominal organs (CT & MRI): In this task, the interesting part is that CT datasets have only liver, but the MRI datasets have four annotated abdominal organs Thus, in addition to the "cross modality"  task described in "task 1", here the output of a system (i.e. single output vs four) should change based on the modality.  

5. Segmentation of abdominal organs (MRI only): The same task given in "Task 3" but extended to four abdominal organs.