Use case 4 (Icometrix / VITO)

AI-based clinical decision support systems for MS monitoring



A major enabler but also a main bottleneck in the development of medical image analysis software such as icobrain is the availability of high quality “ground truth” labels that can be used to train pattern recognition modules in a supervised fashion.

icometrix’s flagship product icobrain ms performs several such pattern recognition tasks: semantic segmentation of brain tissues (white matter, gray matter) and brain structures (such as hippocampus, brain stem, ventricles, ...) and segmentation of white matter lesions that are characteristic of multiple sclerosis.

Early versions of icobrain ms performed these tasks using mainly traditional (unsupervised) image processing techniques, which were relatively slow and whose performance plateaued at a level that left much to be desired in clinical practice.

Over the course of the Vivaldy project, icobrain ms has evolved into a deep learning-powered software, in which inference is sped up considerably, and in which incremental improvements are unlocked by supplying newly labeled training data. Furthermore, the data-based paradigm in deep learning greatly improves flexibility, in this case the possibility to explicitly promote consistent accuracy across various MRI scanners (which is an Achilles heel of many existing softwares).

We demonstrate the following innovations to the icobrain ms software, and the main improvements they have led to:


impacted KPI

annotation tool, coined “minimally interactive adjustments (MIA)”

  • enable fast and accurate generation of new labeled data or refinement of existing labels
  • increased accuracy after retraining with novel training data

transition from a traditional image processing pipeline to a hybrid pipeline utilizing both traditional techniques as well as deep learning, and ultimately to a fully deep learning-based pipeline

  • increased accuracy for the segmentation tasks
  • reduced processing time (serving time-sensitive users)

novel training data augmentation technique used by the deep learning modules, which mimics realistic scanner-to-scanner variations in image contrast

  • improved consistency of the software’s volumetric measurements across different MRI scanners (greatly facilitating clinical interpretation)


“Minimally interactive adjustments (MIA)” tool to generate high quality training data

To enable a transition into a largely deep learning-based software, a sufficiently large and high quality training set with labeled data is required. The creation of a training set for learning the segmentation of brain tissues and structures (with tens of different structures to be identified!) is a formidable task. icometrix managed the complexity of this task by adopting a two-step approach:

  • In step 1, the predicted segmentation masks for all target structures are obtained for the training cases via an old version of icobrain. These predictions are of moderate quality (“silver” ground truth) and need to be refined.
  • In step 2, using a novel annotation tool (coined “MIA”), in-house experts refine the predicted segmentation masks for all structures across all training cases. The final segmentation masks are of high quality and are fit for use as training data for a deep learning model.


Figure 1: The MIA annotation tool integrates a DICOM viewer with many options (e.g. brush with adjustable size) for correcting labeled segmentation masks of brain structures.


Transitioning to a fully deep learning-based pipeline

The first upgrade to the icobrain ms software performed under the Vivaldy project consisted in moving to a hybrid lesion segmentation module (superseding the original, unsupervised module for lesion segmentation based on classical image processing):


Figure 2: icobrain ms has been upgraded from an (initially) traditional image processing pipeline based on unsupervised segmentation algorithms, to a hybrid pipeline using also supervised deep learning (shown), to ultimately be upgraded to a fully deep learning-powered pipeline.


Segmentation of brain tissues (gray and white matter) and clinically relevant structures (such as the hippocampus) was at this stage still done via classical techniques such as expectation-maximization (EM) and registration to multiple brain atlases, which are iterative, slow techniques.

Via the newly curated training data obtained with the in-house MIA tool, dedicated deep learning modules were developed and trained for both the lesion segmentation and the brain tissues and structures segmentation tasks. This further improved accuracy and obviated the need for many of the slower computation steps described above. Indeed, in production, the inference time of these deep learning blocks is comparatively small: the upgraded pipeline processes scans 40% faster, on average.


Scan contrast augmentation to improve across-scanner consistency

A serious limitation of many existing software packages for MRI image analysis is that the quantitative measurements are biased by the physical machine on which the MRI scan was acquired (since controllable scanner settings but also uncontrollable scanner properties, such as magnetic field strength and homogeneity, all influence the resulting image contrast and noise properties, and hence also derived measurements by automated techniques).

In clinical practice, it is vital, however, that measurements can be compared across scanners (e.g. to compare a patient to a normative reference population). icometrix developed a novel data augmentation technique that increases the variability of the training data by mimicking realistic contrast variations of the scans for a fixed anatomy.



Figure 3: Augmenting the training data by modulating the image contrast in the MR scans helps the deep learning models to generalize across multiple scanner models.


The use of such contrast augmentation during the training of the deep learning models improved its consistency across scanners.


Figure 4: After training with an augmentation technique that mimics scanner-to-scanner variations, the newer icobrain pipeline (left) boasts a better consistency than earlier versions (right). Namely, differences in measured volume of several brain structures between repeated scans of the same subject in the same scanner (“intra”) but also in different scanners (“inter”) are smaller after training the models with a novel augmentation strategy.



By closing a feedback loop between the training of DL modes and refinement of training data, the icobrain ms software underwent large changes that improved its accuracy, consistency and speed.


Figure 5: Transforming icobrain ms into a pipeline mainly relying on deep learning helped to push performance in ways that were previously hard to achieve. Notably, via the iterative refinement of (new) training data and via realistic training data augmentations, the models’ accuracy and ability to generalize is improved.


All these innovations combined greatly improved the utility of the icobrain ms software for end users.

This is illustrated by a testimonial by neuroradiologist Dr. Van Goethem (affiliated to the AZ Sint-Niklaas hospital) about the impact of the new icobrain ms versions in clinical practice:

“Since I am a neuroradiologist, I'm mainly interested in using the application for multiple sclerosis and dementia. I use these applications every day in the AZ Sint-Niklaas. We have several patients coming for multiple sclerosis follow up and the software of icometrix allows us to do volumetric measurements of the brain and also look at lesion load. Moreover, the system retains the previous examination and it's very useful in comparing the new examination with the previous one where the icobrain ms will give you new lesions, lesions that are enlarged and lesions that have lost volume. It's very easy for radiologists to have these features to help us to identify new lesions, especially with patients that have a lot of lesions. Also, quantifying the lesion load is very important because it's a measure that can be followed in time, which is very useful for us as radiologists, but of course, very useful, especially for the neurologist, and it's very important for the patients.”

This video explains use case 3: AI based clinical decision support for radiology, surgical and dermatology system