Title: The Future of Medical Imaging: Introducing ScribblePrompt

As a PhD student in the Clinical and Applied Machine Learning Group at MIT CSAIL, I have always been fascinated by the potential of artificial intelligence to revolutionize medical imaging. Recently, my team and I have made significant progress in developing a tool called ScribblePrompt, designed to help clinicians and medical researchers annotate medical images more quickly and efficiently.

ScribblePrompt is an interactive AI tool that enables annotators to work with an AI model to complete the segmentation of different regions within medical images. This process is crucial for both biomedical research and clinical care, but it is often time-consuming and tedious, especially when researchers are trying to segment new regions of interest. Our tool speeds up this process by providing a flexible interface that allows users to interact with the model in various ways, including boundary boxes, clicks, or scribbles.

One of the key features of ScribblePrompt is its ability to generalize to new segmentation tasks and adapt to different types of user interactions. To achieve this, we developed new algorithms that simulate diverse and varied user interactions, including scribbles, allowing us to create a model that can learn from both real and synthetic data. Our database consists of over 50,000 scans covering 77 different datasets of biomedical image segmentation tasks, which enables the system to generalize to new tasks without retraining.

In the online demo of ScribblePrompt, I uploaded an abdominal CT image for users to segment and interact with the model. The annotator provides an image they want to segment and then draws scribbles, clicks, or bounding boxes to indicate the region of interest. With negative scribbles, users can clean up the prediction, and ScribblePrompt automatically updates and produces a new prediction. In this cardiac MRI, ScribblePrompt is able to segment the left and right ventricle, which are commonly labeled regions in our training data.

The model's ability to segment other regions that might not correspond to any of our training data is also noteworthy. For instance, scribbles are quite helpful when segmenting large and complex shapes like the femur in this X-ray. We can use scribbles iteratively, first to get an initial prediction and then to make corrections. As we provide more interactions, ScribblePrompt updates its prediction, allowing us to refine our output exactly.

Existing methods often fail when segmenting thin structures like veins and retinal images. However, with a few clicks, ScribblePrompt is able to complete this challenging task. We have an online web app where users can try the model for themselves. The user starts by uploading an image they want to segment and then providing different interactions, such as bounding boxes, clicks, or scribbles on the image to indicate the region of interest. The model produces a prediction, and if it doesn't get it right the first time, the user can add more interactions and give corrections to the model, which will refine its predictions.

Developing ScribblePrompt was not without its challenges. One of our biggest hurdles was training a model that could generalize to new segmentation tasks. We found that the model would often memorize certain rare types of images to always segment the same thing. To address this issue, we trained on both real and synthetic labels, including these synthetic labels during training, which forced the model to pay attention to interactions so that it couldn't memorize to always predict the same thing given a certain type of image.

ScribblePrompt is already having a significant impact in helping researchers annotate medical images faster. We have several ongoing collaborations with local researchers and are working on continuing to refine the model and system based on user feedback. ScribblePrompt enables users to segment any region of any medical image, and we believe this technology will enable new applications in clinical care and accelerate medical research by allowing users to annotate medical images efficiently.

In conclusion, ScribblePrompt is a groundbreaking tool that has the potential to revolutionize the field of medical imaging. Our interactive AI model is designed to help clinicians and researchers work more efficiently, and its ability to generalize to new segmentation tasks and adapt to different types of user interactions makes it an invaluable asset for any medical imaging application. We are excited to see how this technology accelerates medical research and enables new applications in clinical care, and we look forward to continuing our work on refining ScribblePrompt to make it even more effective.

"WEBVTTKind: captionsLanguage: en(logo whooshing)- I'm Hallee Wong.I'm a PhD studentin the Clinical and AppliedMachine Learning Groupat MIT CSAIL.My research focuses onAI for medical imaging.Today I'm excited to tell youabout a project called ScribblePrompt.ScribblePrompt is a tooldesigned to help cliniciansand medical researchersannotate medical images more quickly.In medical imaging,segmenting different regions of the imageis a core part of both biomedical researchand clinical care,but this process is reallytime-consuming and tedious,especially when researchersare trying to segmentnew regions of interest.ScribblePrompt speeds up this processby providing an interactive AI toolwhere the annotator works with an AI modelto complete the segmentation.With ScribblePrompt, we designed a systemthinking about usabilityand practicality from the beginning.ScribblePrompt is flexibleto a lot of differenttypes of interactions.Users can use boundaryboxes, clicks, or scribbles.It's designed to generalizeto new segmentation tasks,and it's also really fast.In ScribblePrompt, wedeveloped new algorithmsfor simulating very diverseand varied user interactions,including scribbles,so we could develop amodel that's amenableto multiple differenttypes of user interactions.With ScribblePrompt, we designed the modelto generalize to new segmentation tasks.To achieve this, we builta really large databaseof over 50,000 scanscovering 77 different data setsof biomedical image segmentation tasks.In addition to trainingon all of this real data,we also generated synthetic labels.So during training,the model was forced to learnto segment a lot of differentobjects in the medical images.This enabled the systemto be able to generalizeto new segmentation tasksat inference time without retraining.In the online demo hereI've uploaded an abdominal CT.To interact with the model,users provide an imagethey'd like to segmentand then draw scribbles,clicks, or bounding boxes,or a combination of all three,to indicate the region of interest.Here I can use negative scribblesto clean up the prediction.And now I'm using scribblesto segment the same object.The model automatically updatesand produces a prediction.In this cardiac MRI,ScribblePrompt is able to segmentthe left and right ventricle,which are commonly labeledregions in our training data.The model's also ableto segment other regionsthat might be of interest to researchers,even though they don't correspondto any of our training data.Scribbles are quite helpfulwhen segmenting large and complex shapes,like the femur in this X-ray.We can use scribbles iteratively,first to get an initial predictionand then to make corrections.So here I provide some rough scribblesto get a first prediction,but then I can add additional interactionsto clean up the edges.As you provide more interactions,ScribblePrompt updates its prediction,so you can keep interactingto get exactly the output that you want.Existing methods often failwhen segmenting thin structures,like veins and retinal images.With a few clicks, ScribblePrompt is ableto complete this challenging task.We have an online web appwhere you can try the model for yourself.The user would startby uploading an imagethat they wanna segmentand then providing different interactions,like bounding boxes, clicks,or scribbles on the image,to indicate the region they'reinterested in segmenting.The model produces a prediction,and if it doesn't getit right the first time,they can add more interactionsand give corrections to the model,which will refine its predictions.Developing ScribblePromptwasn't without its challenges.One of our biggest hurdleswas training a modelthat would be able to generalizeto new segmentation tasks.For example, we found that the modelwould often memorize forcertain rare types of imagesto always segment the same thing.In many retinal image data sets,we only had labels for the veins,so the model would learn everytime it saw a retinal image,just predict to segment the veins.So we found it was usefulto train on both a mix of real labelsas well as synthetic ones.Including these syntheticlabels during trainingforced the model to payattention to the interactionsso that it couldn't memorizeto always predict the same thinggiven a certain type of image.ScribblePrompt is alreadyhaving a significant impactin helping researchersannotate medical images faster.We have several ongoing collaborationswith local researchersand we're working on continuingto refine the model andsystem based on user feedback.ScribblePrompt enables usersto segment any regionof any medical image,and we believe this technologywill enable new applicationsin clinical careand accelerate medical researchby allowing users to annotatemedical images efficiently.Thanks for taking the time tolearn about ScribblePrompt.We're excited to see how this technologyaccelerates medical researchand enables new applicationsin clinical care.\n"