Image annotation remains a significant bottleneck in image analysis pipelines. In research settings especially, annotating large image corpora demands substantial effort, often forcing teams to compromise on quality by resorting to coarser methods like point annotations rather than full outlines.
To address this, we collaborated with the Helmholtz Institute for Functional Marine Biodiversity (HIFMB) to develop IQUANA (Image QUantification and ANAlysis), an annotation platform that integrates state-of-the-art computer vision models to accelerate annotation, raise quality, and free researchers from tedious manual work.
After uploading a dataset and defining a label scheme, users can annotate images with several modes of AI assistance:
- Prompted Object Segmentation — Models like SAM 2 enable rapid annotation of individual objects via point, bounding box, or polygon prompts.
- Prompted Concept Segmentation — SAM 3 extends this by annotating all instances of a given concept (e.g., cat) from a noun prompt and a few exemplars. While powerful in general domains, performance can degrade on niche or specialized datasets.
- Instance Segmentation — Models such as DETR are trained on-the-fly to fully automate annotation of entire images, with the goal of generating a dataset-wide pre-annotation that users simply review and correct.
💡 Active Research Questions
- Positive-Unlabelled Instance Segmentation: Can models learn on partial instance annotations to complete instance segmentations? Following Kadir et al.: how can we select the most informative samples?
- In-context Instance Segmentation: Can we provide context for models to properly annotate all instances in an image?
- Hierarchical / Graph Labels: Can we model a more informative label space to improve downstream analysis and model performance?
- Impact of AI models on users: How does IQUANA and its models affect users’ efficiency, user experience, and trust in AI?
Beyond AI-assisted labelling, IQUANA features a hierarchical label space that enables rich downstream analysis. Morphological features, for instance, can be annotated directly within a parent organism annotation, capturing object relationships in the database. This not only supports fine-grained relational queries but can also improve model training — as demonstrated in Deep Hierarchical Segmentation.
IQUANA also serves as a testbed for new computer vision models. Its modular architecture makes it straightforward to integrate and evaluate novel models directly within a real annotation workflow, providing immediate, practical feedback on model performance, usability, and domain fit. Rather than benchmarking in isolation, researchers can observe how a model behaves in the hands of actual users on real datasets, making IQUANA a valuable environment for applied computer vision research.
Planned user evaluations will assess IQUANA’s impact on efficiency, user experience, and trustworthiness, with study participants drawn from both marine ecology and medical imaging research.
ℹ️ Open Source
IQUANA is fully open-source and self-hostable. View the code at github.com/Iquana-tool.
References
- Ravi, N. et al. “SAM 2: Segment anything in images and videos.” ICLR 2025.
- Carion, N. et al. “SAM 3: Segment anything with concepts.” arXiv:2511.16719 (2025).
- Carion, N. et al. “End-to-end object detection with transformers.” ECCV. Springer, 2020.
- Kadir, M.A. et al. “Partial Image Active Annotation (PIAA).” Künstl Intell 38, 133–144 (2024). doi:10.1007/s13218-024-00849-6
- Li, L. et al. “Deep hierarchical semantic segmentation.” CVPR 2022.
