2020 |
Journal Articles |
Biswas, Rajarshi; Barz, Michael; Sonntag, Daniel Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking Journal Article KI - Künstliche Intelligenz German Journal of Artificial Intelligence, 2020, ISSN: 0933-1875. @article{Biswas2020, title = {Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking}, author = {Rajarshi Biswas and Michael Barz and Daniel Sonntag}, url = {https://doi.org/10.1007/s13218-020-00679-2}, doi = {10.1007/s13218-020-00679-2}, issn = {0933-1875}, year = {2020}, date = {2020-07-08}, journal = {KI - Künstliche Intelligenz German Journal of Artificial Intelligence}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
Inproceedings |
Barz, Michael; Altmeyer, Kristin; Malone, Sarah; Lauer, Luisa; Daniel Sonntag, Digital Pen Features Predict Task Difficulty and User Performance of Cognitive Tests Inproceedings Forthcoming Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, pp. 10, ACM, Forthcoming. @inproceedings{Barz2020b, title = {Digital Pen Features Predict Task Difficulty and User Performance of Cognitive Tests}, author = {Michael Barz and Kristin Altmeyer and Sarah Malone and Luisa Lauer and Daniel Sonntag}, url = {https://doi.org/10.1145/3340631.3394839 https://iml.dfki.de/wp-content/uploads/2020/04/feature_table.pdf}, doi = {10.1145/3340631.3394839}, year = {2020}, date = {2020-07-01}, booktitle = {Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization}, pages = {10}, publisher = {ACM}, series = {UMAP '20}, keywords = {}, pubstate = {forthcoming}, tppubtype = {inproceedings} } |
Barz, Michael; Stauden, Sven; Sonntag, Daniel Visual Search Target Inference in Natural Interaction Settings with Machine Learning Inproceedings Symposium on Eye Tracking Research and Applications, Association for Computing Machinery, Stuttgart, Germany, 2020, ISBN: 9781450371339. @inproceedings{Barz2020a, title = {Visual Search Target Inference in Natural Interaction Settings with Machine Learning}, author = {Michael Barz and Sven Stauden and Daniel Sonntag}, url = {https://doi.org/10.1145/3379155.3391314 https://github.com/DFKI-Interactive-Machine-Learning/STI-Dataset}, doi = {10.1145/3379155.3391314}, isbn = {9781450371339}, year = {2020}, date = {2020-06-01}, booktitle = {Symposium on Eye Tracking Research and Applications}, publisher = {Association for Computing Machinery}, address = {Stuttgart, Germany}, series = {ETRA ’20}, abstract = {Visual search is a perceptual task in which humans aim at identifying a search target object such as a traffic sign among other objects. Search target inference subsumes computational methods for predicting this target by tracking and analyzing overt behavioral cues of that person, e.g., the human gaze and fixated visual stimuli. We present a generic approach to inferring search targets in natural scenes by predicting the class of the surrounding image segment. Our method encodes visual search sequences as histograms of fixated segment classes determined by SegNet, a deep learning image segmentation model for natural scenes. We compare our sequence encoding and model training (SVM) to a recent baseline from the literature for predicting the target segment. Also, we use a new search target inference dataset. The results show that, first, our new segmentation-based sequence encoding outperforms the method from the literature, and second, that it enables target inference in natural settings.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Visual search is a perceptual task in which humans aim at identifying a search target object such as a traffic sign among other objects. Search target inference subsumes computational methods for predicting this target by tracking and analyzing overt behavioral cues of that person, e.g., the human gaze and fixated visual stimuli. We present a generic approach to inferring search targets in natural scenes by predicting the class of the surrounding image segment. Our method encodes visual search sequences as histograms of fixated segment classes determined by SegNet, a deep learning image segmentation model for natural scenes. We compare our sequence encoding and model training (SVM) to a recent baseline from the literature for predicting the target segment. Also, we use a new search target inference dataset. The results show that, first, our new segmentation-based sequence encoding outperforms the method from the literature, and second, that it enables target inference in natural settings. |
2019 |
Inproceedings |
Biswas, Rajarshi; Mogadala, Aditya; Barz, Michael; Sonntag, Daniel; Klakow, Dietrich Automatic Judgement of Neural Network-Generated Image Captions Inproceedings Martin-Vide, Carlos; Purver, Matthew; Pollak, Senja (Ed.): Statistical Language and Speech Processing - 7th International Conference, {SLSP} 2019, Ljubljana, Slovenia, October 14-16, 2019, Proceedings, pp. 261 – 272 , Springer, 2019, ISBN: 978-3-030-31371-5. @inproceedings{Biswas19, title = {Automatic Judgement of Neural Network-Generated Image Captions}, author = {Rajarshi Biswas and Aditya Mogadala and Michael Barz and Daniel Sonntag and Dietrich Klakow}, editor = {Carlos Martin-Vide and Matthew Purver and Senja Pollak}, url = {https://doi.org/10.1007/978-3-030-31372-2_22}, doi = {10.1007/978-3-030-31372-2_22}, isbn = {978-3-030-31371-5}, year = {2019}, date = {2019-09-27}, booktitle = {Statistical Language and Speech Processing - 7th International Conference, {SLSP} 2019, Ljubljana, Slovenia, October 14-16, 2019, Proceedings}, volume = {11816}, pages = {261 -- 272 }, publisher = {Springer}, series = {Lecture Notes in Computer Science}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
Prange, Alexander; Niemann, Mira; Latendorf, Antje; Steinert, Anika; Sonntag, Daniel Multimodal Speech-based Dialogue for the Mini-Mental State Examination Inproceedings Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. CS13:1–CS13:8, ACM, Glasgow, Scotland Uk, 2019, ISBN: 978-1-4503-5971-9. @inproceedings{Prange:2019:MSD:3290607.3299040, title = {Multimodal Speech-based Dialogue for the Mini-Mental State Examination}, author = {Alexander Prange and Mira Niemann and Antje Latendorf and Anika Steinert and Daniel Sonntag}, url = {http://doi.acm.org/10.1145/3290607.3299040}, doi = {10.1145/3290607.3299040}, isbn = {978-1-4503-5971-9}, year = {2019}, date = {2019-01-01}, booktitle = {Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems}, pages = {CS13:1--CS13:8}, publisher = {ACM}, address = {Glasgow, Scotland Uk}, series = {CHI EA '19}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
Kalimuthu, Marimuthu; Barz, Michael; Sonntag, Daniel Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings Inproceedings Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP), pp. 1–10, ACL, Florence, Italy, 2019. @inproceedings{kalimuthu-etal-2019-incremental, title = {Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings}, author = {Marimuthu Kalimuthu and Michael Barz and Daniel Sonntag}, url = {https://www.aclweb.org/anthology/W19-4601}, year = {2019}, date = {2019-01-01}, booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP)}, pages = {1--10}, publisher = {ACL}, address = {Florence, Italy}, abstract = {We study the problem of incremental domain adaptation of a generic neural machine translation model with limited resources (e.g., budget and time) for human translations or model training. In this paper, we propose a novel query strategy for selecting ``unlabeled'' samples from a new domain based on sentence embeddings for Arabic. We accelerate the fine-tuning process of the generic model to the target domain. Specifically, our approach estimates the informativeness of instances from the target domain by comparing the distance of their sentence embeddings to embeddings from the generic domain. We perform machine translation experiments (Ar-to-En direction) for comparing a random sampling baseline with our new approach, similar to active learning, using two small update sets for simulating the work of human translators. For the prescribed setting we can save more than 50% of the annotation costs without loss in quality, demonstrating the effectiveness of our approach.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We study the problem of incremental domain adaptation of a generic neural machine translation model with limited resources (e.g., budget and time) for human translations or model training. In this paper, we propose a novel query strategy for selecting ``unlabeled'' samples from a new domain based on sentence embeddings for Arabic. We accelerate the fine-tuning process of the generic model to the target domain. Specifically, our approach estimates the informativeness of instances from the target domain by comparing the distance of their sentence embeddings to embeddings from the generic domain. We perform machine translation experiments (Ar-to-En direction) for comparing a random sampling baseline with our new approach, similar to active learning, using two small update sets for simulating the work of human translators. For the prescribed setting we can save more than 50% of the annotation costs without loss in quality, demonstrating the effectiveness of our approach. |
2018 |
Journal Articles |
Sonntag, Daniel; Profitlich, Hans-Jürgen An architecture of open-source tools to combine textual information extraction, faceted search and information visualisation Journal Article CoRR, abs/1810.12627 , 2018. @article{information_extraction-search_visualisation-arxiv, title = {An architecture of open-source tools to combine textual information extraction, faceted search and information visualisation}, author = {Daniel Sonntag and Hans-Jürgen Profitlich}, url = {https://arxiv.org/abs/1810.12627}, year = {2018}, date = {2018-10-30}, journal = {CoRR}, volume = {abs/1810.12627}, abstract = {This article presents our steps to integrate complex and partly unstructured medical data into a clinical research database with subsequent decision support. Our main application is an integrated faceted search tool, accompanied by the visualisation of results of automatic information extraction from textual documents. We describe the details of our technical architecture (open-source tools), to be replicated at other universities, research institutes, or hospitals. Our exemplary use cases are nephrology and mammography. The software was first developed in the nephrology domain and then adapted to the mammography use case. We report on these case studies, illustrating how the application can be used by a clinician and which questions can be answered. We show that our architecture and the employed software modules are suitable for both areas of application with a limited amount of adaptations. For example, in nephrology we try to answer questions about the temporal characteristics of event sequences to gain significant insight from the data for cohort selection. We present a versatile time-line tool that enables the user to explore relations between a multitude of diagnosis and laboratory values.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This article presents our steps to integrate complex and partly unstructured medical data into a clinical research database with subsequent decision support. Our main application is an integrated faceted search tool, accompanied by the visualisation of results of automatic information extraction from textual documents. We describe the details of our technical architecture (open-source tools), to be replicated at other universities, research institutes, or hospitals. Our exemplary use cases are nephrology and mammography. The software was first developed in the nephrology domain and then adapted to the mammography use case. We report on these case studies, illustrating how the application can be used by a clinician and which questions can be answered. We show that our architecture and the employed software modules are suitable for both areas of application with a limited amount of adaptations. For example, in nephrology we try to answer questions about the temporal characteristics of event sequences to gain significant insight from the data for cohort selection. We present a versatile time-line tool that enables the user to explore relations between a multitude of diagnosis and laboratory values. |
Sonntag, Daniel Interactive Cognitive Assessment Tools: A Case Study on Digital Pens for the Clinical Assessment of Dementia Journal Article CoRR, abs/1810.04943 , 2018. @article{cognitive_assessment_tool-pen-study, title = {Interactive Cognitive Assessment Tools: A Case Study on Digital Pens for the Clinical Assessment of Dementia}, author = {Sonntag, Daniel}, url = {https://arxiv.org/abs/1810.04943}, year = {2018}, date = {2018-10-11}, journal = {CoRR}, volume = {abs/1810.04943}, abstract = {Interactive cognitive assessment tools may be valuable for doctors and therapists to reduce costs and improve quality in healthcare systems. Use cases and scenarios include the assessment of dementia. In this paper, we present our approach to the semi-automatic assessment of dementia. We describe a case study with digital pens for the patients including background, problem description and possible solutions. We conclude with lessons learned when implementing digital tests, and a generalisation for use outside the cognitive impairments field.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Interactive cognitive assessment tools may be valuable for doctors and therapists to reduce costs and improve quality in healthcare systems. Use cases and scenarios include the assessment of dementia. In this paper, we present our approach to the semi-automatic assessment of dementia. We describe a case study with digital pens for the patients including background, problem description and possible solutions. We conclude with lessons learned when implementing digital tests, and a generalisation for use outside the cognitive impairments field. |
Zacharias, Jan; Barz, Michael; Sonntag, Daniel A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces Journal Article CoRR, abs/1803.04818 , 2018. @article{DBLP:journals/corr/abs-1803-04818, title = {A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces}, author = {Jan Zacharias and Michael Barz and Daniel Sonntag}, url = {http://arxiv.org/abs/1803.04818}, year = {2018}, date = {2018-01-01}, journal = {CoRR}, volume = {abs/1803.04818}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
Prange, Alexander; Barz, Michael; Sonntag, Daniel A categorisation and implementation of digital pen features for behaviour characterisation Journal Article CoRR, abs/1810.03970 , 2018. @article{ink_features_arxiv, title = {A categorisation and implementation of digital pen features for behaviour characterisation}, author = {Alexander Prange and Michael Barz and Daniel Sonntag}, url = {https://arxiv.org/abs/1810.03970}, year = {2018}, date = {2018-01-01}, journal = {CoRR}, volume = {abs/1810.03970}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
Booklets |
Barz, Michael; Polzehl, Tim; Sonntag, Daniel Towards Hybrid Human-Machine Translation Services Booklet EasyChair Preprint no. 333, 2018. @booklet{EasyChair:333, title = {Towards Hybrid Human-Machine Translation Services}, author = {Michael Barz and Tim Polzehl and Daniel Sonntag}, url = {https://easychair.org/publications/preprint/wfm6}, doi = {10.29007/kw5h}, year = {2018}, date = {2018-07-09}, howpublished = {EasyChair Preprint no. 333}, month = {07}, keywords = {}, pubstate = {published}, tppubtype = {booklet} } |
Inproceedings |
Stauden, Sven; Barz, Michael; Sonntag, Daniel Visual Search Target Inference using Bag of Deep Visual Words Inproceedings Trollmann, Frank; Turhan, Anni-Yasmin (Ed.): KI 2018: Advances in Artificial Intelligence, pp. 297––304, Springer International Publishing, 2018, ISBN: 978-3-030-00111-7. @inproceedings{Stauden2018, title = {Visual Search Target Inference using Bag of Deep Visual Words}, author = {Sven Stauden and Michael Barz and Daniel Sonntag}, editor = {Frank Trollmann and Anni-Yasmin Turhan}, url = {https://link.springer.com/chapter/10.1007%2F978-3-030-00111-7_25}, doi = {10.1007/978-3-030-00111-7_25}, isbn = {978-3-030-00111-7}, year = {2018}, date = {2018-09-28}, booktitle = {KI 2018: Advances in Artificial Intelligence}, pages = {297----304}, publisher = {Springer International Publishing}, abstract = {Visual Search target inference subsumes methods for predicting the target object through eye tracking. A person intents to find an object in a visual scene which we predict based on the fixation behavior. Knowing about the search target can improve intelligent user interaction. In this work, we implement a new feature encoding, the Bag of Deep Visual Words, for search target inference using a pre-trained convolutional neural network (CNN). Our work is based on a recent approach from the literature that uses Bag of Visual Words, common in computer vision applications. We evaluate our method using a gold standard dataset. The results show that our new feature encoding outperforms the baseline from the literature, in particular, when excluding fixations on the target.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Visual Search target inference subsumes methods for predicting the target object through eye tracking. A person intents to find an object in a visual scene which we predict based on the fixation behavior. Knowing about the search target can improve intelligent user interaction. In this work, we implement a new feature encoding, the Bag of Deep Visual Words, for search target inference using a pre-trained convolutional neural network (CNN). Our work is based on a recent approach from the literature that uses Bag of Visual Words, common in computer vision applications. We evaluate our method using a gold standard dataset. The results show that our new feature encoding outperforms the baseline from the literature, in particular, when excluding fixations on the target. |
Barz, Michael; Daiber, Florian; Sonntag, Daniel; Bulling, Andreas Error-aware gaze-based interfaces for robust mobile gaze interaction Inproceedings Sharif, Bonita; Krejtz, Krzysztof (Ed.): Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA 2018, Warsaw, Poland, June 14-17, 2018, pp. 24:1–24:10, ACM, 2018. @inproceedings{DBLP:conf/etra/BarzDSB18, title = {Error-aware gaze-based interfaces for robust mobile gaze interaction}, author = {Michael Barz and Florian Daiber and Daniel Sonntag and Andreas Bulling}, editor = {Bonita Sharif and Krzysztof Krejtz}, url = {http://doi.acm.org/10.1145/3204493.3204536}, doi = {10.1145/3204493.3204536}, year = {2018}, date = {2018-06-14}, booktitle = {Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA 2018, Warsaw, Poland, June 14-17, 2018}, pages = {24:1--24:10}, publisher = {ACM}, abstract = {Gaze estimation error can severely hamper usability and performance of mobile gaze-based interfaces given that the error varies constantly for different interaction positions. In this work, we explore error-aware gaze-based interfaces that estimate and adapt to gaze estimation error on-the-fly. We implement a sample error-aware user interface for gaze-based selection and different error compensation methods: a naïve approach that increases component size directly proportional to the absolute error, a recent model by Feit et al. that is based on the two-dimensional error distribution, and a novel predictive model that shifts gaze by a directional error estimate. We evaluate these models in a 12-participant user study and show that our predictive model significantly outperforms the others in terms of selection rate, particularly for small gaze targets. These results underline both the feasibility and potential of next generation error-aware gaze-based user interfaces.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Gaze estimation error can severely hamper usability and performance of mobile gaze-based interfaces given that the error varies constantly for different interaction positions. In this work, we explore error-aware gaze-based interfaces that estimate and adapt to gaze estimation error on-the-fly. We implement a sample error-aware user interface for gaze-based selection and different error compensation methods: a naïve approach that increases component size directly proportional to the absolute error, a recent model by Feit et al. that is based on the two-dimensional error distribution, and a novel predictive model that shifts gaze by a directional error estimate. We evaluate these models in a 12-participant user study and show that our predictive model significantly outperforms the others in terms of selection rate, particularly for small gaze targets. These results underline both the feasibility and potential of next generation error-aware gaze-based user interfaces. |
Prange, Alexander; Barz, Michael; Sonntag, Daniel Medical 3D Images in Multimodal Virtual Reality Inproceedings Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion, Tokyo, Japan, March 07-11, 2018, pp. 19:1–19:2, ACM, 2018. @inproceedings{DBLP:conf/iui/PrangeBS18, title = {Medical 3D Images in Multimodal Virtual Reality}, author = {Alexander Prange and Michael Barz and Daniel Sonntag}, url = {http://doi.acm.org/10.1145/3180308.3180327}, doi = {10.1145/3180308.3180327}, year = {2018}, date = {2018-01-01}, booktitle = {Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion, Tokyo, Japan, March 07-11, 2018}, pages = {19:1--19:2}, publisher = {ACM}, abstract = {We present a multimodal medical 3D image system for radiologists in an virtual reality (VR) environment. Users can walk freely inside the virtual room and interact with the system using speech, going through patient records, and manipulate 3D image data with hand gestures. Medical images are retrieved from the hospital's Picture and Archiving System (PACS) and displayed as 3D objects inside VR. Our system incorporates a dialogue-based decision support system for treatments. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of real patient records, 3D DICOM radiology image data, and real-time therapy predictions in VR.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We present a multimodal medical 3D image system for radiologists in an virtual reality (VR) environment. Users can walk freely inside the virtual room and interact with the system using speech, going through patient records, and manipulate 3D image data with hand gestures. Medical images are retrieved from the hospital's Picture and Archiving System (PACS) and displayed as 3D objects inside VR. Our system incorporates a dialogue-based decision support system for treatments. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of real patient records, 3D DICOM radiology image data, and real-time therapy predictions in VR. |
Barz, Michael; Büyükdemircioglu, Neslihan; Surya, Rikhu Prasad; Polzehl, Tim; Sonntag, Daniel Device-Type Influence in Crowd-based Natural Language Translation Tasks Inproceedings Checco, Alessandro; Demartini, Gianluca; Gadiraju, Ujwal; Sarasua, Cristina; Aroyo, Lora; Dumitrache, Anca; Paritosh, Praveen; Quinn, Alex; Welty, Chris (Ed.): 1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing 2018, and CrowdBias'18: Disentangling the Relation Between Crowdsourcing and Bias Management, pp. 93–97, Aachen, 2018, ISSN: 1613-0073. @inproceedings{BarzEtAl:SAD_CROWDBIAS_HCOMP2018, title = {Device-Type Influence in Crowd-based Natural Language Translation Tasks}, author = {Michael Barz and Neslihan Büyükdemircioglu and Rikhu Prasad Surya and Tim Polzehl and Daniel Sonntag}, editor = {Alessandro Checco and Gianluca Demartini and Ujwal Gadiraju and Cristina Sarasua and Lora Aroyo and Anca Dumitrache and Praveen Paritosh and Alex Quinn and Chris Welty}, url = {http://ceur-ws.org/Vol-2276/}, issn = {1613-0073}, year = {2018}, date = {2018-01-01}, booktitle = {1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing 2018, and CrowdBias'18: Disentangling the Relation Between Crowdsourcing and Bias Management}, number = {2276}, pages = {93--97}, address = {Aachen}, series = {CEUR Workshop Proceedings}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
Niemann, Mira; Prange, Alexander; Sonntag, Daniel Towards a Multimodal Multisensory Cognitive Assessment Framework Inproceedings 31st IEEE International Symposium on Computer-Based Medical Systems, CBMS 2018, Karlstad, Sweden, June 18-21, 2018, pp. 24–29, 2018. @inproceedings{DBLP:conf/cbms/NiemannPS18, title = {Towards a Multimodal Multisensory Cognitive Assessment Framework}, author = {Mira Niemann and Alexander Prange and Daniel Sonntag}, url = {https://doi.org/10.1109/CBMS.2018.00012}, doi = {10.1109/CBMS.2018.00012}, year = {2018}, date = {2018-01-01}, booktitle = {31st IEEE International Symposium on Computer-Based Medical Systems, CBMS 2018, Karlstad, Sweden, June 18-21, 2018}, pages = {24--29}, crossref = {DBLP:conf/cbms/2018}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
2017 |
Journal Articles |
Schmidt, Danilo; Budde, Klemens; Sonntag, Daniel; Profitlich, Hans-Jürgen; Ihle, Matthias; Staeck, Oliver A novel tool for the identification of correlations in medical data by faceted search Journal Article Comp. in Bio. and Med., 85 , pp. 98–105, 2017. @article{DBLP:journals/cbm/SchmidtBSPIS17, title = {A novel tool for the identification of correlations in medical data by faceted search}, author = {Danilo Schmidt and Klemens Budde and Daniel Sonntag and Hans-Jürgen Profitlich and Matthias Ihle and Oliver Staeck}, url = {https://doi.org/10.1016/j.compbiomed.2017.04.011}, doi = {10.1016/j.compbiomed.2017.04.011}, year = {2017}, date = {2017-01-01}, journal = {Comp. in Bio. and Med.}, volume = {85}, pages = {98--105}, abstract = {This work focuses on the integration of multifaceted extensive data sets (e.g. laboratory values, vital data, medications) and partly unstructured medical data such as discharge letters, diagnostic reports, clinical notes etc. in a research database. Our main application is an integrated faceted search in nephrology based on information extraction results. We describe the details of the application of transplant medicine and the resulting technical architecture of the faceted search application.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This work focuses on the integration of multifaceted extensive data sets (e.g. laboratory values, vital data, medications) and partly unstructured medical data such as discharge letters, diagnostic reports, clinical notes etc. in a research database. Our main application is an integrated faceted search in nephrology based on information extraction results. We describe the details of the application of transplant medicine and the resulting technical architecture of the faceted search application. |
Sonntag, Daniel; Huber, Martin; Möller, Manuel; Ndiaye, Alassane; Zillner, Sonja; Cavallaro, Alexander Design and Implementation of a Semantic Dialogue System for Radiologists Journal Article CoRR, abs/1701.07381 , 2017. @article{DBLP:journals/corr/SonntagHMNZC17, title = {Design and Implementation of a Semantic Dialogue System for Radiologists}, author = {Daniel Sonntag and Martin Huber and Manuel Möller and Alassane Ndiaye and Sonja Zillner and Alexander Cavallaro}, url = {http://arxiv.org/abs/1701.07381}, year = {2017}, date = {2017-01-01}, journal = {CoRR}, volume = {abs/1701.07381}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
Sonntag, Daniel; Barz, Michael; Zacharias, Jan; Stauden, Sven; Rahmani, Vahid; Fóthi, Áron; Lőrincz, András Fine-tuning deep CNN models on specific MS COCO categories Journal Article CoRR, abs/1709.01476 , 2017. @article{DBLP:journals/corr/abs-1709-01476, title = {Fine-tuning deep CNN models on specific MS COCO categories}, author = {Daniel Sonntag and Michael Barz and Jan Zacharias and Sven Stauden and Vahid Rahmani and Áron Fóthi and András Lőrincz}, url = {http://arxiv.org/abs/1709.01476}, year = {2017}, date = {2017-01-01}, journal = {CoRR}, volume = {abs/1709.01476}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
Sonntag, Daniel Interakt - A Multimodal Multisensory Interactive Cognitive Assessment Tool Journal Article CoRR, abs/1709.01796 , 2017. @article{DBLP:journals/corr/abs-1709-01796, title = {Interakt - A Multimodal Multisensory Interactive Cognitive Assessment Tool}, author = {Daniel Sonntag}, url = {http://arxiv.org/abs/1709.01796}, year = {2017}, date = {2017-01-01}, journal = {CoRR}, volume = {abs/1709.01796}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
Inproceedings |
Barz, Michael; Poller, Peter; Sonntag, Daniel Evaluating Remote and Head-worn Eye Trackers in Multi-modal Speech-based HRI Inproceedings Mutlu, Bilge; Tscheligi, Manfred; Weiss, Astrid; Young, James E (Ed.): Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, Vienna, Austria, March 6-9, 2017, pp. 79–80, ACM, 2017. @inproceedings{DBLP:conf/hri/BarzPS17a, title = {Evaluating Remote and Head-worn Eye Trackers in Multi-modal Speech-based HRI}, author = {Michael Barz and Peter Poller and Daniel Sonntag}, editor = {Bilge Mutlu and Manfred Tscheligi and Astrid Weiss and James E Young}, url = {http://doi.acm.org/10.1145/3029798.3038367}, doi = {10.1145/3029798.3038367}, year = {2017}, date = {2017-03-09}, booktitle = {Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, Vienna, Austria, March 6-9, 2017}, pages = {79--80}, publisher = {ACM}, abstract = {Gaze is known to be a dominant modality for conveying spatial information, and it has been used for grounding in human-robot dialogues. In this work, we present the prototype of a gaze-supported multi-modal dialogue system that enhances two core tasks in human-robot collaboration: 1) our robot is able to learn new objects and their location from user instructions involving gaze, and 2) it can instruct the user to move objects and passively track this movement by interpreting the user's gaze. We performed a user study to investigate the impact of different eye trackers on user performance. In particular, we compare a head-worn device and an RGB-based remote eye tracker. Our results show that the head-mounted eye tracker outperforms the remote device in terms of task completion time and the required number of utterances due to its higher precision.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Gaze is known to be a dominant modality for conveying spatial information, and it has been used for grounding in human-robot dialogues. In this work, we present the prototype of a gaze-supported multi-modal dialogue system that enhances two core tasks in human-robot collaboration: 1) our robot is able to learn new objects and their location from user instructions involving gaze, and 2) it can instruct the user to move objects and passively track this movement by interpreting the user's gaze. We performed a user study to investigate the impact of different eye trackers on user performance. In particular, we compare a head-worn device and an RGB-based remote eye tracker. Our results show that the head-mounted eye tracker outperforms the remote device in terms of task completion time and the required number of utterances due to its higher precision. |
Sonntag, Daniel; Profitlich, Hans-Jürgen Integrated Decision Support by Combining Textual Information Extraction, Facetted Search and Information Visualisation Inproceedings Bamidis, Panagiotis D; Konstantinidis, Stathis Th.; Rodrigues, Pedro Pereira (Ed.): 30th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2017, Thessaloniki, Greece, June 22-24, 2017, pp. 95–100, IEEE Computer Society, 2017. @inproceedings{DBLP:conf/cbms/SonntagP17, title = {Integrated Decision Support by Combining Textual Information Extraction, Facetted Search and Information Visualisation}, author = {Daniel Sonntag and Hans-Jürgen Profitlich}, editor = {Panagiotis D Bamidis and Stathis Th. Konstantinidis and Pedro Pereira Rodrigues}, url = {https://doi.org/10.1109/CBMS.2017.119}, doi = {10.1109/CBMS.2017.119}, year = {2017}, date = {2017-01-01}, booktitle = {30th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2017, Thessaloniki, Greece, June 22-24, 2017}, pages = {95--100}, publisher = {IEEE Computer Society}, abstract = {This work focusses on our integration steps of complex and partly unstructured medical data into a clinical research database with subsequent decision support. Our main application is an integrated facetted search tool, followed by information visualisation based on automatic information extraction results from textual documents. We describe the details of our technical architecture (open-source tools), to be replicated at other universities, research institutes, or hospitals. Our exemplary use case is nephrology, where we try to answer questions about the temporal characteristics of sequences and gain significant insight from the data for cohort selection. We report on this case study, illustrating how the application can be used by a clinician and which questions can be answered.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } This work focusses on our integration steps of complex and partly unstructured medical data into a clinical research database with subsequent decision support. Our main application is an integrated facetted search tool, followed by information visualisation based on automatic information extraction results from textual documents. We describe the details of our technical architecture (open-source tools), to be replicated at other universities, research institutes, or hospitals. Our exemplary use case is nephrology, where we try to answer questions about the temporal characteristics of sequences and gain significant insight from the data for cohort selection. We report on this case study, illustrating how the application can be used by a clinician and which questions can be answered. |
Prange, Alexander; Schmidt, Danilo; Sonntag, Daniel A Digital Pen Based Tool for Instant Digitisation and Digitalisation of Biopsy Protocols Inproceedings Bamidis, Panagiotis D; Konstantinidis, Stathis Th.; Rodrigues, Pedro Pereira (Ed.): 30th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2017, Thessaloniki, Greece, June 22-24, 2017, pp. 773–774, IEEE Computer Society, 2017. @inproceedings{DBLP:conf/cbms/PrangeSS17, title = {A Digital Pen Based Tool for Instant Digitisation and Digitalisation of Biopsy Protocols}, author = {Alexander Prange and Danilo Schmidt and Daniel Sonntag}, editor = {Panagiotis D Bamidis and Stathis Th. Konstantinidis and Pedro Pereira Rodrigues}, url = {https://doi.org/10.1109/CBMS.2017.132}, doi = {10.1109/CBMS.2017.132}, year = {2017}, date = {2017-01-01}, booktitle = {30th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2017, Thessaloniki, Greece, June 22-24, 2017}, pages = {773--774}, publisher = {IEEE Computer Society}, abstract = {In order to improve medical processes in nephrology, we present an application that allows doctors to create biopsy protocols by using a digital pen on a tablet. The biopsy protocol app is seamlessly integrated into the existing infrastructure at the hospital (see figure 1). Compared to other reporting tools, we provide (1) real-time hand-writing/gesture recognition and real-time feedback on the recognition results on the screen; (2) a real-time digitisation into structured data and PDF documents; and (3) the mapping of the transcribed contents into concepts of the Banff classification. Our approach combines the benefits of paper with the automatic digitisation and digitalisation of hand-written user input. A fully digital and mobile approach should empower nephrologists to produce high quality data more effectively and in real-time so that it can be directly used in hospital processes.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } In order to improve medical processes in nephrology, we present an application that allows doctors to create biopsy protocols by using a digital pen on a tablet. The biopsy protocol app is seamlessly integrated into the existing infrastructure at the hospital (see figure 1). Compared to other reporting tools, we provide (1) real-time hand-writing/gesture recognition and real-time feedback on the recognition results on the screen; (2) a real-time digitisation into structured data and PDF documents; and (3) the mapping of the transcribed contents into concepts of the Banff classification. Our approach combines the benefits of paper with the automatic digitisation and digitalisation of hand-written user input. A fully digital and mobile approach should empower nephrologists to produce high quality data more effectively and in real-time so that it can be directly used in hospital processes. |
Barz, Michael; Poller, Peter; Schneider, Martin; Zillner, Sonja; Sonntag, Daniel Human-in-the-Loop Control Processes in Gas Turbine Maintenance Inproceedings Marík, Vladimír; Wahlster, Wolfgang; Strasser, Thomas I; Kadera, Petr (Ed.): Industrial Applications of Holonic and Multi-Agent Systems - 8th International Conference, HoloMAS 2017, Lyon, France, August 28-30, 2017, Proceedings, pp. 255–268, Springer, 2017. @inproceedings{DBLP:conf/holomas/BarzPSZS17, title = {Human-in-the-Loop Control Processes in Gas Turbine Maintenance}, author = {Michael Barz and Peter Poller and Martin Schneider and Sonja Zillner and Daniel Sonntag}, editor = {Vladimír Marík and Wolfgang Wahlster and Thomas I Strasser and Petr Kadera}, url = {https://doi.org/10.1007/978-3-319-64635-0_19}, doi = {10.1007/978-3-319-64635-0_19}, year = {2017}, date = {2017-01-01}, booktitle = {Industrial Applications of Holonic and Multi-Agent Systems - 8th International Conference, HoloMAS 2017, Lyon, France, August 28-30, 2017, Proceedings}, volume = {10444}, pages = {255--268}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, abstract = {In this applied research paper, we describe an architecture for seamlessly integrating factory workers in industrial cyber-physical production environments. Our human-in-the-loop control process uses novel input techniques and relies on state-of-the-art industry standards. Our architecture allows for real-time processing of semantically annotated data from multiple sources (e.g., machine sensors, user input devices) and real-time analysis of data for anomaly detection and recovery. We use a semantic knowledge base for storing and querying data (http://www.metaphacts.com) and the Business Process Model and Notation (BPMN) for modelling and controlling the process. We exemplify our industrial solution in the use case of the maintenance of a Siemens gas turbine. We report on this case study and show the advantages of our approach for smart factories. An informal evaluation in the gas turbine maintenance use case shows the utility of automated anomaly detection and handling: workers can fill in paper-based incident reports by using a digital pen; the digitised version is stored in metaphacts and linked to semantic knowledge sources such as process models, structure models, business process models, and user models. Subsequently, automatic maintenance and recovery processes that involve human experts are triggered.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } In this applied research paper, we describe an architecture for seamlessly integrating factory workers in industrial cyber-physical production environments. Our human-in-the-loop control process uses novel input techniques and relies on state-of-the-art industry standards. Our architecture allows for real-time processing of semantically annotated data from multiple sources (e.g., machine sensors, user input devices) and real-time analysis of data for anomaly detection and recovery. We use a semantic knowledge base for storing and querying data (http://www.metaphacts.com) and the Business Process Model and Notation (BPMN) for modelling and controlling the process. We exemplify our industrial solution in the use case of the maintenance of a Siemens gas turbine. We report on this case study and show the advantages of our approach for smart factories. An informal evaluation in the gas turbine maintenance use case shows the utility of automated anomaly detection and handling: workers can fill in paper-based incident reports by using a digital pen; the digitised version is stored in metaphacts and linked to semantic knowledge sources such as process models, structure models, business process models, and user models. Subsequently, automatic maintenance and recovery processes that involve human experts are triggered. |
Prange, Alexander; Barz, Michael; Sonntag, Daniel Speech-based Medical Decision Support in VR using a Deep Neural Network (Demonstration) Inproceedings Sierra, Carles (Ed.): Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 5241–5242, ijcai.org, 2017. @inproceedings{DBLP:conf/ijcai/PrangeBS17, title = {Speech-based Medical Decision Support in VR using a Deep Neural Network (Demonstration)}, author = {Alexander Prange and Michael Barz and Daniel Sonntag}, editor = {Carles Sierra}, url = {https://doi.org/10.24963/ijcai.2017/777}, doi = {10.24963/ijcai.2017/777}, year = {2017}, date = {2017-01-01}, booktitle = {Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017}, pages = {5241--5242}, publisher = {ijcai.org}, abstract = {We present a speech dialogue system that facilitates medical decision support for doctors in a virtual reality (VR) application. The therapy prediction is based on a recurrent neural network model that incorporates the examination history of patients. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of patient records, radiology image data, and the therapy prediction results in VR.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We present a speech dialogue system that facilitates medical decision support for doctors in a virtual reality (VR) application. The therapy prediction is based on a recurrent neural network model that incorporates the examination history of patients. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of patient records, radiology image data, and the therapy prediction results in VR. |
Prange, Alexander; Chikobava, Margarita; Poller, Peter; Barz, Michael; Sonntag, Daniel A Multimodal Dialogue System for Medical Decision Support inside Virtual Reality Inproceedings Jokinen, Kristiina; Stede, Manfred; DeVault, David; Louis, Annie (Ed.): Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbrücken, Germany, August 15-17, 2017, pp. 23–26, Association for Computational Linguistics, 2017. @inproceedings{DBLP:conf/sigdial/PrangeCPBS17, title = {A Multimodal Dialogue System for Medical Decision Support inside Virtual Reality}, author = {Alexander Prange and Margarita Chikobava and Peter Poller and Michael Barz and Daniel Sonntag}, editor = {Kristiina Jokinen and Manfred Stede and David DeVault and Annie Louis}, url = {https://aclanthology.info/papers/W17-5504/w17-5504}, year = {2017}, date = {2017-01-01}, booktitle = {Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbrücken, Germany, August 15-17, 2017}, pages = {23--26}, publisher = {Association for Computational Linguistics}, abstract = {We present a multimodal dialogue system that allows doctors to interact with a medi- cal decision support system in virtual real- ity (VR). We integrate an interactive visu- alization of patient records and radiology image data, as well as therapy predictions. Therapy predictions are computed in real- time using a deep learning model.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We present a multimodal dialogue system that allows doctors to interact with a medi- cal decision support system in virtual real- ity (VR). We integrate an interactive visu- alization of patient records and radiology image data, as well as therapy predictions. Therapy predictions are computed in real- time using a deep learning model. |
2016 |
Journal Articles |
Sonntag, Daniel; Tresp, Volker; Zillner, Sonja; Cavallaro, Alexander; Hammon, Matthias; Reis, André; Fasching, Peter A; Sedlmayr, Martin; Ganslandt, Thomas; Prokosch, Hans-Ulrich; Budde, Klemens; Schmidt, Danilo; Hinrichs, Carl; Wittenberg, Thomas; Daumke, Philipp; Oppelt, Patricia G The Clinical Data Intelligence Project Journal Article Informatik-Spektrum, 39 (4), pp. 290–300, 2016, ISSN: 1432-122X. @article{Sonntag2016, title = {The Clinical Data Intelligence Project}, author = {Daniel Sonntag and Volker Tresp and Sonja Zillner and Alexander Cavallaro and Matthias Hammon and André Reis and Peter A Fasching and Martin Sedlmayr and Thomas Ganslandt and Hans-Ulrich Prokosch and Klemens Budde and Danilo Schmidt and Carl Hinrichs and Thomas Wittenberg and Philipp Daumke and Patricia G Oppelt}, url = {https://doi.org/10.1007/s00287-015-0913-x}, doi = {10.1007/s00287-015-0913-x}, issn = {1432-122X}, year = {2016}, date = {2016-01-01}, journal = {Informatik-Spektrum}, volume = {39}, number = {4}, pages = {290--300}, abstract = {This article is about a new project that combines clinical data intelligence and smart data. It provides an introduction to the ``Klinische Datenintelligenz'' (KDI) project which is founded by the Federal Ministry for Economic Affairs and Energy (BMWi); we transfer research and development results (R&D) of the analysis of data which are generated in the clinical routine in specific medical domain. We present the project structure and goals, how patient care should be improved, and the joint efforts of data and knowledge engineering, information extraction (from textual and other unstructured data), statistical machine learning, decision support, and their integration into special use cases moving towards individualised medicine. In particular, we describe some details of our medical use cases and cooperation with two major German university hospitals.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This article is about a new project that combines clinical data intelligence and smart data. It provides an introduction to the ``Klinische Datenintelligenz'' (KDI) project which is founded by the Federal Ministry for Economic Affairs and Energy (BMWi); we transfer research and development results (R&D) of the analysis of data which are generated in the clinical routine in specific medical domain. We present the project structure and goals, how patient care should be improved, and the joint efforts of data and knowledge engineering, information extraction (from textual and other unstructured data), statistical machine learning, decision support, and their integration into special use cases moving towards individualised medicine. In particular, we describe some details of our medical use cases and cooperation with two major German university hospitals. |
Inproceedings |
Prange, Alexander; Sonntag, Daniel Digital PI-RADS: Smartphone Sketches for Instant Knowledge Acquisition in Prostate Cancer Detection Inproceedings 29th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2016, Belfast, UK and Dublin, Ireland, June 20-24, 2016, pp. 13–18, IEEE Computer Society, 2016. @inproceedings{DBLP:conf/cbms/PrangeS16, title = {Digital PI-RADS: Smartphone Sketches for Instant Knowledge Acquisition in Prostate Cancer Detection}, author = {Alexander Prange and Daniel Sonntag}, url = {https://doi.org/10.1109/CBMS.2016.23}, doi = {10.1109/CBMS.2016.23}, year = {2016}, date = {2016-01-01}, booktitle = {29th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2016, Belfast, UK and Dublin, Ireland, June 20-24, 2016}, pages = {13--18}, publisher = {IEEE Computer Society}, abstract = {In order to improve reporting practices for the detection of prostate cancer, we present an application that allows urologists to create structured reports by using a digital pen on a smartphone. In this domain, printed documents cannot be easily replaced by computer systems because they contain free-form sketches and textual annotations, and the acceptance of traditional PC reporting tools is rather low among the doctors. Our approach provides an instant knowledge acquisition system by automatically interpreting the written strokes, texts, and sketches. We have incorporated this structured reporting system for MRI of the prostate (PI-RADS). Our system imposes only minimal overhead on traditional form-filling processes and provides for a direct, ontology-based structuring of the user input for semantic search and retrieval applications.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } In order to improve reporting practices for the detection of prostate cancer, we present an application that allows urologists to create structured reports by using a digital pen on a smartphone. In this domain, printed documents cannot be easily replaced by computer systems because they contain free-form sketches and textual annotations, and the acceptance of traditional PC reporting tools is rather low among the doctors. Our approach provides an instant knowledge acquisition system by automatically interpreting the written strokes, texts, and sketches. We have incorporated this structured reporting system for MRI of the prostate (PI-RADS). Our system imposes only minimal overhead on traditional form-filling processes and provides for a direct, ontology-based structuring of the user input for semantic search and retrieval applications. |
Schmidt, Danilo; Profitlich, Hans-Jürgen; Sonntag, Daniel Towards Integrated Information Extraction and Facetted Search Applications in Nephrology Inproceedings Dragoni, Mauro; Recupero, Diego Reforgiato; Denecke, Kerstin; Deng, Yihan; Declerck, Thierry (Ed.): Joint Proceedings of the 2th Workshop on Emotions, Modality, Sentiment Analysis and the Semantic Web and the 1st International Workshop on Extraction and Processing of Rich Semantics from Medical Texts co-located with ESWC 2016, Heraklion, Greece, May 29, 2016., CEUR-WS.org, 2016. @inproceedings{DBLP:conf/esws/SchmidtPS16, title = {Towards Integrated Information Extraction and Facetted Search Applications in Nephrology}, author = {Danilo Schmidt and Hans-Jürgen Profitlich and Daniel Sonntag}, editor = {Mauro Dragoni and Diego Reforgiato Recupero and Kerstin Denecke and Yihan Deng and Thierry Declerck}, url = {http://ceur-ws.org/Vol-1613/paper_7.pdf}, year = {2016}, date = {2016-01-01}, booktitle = {Joint Proceedings of the 2th Workshop on Emotions, Modality, Sentiment Analysis and the Semantic Web and the 1st International Workshop on Extraction and Processing of Rich Semantics from Medical Texts co-located with ESWC 2016, Heraklion, Greece, May 29, 2016.}, volume = {1613}, publisher = {CEUR-WS.org}, series = {CEUR Workshop Proceedings}, abstract = {This work focusses on our first integration steps of complex and partly unstructured medical data into a clinical research database. Our main application is an integrated facetted search tool in nephrology based on automatic information extraction results from textual docu- ments. We describe the details of our technical architecture which is based on open-source tools—to be replicated at other universities, re- search institutes, or hospitals.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } This work focusses on our first integration steps of complex and partly unstructured medical data into a clinical research database. Our main application is an integrated facetted search tool in nephrology based on automatic information extraction results from textual docu- ments. We describe the details of our technical architecture which is based on open-source tools—to be replicated at other universities, re- search institutes, or hospitals. |
Barz, Michael; Moniri, Mohammad Mehdi; Weber, Markus; Sonntag, Daniel Multimodal multisensor activity annotation tool Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 17–20, ACM, 2016. @inproceedings{DBLP:conf/huc/BarzMWS16, title = {Multimodal multisensor activity annotation tool}, author = {Michael Barz and Mohammad Mehdi Moniri and Markus Weber and Daniel Sonntag}, editor = {Paul Lukowicz and Antonio Krüger and Andreas Bulling and Youn-Kyung Lim and Shwetak N Patel}, url = {http://doi.acm.org/10.1145/2968219.2971459}, doi = {10.1145/2968219.2971459}, year = {2016}, date = {2016-01-01}, booktitle = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016}, pages = {17--20}, publisher = {ACM}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
Barz, Michael; Sonntag, Daniel Gaze-guided object classification using deep neural networks for attention-based computing Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 253–256, ACM, 2016. @inproceedings{DBLP:conf/huc/BarzS16, title = {Gaze-guided object classification using deep neural networks for attention-based computing}, author = {Michael Barz and Daniel Sonntag}, editor = {Paul Lukowicz and Antonio Krüger and Andreas Bulling and Youn-Kyung Lim and Shwetak N Patel}, url = {http://doi.acm.org/10.1145/2968219.2971389}, doi = {10.1145/2968219.2971389}, year = {2016}, date = {2016-01-01}, booktitle = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016}, pages = {253--256}, publisher = {ACM}, abstract = {Recent advances in eye tracking technologies opened the way to design novel attention-based user interfaces. This is promising for pro-active and assistive technologies for cyber-physical systems in the domains of, e.g., healthcare and industry 4.0. Prior approaches to recognize a user's attention are usually limited to the raw gaze signal or sensors in instrumented environments. We propose a system that (1) incorporates the gaze signal and the egocentric camera of the eye tracker to identify the objects the user focuses at; (2) employs object classification based on deep learning which we recompiled for our purposes on a GPU-based image classification server; (3) detects whether the user actually draws attention to that object; and (4) combines these modules for constructing episodic memories of egocentric events in real-time.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Recent advances in eye tracking technologies opened the way to design novel attention-based user interfaces. This is promising for pro-active and assistive technologies for cyber-physical systems in the domains of, e.g., healthcare and industry 4.0. Prior approaches to recognize a user's attention are usually limited to the raw gaze signal or sensors in instrumented environments. We propose a system that (1) incorporates the gaze signal and the egocentric camera of the eye tracker to identify the objects the user focuses at; (2) employs object classification based on deep learning which we recompiled for our purposes on a GPU-based image classification server; (3) detects whether the user actually draws attention to that object; and (4) combines these modules for constructing episodic memories of egocentric events in real-time. |
Luxenburger, Andreas; Prange, Alexander; Moniri, Mohammad Mehdi; Sonntag, Daniel MedicaLVR: towards medical remote collaboration using virtual reality Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 321–324, ACM, 2016. @inproceedings{DBLP:conf/huc/LuxenburgerPMS16, title = {MedicaLVR: towards medical remote collaboration using virtual reality}, author = {Andreas Luxenburger and Alexander Prange and Mohammad Mehdi Moniri and Daniel Sonntag}, editor = {Paul Lukowicz and Antonio Krüger and Andreas Bulling and Youn-Kyung Lim and Shwetak N Patel}, url = {http://doi.acm.org/10.1145/2968219.2971392}, doi = {10.1145/2968219.2971392}, year = {2016}, date = {2016-01-01}, booktitle = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016}, pages = {321--324}, publisher = {ACM}, abstract = {We present a virtual reality framework and assistive tool for design practices for new medical environments, grounded on human visual perception, attention and action. This includes an interactive visualization of shared electronic patient records, previously acquired with a remote tablet device, in a virtual environment incorporating hand tracking, eye tracking and a vision-based peripheral view monitoring. The goal is to influence medical environments' affordances, especially for e-health and m-health applications as well as user experience and design conception for tele-medicine.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We present a virtual reality framework and assistive tool for design practices for new medical environments, grounded on human visual perception, attention and action. This includes an interactive visualization of shared electronic patient records, previously acquired with a remote tablet device, in a virtual environment incorporating hand tracking, eye tracking and a vision-based peripheral view monitoring. The goal is to influence medical environments' affordances, especially for e-health and m-health applications as well as user experience and design conception for tele-medicine. |
Moniri, Mohammad Mehdi; Sonntag, Daniel; Luxenburger, Andreas Peripheral view calculation in virtual reality applications Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 333–336, ACM, 2016. @inproceedings{DBLP:conf/huc/MoniriSL16, title = {Peripheral view calculation in virtual reality applications}, author = {Mohammad Mehdi Moniri and Daniel Sonntag and Andreas Luxenburger}, editor = {Paul Lukowicz and Antonio Krüger and Andreas Bulling and Youn-Kyung Lim and Shwetak N Patel}, url = {http://doi.acm.org/10.1145/2968219.2971391}, doi = {10.1145/2968219.2971391}, year = {2016}, date = {2016-01-01}, booktitle = {Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016}, pages = {333--336}, publisher = {ACM}, abstract = {We present an application based on a general peripheral view calculation model which extends previous work on attention-based user interfaces that use eye gaze. An intuitive, two dimensional visibility measure based on the concept of solid angle is developed. We determine to which extent an object of interest, observed by a user, intersects with each region of the underlying visual field model. The results are weighted (thereby considering the visual acuity in each visual field) to determine the total visibility of the object. As a proof of concept, we exemplify the proposed model in a virtual reality application which incorporates a head-mounted display with integrated eye tracking functionality. In this context, we implement several proactive system behaviors including contextual information presentation with an adaptive level of detail and attention guidance; the latter is implemented by detecting visual acuity limitations or attention drifts.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We present an application based on a general peripheral view calculation model which extends previous work on attention-based user interfaces that use eye gaze. An intuitive, two dimensional visibility measure based on the concept of solid angle is developed. We determine to which extent an object of interest, observed by a user, intersects with each region of the underlying visual field model. The results are weighted (thereby considering the visual acuity in each visual field) to determine the total visibility of the object. As a proof of concept, we exemplify the proposed model in a virtual reality application which incorporates a head-mounted display with integrated eye tracking functionality. In this context, we implement several proactive system behaviors including contextual information presentation with an adaptive level of detail and attention guidance; the latter is implemented by detecting visual acuity limitations or attention drifts. |
Moniri, Mohammad Mehdi; Valcarcel, Fabio Andres Espinosa; Merkel, Dieter; Sonntag, Daniel Human Gaze and Focus-of-Attention in Dual Reality Human-Robot Collaboration Inproceedings 12th International Conference on Intelligent Environments, IE 2016, London, United Kingdom, September 14-16, 2016, pp. 238–241, IEEE, 2016. @inproceedings{DBLP:conf/intenv/MoniriVMS16, title = {Human Gaze and Focus-of-Attention in Dual Reality Human-Robot Collaboration}, author = {Mohammad Mehdi Moniri and Fabio Andres Espinosa Valcarcel and Dieter Merkel and Daniel Sonntag}, url = {https://doi.org/10.1109/IE.2016.54}, doi = {10.1109/IE.2016.54}, year = {2016}, date = {2016-01-01}, booktitle = {12th International Conference on Intelligent Environments, IE 2016, London, United Kingdom, September 14-16, 2016}, pages = {238--241}, publisher = {IEEE}, abstract = {Human gaze is an important indicator of the direction of visual focus-of-attention. This information can be very useful in human-robot interaction scenarios. This paper describes a research prototype that utilizes the user's visual attention in a collaborative dual reality environment. We add an additional dimension to existing human-robot interaction scenarios and describe a human-robot collaboration scenario in which the involved human participants are in two different physical locations. One user has the same physical actions space as the robot, the second user is monitoring the setup and thereby collaborating through a virtual reality system. The proposed research prototype monitors the user's visual attention in both real and virtual environments. The prototype also provides information in both the virtual and real environment which results in a dual reality collaboration scenario. As a result, new human-robot interactions brought about by Industrie 4.0, with novel forms of collaborative factory work, can be constructed.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Human gaze is an important indicator of the direction of visual focus-of-attention. This information can be very useful in human-robot interaction scenarios. This paper describes a research prototype that utilizes the user's visual attention in a collaborative dual reality environment. We add an additional dimension to existing human-robot interaction scenarios and describe a human-robot collaboration scenario in which the involved human participants are in two different physical locations. One user has the same physical actions space as the robot, the second user is monitoring the setup and thereby collaborating through a virtual reality system. The proposed research prototype monitors the user's visual attention in both real and virtual environments. The prototype also provides information in both the virtual and real environment which results in a dual reality collaboration scenario. As a result, new human-robot interactions brought about by Industrie 4.0, with novel forms of collaborative factory work, can be constructed. |
Bellon, Richárd; Choi, Younggeon; Ekker, Nikoletta; Lepetit, Vincent; Olasz, Mike L; Sonntag, Daniel; Tősér, Zoltán; Yoo, Kyounghwan; Lőrincz, András Model Based Augmentation and Testing of an Annotated Hand Pose Dataset Inproceedings Friedrich, Gerhard; Helmert, Malte; Wotawa, Franz (Ed.): KI 2016: Advances in Artificial Intelligence - 39th Annual German Conference on AI, Klagenfurt, Austria, September 26-30, 2016, Proceedings, pp. 17–29, Springer, 2016. @inproceedings{DBLP:conf/ki/BellonCELOSTYL16, title = {Model Based Augmentation and Testing of an Annotated Hand Pose Dataset}, author = {Richárd Bellon and Younggeon Choi and Nikoletta Ekker and Vincent Lepetit and Mike L Olasz and Daniel Sonntag and Zoltán Tősér and Kyounghwan Yoo and András Lőrincz}, editor = {Gerhard Friedrich and Malte Helmert and Franz Wotawa}, url = {https://doi.org/10.1007/978-3-319-46073-4_2}, doi = {10.1007/978-3-319-46073-4_2}, year = {2016}, date = {2016-01-01}, booktitle = {KI 2016: Advances in Artificial Intelligence - 39th Annual German Conference on AI, Klagenfurt, Austria, September 26-30, 2016, Proceedings}, volume = {9904}, pages = {17--29}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, abstract = {Recent advances of deep learning technology enable one to train complex input-output mappings, provided that a high quality training set is available. In this paper, we show how to extend an existing dataset of depth maps of hand annotated with the corresponding 3D hand poses by fitting a 3D hand model to smart glove-based annotations and generating new hand views. We make available our code and the generated data. Based on the present procedure and our previous results, we suggest a pipeline for creating high quality data.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Recent advances of deep learning technology enable one to train complex input-output mappings, provided that a high quality training set is available. In this paper, we show how to extend an existing dataset of depth maps of hand annotated with the corresponding 3D hand poses by fitting a 3D hand model to smart glove-based annotations and generating new hand views. We make available our code and the generated data. Based on the present procedure and our previous results, we suggest a pipeline for creating high quality data. |
Barz, Michael; Daiber, Florian; Bulling, Andreas Prediction of Gaze Estimation Error for Error-Aware Gaze-Based Interfaces Inproceedings Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 275–278, ACM Press, New York, NY, USA, 2016, ISBN: 9781450341257. @inproceedings{Barz2016, title = {Prediction of Gaze Estimation Error for Error-Aware Gaze-Based Interfaces}, author = {Michael Barz and Florian Daiber and Andreas Bulling}, url = {http://dl.acm.org/citation.cfm?doid=2857491.2857493}, doi = {10.1145/2857491.2857493}, isbn = {9781450341257}, year = {2016}, date = {2016-01-01}, booktitle = {Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications}, pages = {275--278}, publisher = {ACM Press}, address = {New York, NY, USA}, series = {ETRA '16}, abstract = {Gaze estimation error is inherent in head-mounted eye trackers and seriously impacts performance, usability, and user experience of gaze-based interfaces. Particularly in mobile settings, this error varies constantly as users move in front and look at different parts of a display. We envision a new class of gaze-based interfaces that are aware of the gaze estimation error and adapt to it in real time. As a first step towards this vision we introduce an error model that is able to predict the gaze estimation error. Our method covers major building blocks of mobile gaze estimation, specifically mapping of pupil positions to scene camera coordinates, marker-based display detection, and mapping of gaze from scene camera to on-screen coordinates. We develop our model through a series of principled measurements of a state-of-the-art head-mounted eye tracker.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Gaze estimation error is inherent in head-mounted eye trackers and seriously impacts performance, usability, and user experience of gaze-based interfaces. Particularly in mobile settings, this error varies constantly as users move in front and look at different parts of a display. We envision a new class of gaze-based interfaces that are aware of the gaze estimation error and adapt to it in real time. As a first step towards this vision we introduce an error model that is able to predict the gaze estimation error. Our method covers major building blocks of mobile gaze estimation, specifically mapping of pupil positions to scene camera coordinates, marker-based display detection, and mapping of gaze from scene camera to on-screen coordinates. We develop our model through a series of principled measurements of a state-of-the-art head-mounted eye tracker. |
2015 |
Journal Articles |
Orlosky, Jason; Toyama, Takumi; Kiyokawa, Kiyoshi; Sonntag, Daniel ModulAR: Eye-Controlled Vision Augmentations for Head Mounted Displays Journal Article IEEE Trans. Vis. Comput. Graph., 21 (11), pp. 1259–1268, 2015. @article{DBLP:journals/tvcg/OrloskyTKS15, title = {ModulAR: Eye-Controlled Vision Augmentations for Head Mounted Displays}, author = {Jason Orlosky and Takumi Toyama and Kiyoshi Kiyokawa and Daniel Sonntag}, url = {https://doi.org/10.1109/TVCG.2015.2459852}, doi = {10.1109/TVCG.2015.2459852}, year = {2015}, date = {2015-01-01}, journal = {IEEE Trans. Vis. Comput. Graph.}, volume = {21}, number = {11}, pages = {1259--1268}, abstract = {In the last few years, the advancement of head mounted display technology and optics has opened up many new possibilities for the field of Augmented Reality. However, many commercial and prototype systems often have a single display modality, fixed field of view, or inflexible form factor. In this paper, we introduce Modular Augmented Reality (ModulAR), a hardware and software framework designed to improve flexibility and hands-free control of video see-through augmented reality displays and augmentative functionality. To accomplish this goal, we introduce the use of integrated eye tracking for on-demand control of vision augmentations such as optical zoom or field of view expansion. Physical modification of the device's configuration can be accomplished on the fly using interchangeable camera-lens modules that provide different types of vision enhancements. We implement and test functionality for several primary configurations using telescopic and fisheye camera-lens systems, though many other customizations are possible. We also implement a number of eye-based interactions in order to engage and control the vision augmentations in real time, and explore different methods for merging streams of augmented vision into the user's normal field of view. In a series of experiments, we conduct an in depth analysis of visual acuity and head and eye movement during search and recognition tasks. Results show that methods with larger field of view that utilize binary on/off and gradual zoom mechanisms outperform snapshot and sub-windowed methods and that type of eye engagement has little effect on performance.}, keywords = {}, pubstate = {published}, tppubtype = {article} } In the last few years, the advancement of head mounted display technology and optics has opened up many new possibilities for the field of Augmented Reality. However, many commercial and prototype systems often have a single display modality, fixed field of view, or inflexible form factor. In this paper, we introduce Modular Augmented Reality (ModulAR), a hardware and software framework designed to improve flexibility and hands-free control of video see-through augmented reality displays and augmentative functionality. To accomplish this goal, we introduce the use of integrated eye tracking for on-demand control of vision augmentations such as optical zoom or field of view expansion. Physical modification of the device's configuration can be accomplished on the fly using interchangeable camera-lens modules that provide different types of vision enhancements. We implement and test functionality for several primary configurations using telescopic and fisheye camera-lens systems, though many other customizations are possible. We also implement a number of eye-based interactions in order to engage and control the vision augmentations in real time, and explore different methods for merging streams of augmented vision into the user's normal field of view. In a series of experiments, we conduct an in depth analysis of visual acuity and head and eye movement during search and recognition tasks. Results show that methods with larger field of view that utilize binary on/off and gradual zoom mechanisms outperform snapshot and sub-windowed methods and that type of eye engagement has little effect on performance. |
Inproceedings |
Toyama, Takumi; Sonntag, Daniel Towards Episodic Memory Support for Dementia Patients by Recognizing Objects, Faces and Text in Eye Gaze Inproceedings Hölldobler, Steffen; Krötzsch, Markus; Peñaloza, Rafael; Rudolph, Sebastian (Ed.): KI 2015: Advances in Artificial Intelligence - 38th Annual German Conference on AI, Dresden, Germany, September 21-25, 2015, Proceedings, pp. 316–323, Springer, 2015. @inproceedings{DBLP:conf/ki/ToyamaS15, title = {Towards Episodic Memory Support for Dementia Patients by Recognizing Objects, Faces and Text in Eye Gaze}, author = {Takumi Toyama and Daniel Sonntag}, editor = {Steffen Hölldobler and Markus Krötzsch and Rafael Peñaloza and Sebastian Rudolph}, url = {https://doi.org/10.1007/978-3-319-24489-1_29}, doi = {10.1007/978-3-319-24489-1_29}, year = {2015}, date = {2015-01-01}, booktitle = {KI 2015: Advances in Artificial Intelligence - 38th Annual German Conference on AI, Dresden, Germany, September 21-25, 2015, Proceedings}, volume = {9324}, pages = {316--323}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, abstract = {Humans forget events or certain types of information sometimes even when the information is important. There are patients who suffer from diseases such as dementia that result in memory loss. Using an eye tracker and image analysis modules, we develop a system that recognizes everyday objects, faces and text that the user looks at. This system can be effectively used to support an individual user’s memory by logging certain types of everyday information that the user perceives and tries to organize in his or her memory. We present a technical implementation of an episodic memory support.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Humans forget events or certain types of information sometimes even when the information is important. There are patients who suffer from diseases such as dementia that result in memory loss. Using an eye tracker and image analysis modules, we develop a system that recognizes everyday objects, faces and text that the user looks at. This system can be effectively used to support an individual user’s memory by logging certain types of everyday information that the user perceives and tries to organize in his or her memory. We present a technical implementation of an episodic memory support. |
Sonntag, Daniel; Orlosky, Jason; Weber, Markus; Gu, Yecheng; Sosnovsky, Sergey; Toyama, Takumi; Toosi, Ehsan Nadjaran Cognitive Monitoring via Eye Tracking in Virtual Reality Pedestrian Environments Inproceedings Gehring, Sven; Krüger, Antonio; Alt, Florian; Taylor, Nick (Ed.): Proceedings of the 4th International Symposium on Pervasive Displays, PerDis 2015, Saarbrücken, Germany, June 10-12, 2015, pp. 269–270, ACM, 2015. @inproceedings{DBLP:conf/perdis/SonntagOWGSTT15, title = {Cognitive Monitoring via Eye Tracking in Virtual Reality Pedestrian Environments}, author = {Daniel Sonntag and Jason Orlosky and Markus Weber and Yecheng Gu and Sergey Sosnovsky and Takumi Toyama and Ehsan Nadjaran Toosi}, editor = {Sven Gehring and Antonio Krüger and Florian Alt and Nick Taylor}, url = {http://doi.acm.org/10.1145/2757710.2776816}, doi = {10.1145/2757710.2776816}, year = {2015}, date = {2015-01-01}, booktitle = {Proceedings of the 4th International Symposium on Pervasive Displays, PerDis 2015, Saarbrücken, Germany, June 10-12, 2015}, pages = {269--270}, publisher = {ACM}, abstract = {Recent advances in virtual and augmented reality have led to the development of a number of simulations for different applications. In particular, virtual reality simulations for monitoring, evaluation, training, and education have started to emerge. In this work, we introduce a virtual reality environment that provides an immersive traffic simulation designed to observe behavior and monitor relevant skills and abilities of pedestrians who may be at risk, such as elderly persons with cognitive impairments. Methods for interaction using hand and arm gestures are complemented by passive, integrated eye tracking to allow for monitoring of cognition, reaction times, and behavior.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Recent advances in virtual and augmented reality have led to the development of a number of simulations for different applications. In particular, virtual reality simulations for monitoring, evaluation, training, and education have started to emerge. In this work, we introduce a virtual reality environment that provides an immersive traffic simulation designed to observe behavior and monitor relevant skills and abilities of pedestrians who may be at risk, such as elderly persons with cognitive impairments. Methods for interaction using hand and arm gestures are complemented by passive, integrated eye tracking to allow for monitoring of cognition, reaction times, and behavior. |
2014 |
Journal Articles |
Sonntag, Daniel; Weber, Markus; Cavallaro, Alexander; Hammon, Matthias Integrating Digital Pens in Breast Imaging for Instant Knowledge Acquisition Journal Article AI Magazine, 35 (1), pp. 26–37, 2014. @article{DBLP:journals/aim/SonntagWCH14, title = {Integrating Digital Pens in Breast Imaging for Instant Knowledge Acquisition}, author = {Daniel Sonntag and Markus Weber and Alexander Cavallaro and Matthias Hammon}, url = {http://www.aaai.org/ojs/index.php/aimagazine/article/view/2501}, year = {2014}, date = {2014-01-01}, journal = {AI Magazine}, volume = {35}, number = {1}, pages = {26--37}, abstract = {Future radiology practices assume that the radiology reports should be uniform, comprehensive, and easily managed. This means that reports must be readable to humans and machines alike. In order to improve reporting practices in breast imaging, we allow the radiologist to write structured reports with a special pen on paper with an invisible dot pattern. In this way, we provide a knowledge acquisition system for printed mammography patient forms for the combined work with printed and digital documents. In this domain, printed documents cannot be easily replaced by computer systems because they contain free-form sketches and textual annotations, and the acceptance of traditional PC reporting tools is rather low among the doctors. This is due to the fact that current electronic reporting systems significantly add to the amount of time it takes to complete the reports. We describe our real-time digital paper application and focus on the use case study of our deployed application. We think that our results motivate the design and implementation of intuitive pen-based user interfaces for the medical reporting process and similar knowledge work domains. Our system imposes only minimal overhead on traditional form-filling processes and provides for a direct, ontology-based structuring of the user input for semantic search and retrieval applications, as well as other applied artificial intelligence scenarios which involve manual form-based data acquisition.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Future radiology practices assume that the radiology reports should be uniform, comprehensive, and easily managed. This means that reports must be readable to humans and machines alike. In order to improve reporting practices in breast imaging, we allow the radiologist to write structured reports with a special pen on paper with an invisible dot pattern. In this way, we provide a knowledge acquisition system for printed mammography patient forms for the combined work with printed and digital documents. In this domain, printed documents cannot be easily replaced by computer systems because they contain free-form sketches and textual annotations, and the acceptance of traditional PC reporting tools is rather low among the doctors. This is due to the fact that current electronic reporting systems significantly add to the amount of time it takes to complete the reports. We describe our real-time digital paper application and focus on the use case study of our deployed application. We think that our results motivate the design and implementation of intuitive pen-based user interfaces for the medical reporting process and similar knowledge work domains. Our system imposes only minimal overhead on traditional form-filling processes and provides for a direct, ontology-based structuring of the user input for semantic search and retrieval applications, as well as other applied artificial intelligence scenarios which involve manual form-based data acquisition. |
Inproceedings |
Palotai, Zsolt; Lang, Miklos; Sarkany, Andras; Toser, Zoltan; Sonntag, Daniel; Toyama, Takumi; Lőrincz, András LabelMovie: Semi-supervised machine annotation tool with quality assurance and crowd-sourcing options for videos Inproceedings 12th International Workshop on Content-Based Multimedia Indexing, CBMI 2014, Klagenfurt, Austria, June 18-20, 2014, pp. 1–4, IEEE, 2014. @inproceedings{DBLP:conf/cbmi/PalotaiLSTSTL14, title = {LabelMovie: Semi-supervised machine annotation tool with quality assurance and crowd-sourcing options for videos}, author = {Zsolt Palotai and Miklos Lang and Andras Sarkany and Zoltan Toser and Daniel Sonntag and Takumi Toyama and András Lőrincz}, url = {https://doi.org/10.1109/CBMI.2014.6849850}, doi = {10.1109/CBMI.2014.6849850}, year = {2014}, date = {2014-01-01}, booktitle = {12th International Workshop on Content-Based Multimedia Indexing, CBMI 2014, Klagenfurt, Austria, June 18-20, 2014}, pages = {1--4}, publisher = {IEEE}, abstract = {For multiple reasons, the automatic annotation of video recordings is challenging. The amount of database video instances to be annotated is huge, tedious manual labeling sessions are required, the multi-modal annotation needs exact information of space, time, and context, and the different labeling opportunities require special agreements between annotators, and alike. Crowd-sourcing with quality assurance by experts may come to the rescue here. We have developed a special tool: individual experts can annotate videos over the Internet, their work can be joined and filtered, the annotated material can be evaluated by machine learning methods, and automated annotation may start according to a predefined confidence level. A relatively small number of manually labeled instances may efficiently bootstrap the machine annotation procedure. We present the new mathematical concepts and algorithms for semi-supervised induction and the corresponding manual annotation tool which features special visualization methods for crowd-sourced users. A special feature is that the annotation tool is usable for users not familiar with machine learning methods; for example, we allow them to ignite and handle a complex bootstrapping process.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } For multiple reasons, the automatic annotation of video recordings is challenging. The amount of database video instances to be annotated is huge, tedious manual labeling sessions are required, the multi-modal annotation needs exact information of space, time, and context, and the different labeling opportunities require special agreements between annotators, and alike. Crowd-sourcing with quality assurance by experts may come to the rescue here. We have developed a special tool: individual experts can annotate videos over the Internet, their work can be joined and filtered, the annotated material can be evaluated by machine learning methods, and automated annotation may start according to a predefined confidence level. A relatively small number of manually labeled instances may efficiently bootstrap the machine annotation procedure. We present the new mathematical concepts and algorithms for semi-supervised induction and the corresponding manual annotation tool which features special visualization methods for crowd-sourced users. A special feature is that the annotation tool is usable for users not familiar with machine learning methods; for example, we allow them to ignite and handle a complex bootstrapping process. |
Toyama, Takumi; Sonntag, Daniel; Dengel, Andreas; Matsuda, Takahiro; Iwamura, Masakazu; Kise, Koichi A mixed reality head-mounted text translation system using eye gaze input Inproceedings Kuflik, Tsvi; Stock, Oliviero; Chai, Joyce Yue; Krüger, Antonio (Ed.): 19th International Conference on Intelligent User Interfaces, IUI 2014, Haifa, Israel, February 24-27, 2014, pp. 329–334, ACM, 2014. @inproceedings{DBLP:conf/iui/ToyamaSDMIK14, title = {A mixed reality head-mounted text translation system using eye gaze input}, author = {Takumi Toyama and Daniel Sonntag and Andreas Dengel and Takahiro Matsuda and Masakazu Iwamura and Koichi Kise}, editor = {Tsvi Kuflik and Oliviero Stock and Joyce Yue Chai and Antonio Krüger}, url = {http://doi.acm.org/10.1145/2557500.2557528}, doi = {10.1145/2557500.2557528}, year = {2014}, date = {2014-01-01}, booktitle = {19th International Conference on Intelligent User Interfaces, IUI 2014, Haifa, Israel, February 24-27, 2014}, pages = {329--334}, publisher = {ACM}, abstract = {Efficient text recognition has recently been a challenge for augmented reality systems. In this paper, we propose a system with the ability to provide translations to the user in real-time. We use eye gaze for more intuitive and efficient input for ubiquitous text reading and translation in head mounted displays (HMDs). The eyes can be used to indicate regions of interest in text documents and activate optical-character-recognition (OCR) and translation functions. Visual feedback and navigation help in the interaction process, and text snippets with translations from Japanese to English text snippets, are presented in a see-through HMD. We focus on travelers who go to Japan and need to read signs and propose two different gaze gestures for activating the OCR text reading and translation function. We evaluate which type of gesture suits our OCR scenario best. We also show that our gaze-based OCR method on the extracted gaze regions provide faster access times to information than traditional OCR approaches. Other benefits include that visual feedback of the extracted text region can be given in real-time, the Japanese to English translation can be presented in real-time, and the augmentation of the synchronized and calibrated HMD in this mixed reality application are presented at exact locations in the augmented user view to allow for dynamic text translation management in head-up display systems.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Efficient text recognition has recently been a challenge for augmented reality systems. In this paper, we propose a system with the ability to provide translations to the user in real-time. We use eye gaze for more intuitive and efficient input for ubiquitous text reading and translation in head mounted displays (HMDs). The eyes can be used to indicate regions of interest in text documents and activate optical-character-recognition (OCR) and translation functions. Visual feedback and navigation help in the interaction process, and text snippets with translations from Japanese to English text snippets, are presented in a see-through HMD. We focus on travelers who go to Japan and need to read signs and propose two different gaze gestures for activating the OCR text reading and translation function. We evaluate which type of gesture suits our OCR scenario best. We also show that our gaze-based OCR method on the extracted gaze regions provide faster access times to information than traditional OCR approaches. Other benefits include that visual feedback of the extracted text region can be given in real-time, the Japanese to English translation can be presented in real-time, and the augmentation of the synchronized and calibrated HMD in this mixed reality application are presented at exact locations in the augmented user view to allow for dynamic text translation management in head-up display systems. |
0000 |
Conferences |
Barz, Michael; Altmeyer, Kristin; Malone, Sarah; Lauer, Luisa; Daniel Sonntag, Digital Pen Features Predict Task Difficulty and User Performance of Cognitive Tests Conference Forthcoming Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ' 20 ACM, Forthcoming, ISBN: 978-1-4503-6861-2/20/07. @conference{Barz2020, title = {Digital Pen Features Predict Task Difficulty and User Performance of Cognitive Tests}, author = {Michael Barz and Kristin Altmeyer and Sarah Malone and Luisa Lauer and Daniel Sonntag}, doi = {10.1145/3340631.3394839}, isbn = {978-1-4503-6861-2/20/07}, booktitle = {Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization}, publisher = {ACM}, series = {UMAP ' 20}, abstract = {Digital pen signals were shown to be predictive for cognitive states, cognitive load and emotion in educational settings. We investigate whether low-level pen-based features can predict the difficulty of tasks in a cognitive test and the learner's performance in these tasks, which is inherently related to cognitive load, without a semantic content analysis. We record data for tasks of varying difficulty in a controlled study with children from elementary school. We include two versions of the Trail Making Test (TMT) and six drawing patterns from the Snijders-Oomen Non-verbal intelligence test (SON) as tasks that feature increasing levels of difficulty. We examine how accurately we can predict the task difficulty and the user performance as a measure for cognitive load using support vector machines and gradient boosted decision trees with different feature selection strategies. The results show that our correlation-based feature selection is beneficial for model training, in particular when samples from TMT and SON are concatenated for joint modelling of difficulty and time. Our findings open up opportunities for technology-enhanced adaptive learning.}, keywords = {}, pubstate = {forthcoming}, tppubtype = {conference} } Digital pen signals were shown to be predictive for cognitive states, cognitive load and emotion in educational settings. We investigate whether low-level pen-based features can predict the difficulty of tasks in a cognitive test and the learner's performance in these tasks, which is inherently related to cognitive load, without a semantic content analysis. We record data for tasks of varying difficulty in a controlled study with children from elementary school. We include two versions of the Trail Making Test (TMT) and six drawing patterns from the Snijders-Oomen Non-verbal intelligence test (SON) as tasks that feature increasing levels of difficulty. We examine how accurately we can predict the task difficulty and the user performance as a measure for cognitive load using support vector machines and gradient boosted decision trees with different feature selection strategies. The results show that our correlation-based feature selection is beneficial for model training, in particular when samples from TMT and SON are concatenated for joint modelling of difficulty and time. Our findings open up opportunities for technology-enhanced adaptive learning. |
2020 |
Journal Articles |
Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking Journal Article KI - Künstliche Intelligenz German Journal of Artificial Intelligence, 2020, ISSN: 0933-1875. |
Inproceedings |
Digital Pen Features Predict Task Difficulty and User Performance of Cognitive Tests Inproceedings Forthcoming Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, pp. 10, ACM, Forthcoming. |
Visual Search Target Inference in Natural Interaction Settings with Machine Learning Inproceedings Symposium on Eye Tracking Research and Applications, Association for Computing Machinery, Stuttgart, Germany, 2020, ISBN: 9781450371339. |
2019 |
Inproceedings |
Automatic Judgement of Neural Network-Generated Image Captions Inproceedings Martin-Vide, Carlos; Purver, Matthew; Pollak, Senja (Ed.): Statistical Language and Speech Processing - 7th International Conference, {SLSP} 2019, Ljubljana, Slovenia, October 14-16, 2019, Proceedings, pp. 261 – 272 , Springer, 2019, ISBN: 978-3-030-31371-5. |
Multimodal Speech-based Dialogue for the Mini-Mental State Examination Inproceedings Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. CS13:1–CS13:8, ACM, Glasgow, Scotland Uk, 2019, ISBN: 978-1-4503-5971-9. |
Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings Inproceedings Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP), pp. 1–10, ACL, Florence, Italy, 2019. |
2018 |
Journal Articles |
An architecture of open-source tools to combine textual information extraction, faceted search and information visualisation Journal Article CoRR, abs/1810.12627 , 2018. |
Interactive Cognitive Assessment Tools: A Case Study on Digital Pens for the Clinical Assessment of Dementia Journal Article CoRR, abs/1810.04943 , 2018. |
A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces Journal Article CoRR, abs/1803.04818 , 2018. |
A categorisation and implementation of digital pen features for behaviour characterisation Journal Article CoRR, abs/1810.03970 , 2018. |
Booklets |
Towards Hybrid Human-Machine Translation Services Booklet EasyChair Preprint no. 333, 2018. |
Inproceedings |
Visual Search Target Inference using Bag of Deep Visual Words Inproceedings Trollmann, Frank; Turhan, Anni-Yasmin (Ed.): KI 2018: Advances in Artificial Intelligence, pp. 297––304, Springer International Publishing, 2018, ISBN: 978-3-030-00111-7. |
Error-aware gaze-based interfaces for robust mobile gaze interaction Inproceedings Sharif, Bonita; Krejtz, Krzysztof (Ed.): Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA 2018, Warsaw, Poland, June 14-17, 2018, pp. 24:1–24:10, ACM, 2018. |
Medical 3D Images in Multimodal Virtual Reality Inproceedings Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion, Tokyo, Japan, March 07-11, 2018, pp. 19:1–19:2, ACM, 2018. |
Device-Type Influence in Crowd-based Natural Language Translation Tasks Inproceedings Checco, Alessandro; Demartini, Gianluca; Gadiraju, Ujwal; Sarasua, Cristina; Aroyo, Lora; Dumitrache, Anca; Paritosh, Praveen; Quinn, Alex; Welty, Chris (Ed.): 1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing 2018, and CrowdBias'18: Disentangling the Relation Between Crowdsourcing and Bias Management, pp. 93–97, Aachen, 2018, ISSN: 1613-0073. |
Towards a Multimodal Multisensory Cognitive Assessment Framework Inproceedings 31st IEEE International Symposium on Computer-Based Medical Systems, CBMS 2018, Karlstad, Sweden, June 18-21, 2018, pp. 24–29, 2018. |
2017 |
Journal Articles |
A novel tool for the identification of correlations in medical data by faceted search Journal Article Comp. in Bio. and Med., 85 , pp. 98–105, 2017. |
Design and Implementation of a Semantic Dialogue System for Radiologists Journal Article CoRR, abs/1701.07381 , 2017. |
Fine-tuning deep CNN models on specific MS COCO categories Journal Article CoRR, abs/1709.01476 , 2017. |
Interakt - A Multimodal Multisensory Interactive Cognitive Assessment Tool Journal Article CoRR, abs/1709.01796 , 2017. |
Inproceedings |
Evaluating Remote and Head-worn Eye Trackers in Multi-modal Speech-based HRI Inproceedings Mutlu, Bilge; Tscheligi, Manfred; Weiss, Astrid; Young, James E (Ed.): Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, Vienna, Austria, March 6-9, 2017, pp. 79–80, ACM, 2017. |
Integrated Decision Support by Combining Textual Information Extraction, Facetted Search and Information Visualisation Inproceedings Bamidis, Panagiotis D; Konstantinidis, Stathis Th.; Rodrigues, Pedro Pereira (Ed.): 30th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2017, Thessaloniki, Greece, June 22-24, 2017, pp. 95–100, IEEE Computer Society, 2017. |
A Digital Pen Based Tool for Instant Digitisation and Digitalisation of Biopsy Protocols Inproceedings Bamidis, Panagiotis D; Konstantinidis, Stathis Th.; Rodrigues, Pedro Pereira (Ed.): 30th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2017, Thessaloniki, Greece, June 22-24, 2017, pp. 773–774, IEEE Computer Society, 2017. |
Human-in-the-Loop Control Processes in Gas Turbine Maintenance Inproceedings Marík, Vladimír; Wahlster, Wolfgang; Strasser, Thomas I; Kadera, Petr (Ed.): Industrial Applications of Holonic and Multi-Agent Systems - 8th International Conference, HoloMAS 2017, Lyon, France, August 28-30, 2017, Proceedings, pp. 255–268, Springer, 2017. |
Speech-based Medical Decision Support in VR using a Deep Neural Network (Demonstration) Inproceedings Sierra, Carles (Ed.): Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 5241–5242, ijcai.org, 2017. |
A Multimodal Dialogue System for Medical Decision Support inside Virtual Reality Inproceedings Jokinen, Kristiina; Stede, Manfred; DeVault, David; Louis, Annie (Ed.): Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbrücken, Germany, August 15-17, 2017, pp. 23–26, Association for Computational Linguistics, 2017. |
2016 |
Journal Articles |
The Clinical Data Intelligence Project Journal Article Informatik-Spektrum, 39 (4), pp. 290–300, 2016, ISSN: 1432-122X. |
Inproceedings |
Digital PI-RADS: Smartphone Sketches for Instant Knowledge Acquisition in Prostate Cancer Detection Inproceedings 29th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2016, Belfast, UK and Dublin, Ireland, June 20-24, 2016, pp. 13–18, IEEE Computer Society, 2016. |
Towards Integrated Information Extraction and Facetted Search Applications in Nephrology Inproceedings Dragoni, Mauro; Recupero, Diego Reforgiato; Denecke, Kerstin; Deng, Yihan; Declerck, Thierry (Ed.): Joint Proceedings of the 2th Workshop on Emotions, Modality, Sentiment Analysis and the Semantic Web and the 1st International Workshop on Extraction and Processing of Rich Semantics from Medical Texts co-located with ESWC 2016, Heraklion, Greece, May 29, 2016., CEUR-WS.org, 2016. |
Multimodal multisensor activity annotation tool Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 17–20, ACM, 2016. |
Gaze-guided object classification using deep neural networks for attention-based computing Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 253–256, ACM, 2016. |
MedicaLVR: towards medical remote collaboration using virtual reality Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 321–324, ACM, 2016. |
Peripheral view calculation in virtual reality applications Inproceedings Lukowicz, Paul; Krüger, Antonio; Bulling, Andreas; Lim, Youn-Kyung; Patel, Shwetak N (Ed.): Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, September 12-16, 2016, pp. 333–336, ACM, 2016. |
Human Gaze and Focus-of-Attention in Dual Reality Human-Robot Collaboration Inproceedings 12th International Conference on Intelligent Environments, IE 2016, London, United Kingdom, September 14-16, 2016, pp. 238–241, IEEE, 2016. |
Model Based Augmentation and Testing of an Annotated Hand Pose Dataset Inproceedings Friedrich, Gerhard; Helmert, Malte; Wotawa, Franz (Ed.): KI 2016: Advances in Artificial Intelligence - 39th Annual German Conference on AI, Klagenfurt, Austria, September 26-30, 2016, Proceedings, pp. 17–29, Springer, 2016. |
Prediction of Gaze Estimation Error for Error-Aware Gaze-Based Interfaces Inproceedings Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 275–278, ACM Press, New York, NY, USA, 2016, ISBN: 9781450341257. |
2015 |
Journal Articles |
ModulAR: Eye-Controlled Vision Augmentations for Head Mounted Displays Journal Article IEEE Trans. Vis. Comput. Graph., 21 (11), pp. 1259–1268, 2015. |
Inproceedings |
Towards Episodic Memory Support for Dementia Patients by Recognizing Objects, Faces and Text in Eye Gaze Inproceedings Hölldobler, Steffen; Krötzsch, Markus; Peñaloza, Rafael; Rudolph, Sebastian (Ed.): KI 2015: Advances in Artificial Intelligence - 38th Annual German Conference on AI, Dresden, Germany, September 21-25, 2015, Proceedings, pp. 316–323, Springer, 2015. |
Cognitive Monitoring via Eye Tracking in Virtual Reality Pedestrian Environments Inproceedings Gehring, Sven; Krüger, Antonio; Alt, Florian; Taylor, Nick (Ed.): Proceedings of the 4th International Symposium on Pervasive Displays, PerDis 2015, Saarbrücken, Germany, June 10-12, 2015, pp. 269–270, ACM, 2015. |
2014 |
Journal Articles |
Integrating Digital Pens in Breast Imaging for Instant Knowledge Acquisition Journal Article AI Magazine, 35 (1), pp. 26–37, 2014. |
Inproceedings |
LabelMovie: Semi-supervised machine annotation tool with quality assurance and crowd-sourcing options for videos Inproceedings 12th International Workshop on Content-Based Multimedia Indexing, CBMI 2014, Klagenfurt, Austria, June 18-20, 2014, pp. 1–4, IEEE, 2014. |
A mixed reality head-mounted text translation system using eye gaze input Inproceedings Kuflik, Tsvi; Stock, Oliviero; Chai, Joyce Yue; Krüger, Antonio (Ed.): 19th International Conference on Intelligent User Interfaces, IUI 2014, Haifa, Israel, February 24-27, 2014, pp. 329–334, ACM, 2014. |
0000 |
Conferences |
Digital Pen Features Predict Task Difficulty and User Performance of Cognitive Tests Conference Forthcoming Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ' 20 ACM, Forthcoming, ISBN: 978-1-4503-6861-2/20/07. |