Embodied conversational agents will be used for exploring multimodal approaches of active input in virtual reality (combination of speech and pen input) to determine the cognitive status of the user. Contact Fabrizio Nunnari
Cognitive assessments have been the subject of recent debate because there are limitations when they are conducted using pen and paper. For example, the collected material is monomodal (written form) and there is no direct digitalization for further and automatic processing. Here we show how such cognitive assessments can be Read more…
We present a virtual reality (VR) application that enables interactive image clustering and fine-tuning of a deep neural network by using natural hand gestures. Using a pre-trained AlexNet, we apply PCA and t-SNE dimensionality reduction techniques to project image data sets into the 3D VR space. Resulting spatial clusters of Read more…
Recently there has been a number of research questions regarding user interaction in virtual reality, but only a few are concerned with the interaction using real world objects. For complex data entry we use a tablet with integrated stylus, that is represented in virtual reality through real-time tracking of markers. Read more…
We explore clustering of objects in a 3D virtual environment. As traditional clustering is done in 2D, we investigate how an additional dimension influences the machine learning process when clustering is done interactively in virtual reality. Through the use of intuitive hand gestures performed using the Oculus Rift controllers, the user Read more…
Traditionally neurocognitive testing is done using pen and paper, which is both expensive and time consuming and often leads to a biased outcome. As part of the Intera-KT project select and digitize existing cognitive tests and support the assessment of cognitive impairments through automated evaluation of different input modalities recorded Read more…
In the medical domain physicians are often forced to decide quickly whether a further examination is appropriate or which of several treatments is best suited for a patient.A first preliminary ruling can sometimes be made on the basis of a few aspects, e.g. with so called ABCDE rule for the Read more…
As medical records may cover a very long history of diseases (up to 30 years) and include a vast number of diagnoses, symptoms, results, medications, and laboratory values, we could highly benefit from advanced search capabilities in clinical information systems to allow for the retrieval of relevant data. We propose Read more…
In comparison to typing, handwriting stimulates different parts of the human brain and brings into play a very different cognitive process. Many of today’s tasks benefit from digitalization and digitization of handwriting input and with the availability of low-cost consumer grade capturing devices, such as digital pens, graphic tablets and Read more…
Handwriting provides a natural and intuitive way to input text in virtual reality. We explore different approaches on how sketched user input can be captured, here we use the Oculus Rift controller as a substitute for a real pen. The user “grabs” the virtual pen using the index finger and Read more…