Research Projects

Empirical Evaluation of Virtual Human Conversational and Affective Animations on Visual Attention in Inter-Personal Simulations

Creating realistic animations of virtual humans remains comparatively complex and expensive. This research explores the degree to which animation fidelity affects users’ gaze behavior when interacting in virtual reality training simulations that include virtual humans. Participants were randomly assigned to one of three conditions, wherein the virtual patient either: 1) was not animated; 2) played idle animations; or 3) played idle animations, looked at the participant when speaking, and lip-synced speech and facial gestures when conversing with the participant. Each participant’s gaze was recorded in an inter-personal interactive patient surveillance simulation. Results suggest that conversational and passive animations elicited visual attention in a similar manner, as compared to the no animation condition. Results also suggest that when participants face critical situations in inter-personal medical simulations, visual attention towards the virtual human decreases while gaze towards goal directed activities increases.

Investigating Implicit Racial Bias with Immersive Virtual Avatars

Immersive virtual avatars can powerfully affect a user's behavior. Some changes in behavior can be positive, although some can be negative and unknown to the user. A hypothesis could be made that embodying an immersive virtual avatar could negatively effect a user's implicit bias, this would be a serious cause for concern with regard to the recent emergence of consumer virtual reality games. We explore to see if one's implicit racial associations can be negatively increased/reinforced towards a particular group.

Applied Perception Research in Virtual Reality

The Real and Virtual Reaching Distance Estimation Apparatus. Do distance estimates differ between real and virtual environments? The potential discrepancy between distance judgements is important towards validating various virtual reality applications and simulations, such as driving and flight simulations. In several experiments we hope to understand reaching distance estimation in real and virtual environments, and how distance judgements are affected by various factors such as visual fidelity, field of view, object location, viewing condition and measurement protocol. In an initial experiment, we investigate reaching distance judgements measured by action and verbal estimates in real and virtual environments

Read more at the Applied Perception Research Page

A setup designed for experiments in near field distance estimation The virtual environment in the distance estimation setup

Near Field Depth Estimation and Perceptual Calibration on Large Screen Stereoscopic Displays

Depth estimation is crucial in many aspects of our daily life; it provides us our effective interaction with our environment. In immersive virtual environments (IVE), distances are mostly underestimated compared to the real world. To overcome these underestimations some studies are examined the effect of calibration on distance estimates in an IVE. But providing such an apparatus to examine this phenomenon can be computationally expensive. Our desire to use other off the shelf display technologies that are easier to set up for virtual reality training experiences and could provide a similar test bed to head mounted based VR systems motivated us to study interaction space or near-field depth perception on these large screen stereoscopic displays. In an attempt to push the envelope of immersion even further, we attempted to evoke a psychological phenomenon called the Rubber Hand Illusion (RHI). It opens up the possibility that a virtual object (In this case a virtual arm) can be experienced as part of user’s body. In this study, this ownership illusion is used in sync with stereoscopic time-multiplexed 3D to investigate near field depth perception by using 3D TV.

Center for Aviation and Automotive Technology Education using Virtual E-Schools (CA2VES)

We are developing virtual environments that are aimed at helping aviation and automotive technical college students in learn psychomotor tasks. The environment can teach students the proper techniques to use precision measurement tools like vernier calipers and micrometers and also educate students on the proper procedure for assembling a circuit board that meets a given specification. The aim of the project is to shift the focus of the contact hours spent with instructors to be more content than psychomotor skills.

From a virtual environments perspective, we will be able to study the effects of various immersive virtual environments techniques on the psychomotor skills acquisition in a measurement and assembly focused task. The process also includes finding the correct interaction paradigm that could maximize the transfer effects of the skills acquired in the virtual environments to practice in the real world.

Screenshots from the interactive virtual environment system for the Virtual E-Schools.

Clinical Virtual Environments for Preventive Medical Procedure Education

The goal of this project is to develop virtual environment system with interactive virtual humans for the purpose of teaching and training users in healthcare preventive medicine practices. From an epidemiological perspective, the spread of infectious diseases in hospitals is a significant factor in the nationwide rates of hospital acquired infections and high mortality rates in clinical settings. Recently, new procedures such as the CDC's Five Moments of Hand Hygiene have been developed for the purpose of teaching healthcare workers proper hand hygiene practices.

From a computational sciences perspective, technical advances will be made in scenario design, modeling and simulation of realistic, compelling and engaging training environment that replicate actual clinical situations. This also includes the modeling and simulation of complex behaviors in training scenarios for interactive virtual humans that represent virtual patients, healthcare workers, and other clinicians.

Screenshots from the interactive virtual environment system for hand hygiene training.
Virtual instructor (Evan), and virtual nurse (Simon) interacting with a virtual patient.

Duck Duck Punch: A Post-Stroke Neurorehabilitation Game

Duck Duck Punch Banner
We built a rehabilitation game for stroke survivors to help them with upper extremity impairment. Our game has been developed in collaboration with stroke therapists at the Medical University of South Carolina. Our game is a virtual shooting gallery where a stroke survivor hits targets with a virtual arm that they control via the Microsoft Kinect. The game compensates for impaired movement and allows users with severe impairment to fully interact with the game. Our game was built in Unity3D and makes heavy use of the Microsoft Kinect SDK.

Screenshot from Duck Duck Punch

SIDNIE: Scaffolded Interviews developed by Nurses In Education

When nurses learn how to interview patients, typically they practice with another student nurse playing the part of the patient. This is problematic since the nurse playing the patient is a far more medically educated individual than a typical patient and may respond to questions that a typical patient would not understand. Sometimes schools hire actors to play the part of the patient but that is expensive and limits the time the students may spend practicing.

We aim to provide an alternative method of practice, where students may interview virtual pediatric patients on a computer and receive feedback on their question choices. The SIDNIE system provides guidance through a virtual nurse, Sidnie, who helps students conduct a thorough interview by guiding them through the interview process and providing scored feedback. SIDNIE provides progressively less guidance as students become more competent and confident, enabling them to progress to independent interviews.

Additionally, the SIDNIE system provides a framework for scored practice for any question-answer task. We plan to continue research in making our system more realistic and effective for learning, as well as providing nurses authorship tools so that they can create their own scenarios.

Interface for SIDNIE
Interface for SIDNIE: Scoring Report

Simulation of Rapidly Deteriorating Patients for Nurse Training

There are signs that nurses can recognize in patients whose condition is rapidly deteriorating, and catching these signs quickly can allow a special team of doctors to come in and diagnose problems to help prevent the patient from crashing and potentially being admitted into the Intensive Care Unit of a hospital. However, nurses often miss the subtle trends of a deteriorating patient due to inexperience and lack of effective continuing education.

We are working with St. Francis Hospital in Greenville, SC to develop a simulation of rapidly deteriorating patients based on hundreds of actual case files of rapidly deteriorating patients annotated with signs that nurses should have noticed. This simulation will be used to train nurses to recognize patients in this condition in hopes of preventing the patient’s condition from getting worse and even in some cases preventing patient death. We have developed an initial prototype virtual hospital and electronic health record that allows a nurse to monitor four patients, observing their behaviors and using instruments to collect their vital signs. The nurses can record and review data in the electronic health record and can determine which patient(s) are undergoing rapid deterioration. Each of our patients’ data is from actual case files.

We designed our system with typical nurse demographics in mind, choosing interaction metaphors that are appropriate for people who may not be familiar with virtual environments. We have conducted user studies that validate our system design and are continuing to expand and improve the system.