Context-Aware, Mixed-Initiative Support for Procedural Tasks: PrISM Project

Every day, we perform many tasks, including cooking, DIY, and medical self-care (like the COVID-19 self-test kit), which involve a series of discrete steps. Accurately executing all the steps can be difficult; when we try a new recipe, for example, we might have questions at any step and might make mistakes by skipping important steps or doing them in the wrong order. This project, Procedural Interaction from Sensing Module (PrISM), aims to support users in executing these kinds of tasks through dialogue-based interactions. By using multimodal sensors such as a camera, wearable devices like a smartwatch, and privacy-preserving ambient sensors like a Doppler Radar, an assistant can infer the user’s context and provide contextually situated help. From an HCI perspective, we explore the effectiveness of real-time human-AI collaboration in fostering user independence and facilitating skill acquisition.

Our framework is designed to be modular and flexible, allowing us to integrate various sensors and interactions for each specific situation. The underlying technology is multimodal Human Activity Recognition (HAR) and step tracking ([J.4] PrISM-Tracker), which allow the assistant to understand the user’s current step within the task. A variety of sensors can be used. Right now, we mainly use a smartwatch for its ubiquity, minimal privacy concerns compared to camera-based systems, and capability for monitoring a user across various daily activities. Because sensor signals can be ambiguous, PrISM uses the temporal structure of a procedure, represented as a graph, to resolve similar-looking actions: for a 12-step post-operative wound care procedure, smartwatch-only tracking improves from 37.3% to 56.7% accuracy.

Then, the assistant offers two types of interactions: question answering ([J.7] PrISM-Q&A) and proactive intervention ([C.20] PrISM-Observer). In Q&A, the assistant answers the user's questions (e.g., "What should I do next?"), where a Large Language Model (LLM) is complemented by step context to generate more accurate responses. In Observer, the assistant proactively intervenes when it detects that the user is about to make mistakes (e.g., "Have you wiped the pan?") by probabilistically modeling and forecasting user behavior using HAR. Across both interactions, the assistant shifts its initiative based on its confidence: it acts proactively when confident and returns the initiative to the user, asking for clarification, when uncertain.

Finally, we enable the assistant to leverage user-assistant dialogue as feedback to improve its supporting strategies ([C.22] PrISM-Dialogue). Mackay introduced the idea of a human-computer partnership, where humans and intelligent agents collaborate to outperform either working alone. Inspired by the idea, we proposed ways to enable our assistant to dynamically adapt through interactions after deployment. This helps the assistant improve context understanding and find a comfortable control balance, which also shifts over time, by exploring the mixed-initiative interaction design. Treating user questions, corrections, and confirmations as an additional sensing channel increased frame-level tracking accuracy from 45.4% to 71.3% and reduced inappropriate responses from 27.1% to 9.4%, while also enabling users to author new tasks and deploy an assistant from a single demonstration.

We are collaborating with medical professionals to apply our PrISM assistant at three clinical and research sites:

• University Hospitals Cleveland Medical Center: supporting post-operative skin cancer patients in following the prescribed wound care procedure at home. Through this NSF-funded collaboration, PrISM has been deployed with more than 200 patients.
• University of Pittsburgh School of Public Health: extending PrISM to caregiving and geriatric assessment.
• Fraunhofer Portugal AICOS: supporting people living with dementia in their daily routines, such as kitchen activities, through a project recognized by the Longitude Prize on Dementia. Check out our Autonomous Project.

These deployments expose challenges beyond controlled experiments, such as shifting sensing conditions, patients' varied routines, and their evolving preferences for proactive support, which motivate our ongoing work.

📣 We are excited to apply our assistant to more settings. Let us know if you are interested in collaborating with us!

This project is supported in part by NSF (No. 2406099), the CMU Center for Machine Learning and Health (CMLH), and the Masason Foundation. The development of the PrISM assistant for health applications is in collaboration with the Department of Dermatology at University Hospitals Cleveland Medical Center, the University of Pittsburgh School of Public Health, and Fraunhofer Portugal AICOS.