About us
We are an advanced Human-Computer Interaction research lab in the Computer Science and
Engineering Department at the University of Michigan. Our mission is to redefine hearing as a
programmable, editable, and hyperpersonal experience—not a fixed sense, but a customizable
interface between humans and their world—much like how web design lets us program, rearrange,
and personalize visual elements.
We design human-centered AI systems that empower people to shape how they hear, perceive, and
interact with sound. Our research spans accessibility, healthcare, and entertainment domains,
with current projects including editable digital media soundscapes, relational audio tools for
cross-neurotype communication, and adaptive hearing systems for clinical environments.
Accessibility is a core driver of our work. We view sound personalization not just as a
convenience, but as a powerful way to make sound more inclusive and equitable. This includes
developing systems for real-time audio captioning, enabling users to edit or customize captions,
and creating tools that translate sound into formats tailored to individual sensory and
cognitive needs.
We focus on accessibility because it is the entry point to future interfaces. Communities that
rely on captioning, sensory augmentation, or neurodivergent-friendly tools are often the
earliest adopters of new technologies. By designing for these users, we uncover what's next for
everyone.
Our work is already making real-world impact. The SoundWatch
app for sound awareness on smartwatches has over 4,000 downloads. Our indoor navigation
system for visually impaired users has been used more than 100,000 times in museums across
India. Our clinical communication tools are being deployed at Michigan Medicine. A
feature we pioneered for people with paralysis is now built into every iPhone, and our
work has directly influenced real-time captioning tools at Google.
These outcomes are made possible by our lab’s deeply collaborative, community-centered research.
Our team includes HCI and AI researchers working with healthcare professionals, neuroscientists,
engineers, psychologists, Deaf and disability studies scholars, musicians, designers, and sound
artists. We partner closely with community members, end users, and organizations to tackle
complex challenges and turn ideas into meaningful, trusted technologies.
Looking forward, we imagine a future of auditory superintelligence—systems that don’t just
support hearing, but expand it. These tools will filter, interpret, and reshape soundscapes in
real time, adapting to our needs and context. From cognitive hearing aids to sound-based memory
support and emotion-aware audio companions, we aim to make hearing a fully customizable
experience.
Explore our projects below—and get in touch if you’re interested in working
with us.
Recent News
Oct 28, 2026: Dhruv Jain is giving a keynote at DCASE 2026 in Boston, entitled
"Beyond Accuracy: The Human
Side of Sound AI"!
Oct 9, 2026: PreView, on rehearsing real routes in VR with low vision, accepted
as a
poster to UIST 2026!
July 14, 2026: SoundTrace, our work on real-time sound recognition with
temporal
context and episodic memory, accepted in IMWUT!
Apr 30, 2026: Our mixed-methods study on noise, hearing, and communication in
the
operating room published in Otolaryngology-Head and Neck Surgery!
Mar 2, 2026: RAVEN accepted at CHI 2026!
Jan 15, 2026: Our pilot study on medical caption technology for
physician-patient
communication published in JMIR Rehabilitation and Assistive Technologies!
Oct 26, 2025: RAVEN demo received the Best Demo Award at ASSETS 2025! Congrats,
Xinyun and team!
July 14, 2025: New funding alert! Our OpenAI NextGenAI proposal on rich
auditory
scene awareness ($50,000) has been funded!
Aug 26, 2025: Best paper honorable mention for CARTGPT at ASSETS 2025!
Aug 14, 2025: Three demo papers accepted to ASSETS 2025: RAVEN, EvolveCaptions,
and
CapTune!
Jul 7, 2025: Our work on sound personalization for digital media accessibility
has
been accepted to ICMI 2025!
Jul 3, 2025: Three papers on improving DHH accessibility accepted to ASSETS
2025:
SoundNarratives, CapTune, and CARTGPT! See you soon in Denver.
Jun 7, 2025: Our work on designing an adaptive music system for exercise has
been
accepted to ISMIR 2025!
