System Integrates Audio, Visual Cues and Artificial Intelligence to Improve Communication, Comprehension

Abstract

The University of Central Florida invention comprises systems and methods for processing a sender’s speech and dynamic visual features to provide the receiver with high signal-to-noise (SNR) ratio visual representations of the sender or other visual cues to improve comprehension. People who are deaf can benefit from the synthesized lip movement of the sender. The measured signals from the sender could also be transcoded into high SNR versions of other modalities, such as tactile signals for people who are blind or have low vision.

Verbal communication between two individuals, whether in person or mediated by technology, involves a sender who is speaking and a receiver who is listening. Various forms of noise can be introduced in the process, potentially making message comprehension by the receiver slower, more difficult, or completely erroneous. Examples of noise include background noise near the sender or the receiver, noise added to the signal during transmission, or even distance, poor articulation, accents, and hearing disorders. When the sender and receiver are physically proximal to each other or communicating via video, multiple visual cues can provide critical information or aid in comprehension.

The invention can be for one person (talking to oneself), two people (person-to-person), one-to-many (for example, public speaking), or many-to-one (such as choral singing). The sender(s) or receiver(s) could be AI agents. The invention can also be for recordings, such as audiobooks or music, where audio is first recorded and then played back later. However, the speaker/singer is not understandable because of the instruments, and so on.

Partnering Opportunity: The research team is seeking partners for licensing, research collaboration, or both.

Stage of Development: Prototype available.

Benefit

Market Application