Talking to Objects 2024 - Ongoing, Dataset, Microscope, Video with Lambda Vue
How do computer vision systems perceive and reconstruct speech from material vibrations, and what does this reveal about the ways AI translates between physical motion, sound, and language? When machines perceive our voices through matter, what new forms of communication and miscommunication emerge?
Computer vision models can recover speech from the subtle vibrations of objects. Using Lambda Vue, a software that amplifies minute motions in video, and a microscope, Talking to Objects captures microscopic voice markers from material vibrations. In this case study, tinfoyl (a phonetic writing of tinfoil) is spoken to a sheet of foil, generating a visual dataset that maps graphemes (letters) to phonemes (sounds).
Talking to Objects examines AI-mediated perception and computational sensing, exploring how machines “see,” “hear,” and “understand” the world through data. Computer vision systems capable of recovering speech from visual motion signal a shift in how communication is conceived: language becomes a physical event, and matter itself becomes a communicative surface. It asks what kind of “listening” occurs when AI detects voice through the movement of materials, and how such systems extend or displace human sensory perception.
By observing how computer vision detects and amplifies microscopic vibrations, the project found that machines treat physical motion as a kind of language—translating the invisible resonance of voice into visual data. These transformations expose the interpretive and speculative nature of machine sensing: AI does not simply capture speech but infers and imagines it, piecing together traces of motion and pattern to produce a voice.
In this way, technology becomes both translator and storyteller, reframing communication as a negotiation between signal, noise, and imagination. Technologies of perception do not just extend human senses, they invent new ways of speaking altogether.
With Caro Trigo, Mavis Yue Cao, Christie Wu