SETI/CETI Tricorder Tech: Tapping Into Whale Talk

Say you want to listen in on a group of super-intelligent aliens whose language you don’t understand, and whose spaceship only flies by Earth once an hour. It’s not unlike what Harvard scientists and others are doing, except their target species, sperm whales, thankfully live here on Earth.

As part of the nonprofit Project CETI (Cetacean Translation Initiative), an ambitious, multi-institutional endeavor to discern the language of sperm whales, engineers in the John A. Paulson School of Engineering and Applied Sciences (SEAS) and others have led the development of a powerful listening device that adheres to whales and records high-fidelity audio and other information that’s later analyzed by machine learning models.

The device, called a bio-logger, collects large, high-quality datasets of whale sounds called codas, which to human ears sound like a series of rhythmic clicks, along with contextual clues like physical behavior and ocean depth. The bio-logger is among the first to be explicitly designed to capture data for interpretation by machine learning algorithms. Modern machine learning techniques can help uncover structured, non-human communication by identifying patterns and frequencies in the whale codas that humans can’t readily perceive. — Harvard

The bio-logger has so far been deployed in whales off the Caribbean coast of Dominica during numerous deep-sea dives. The details of the device’s design and the inspiration behind it are published in PLOS One.

“When we were looking to decode the language of whales, a key value was to have the mics placed at the best spots, for the best audio recording possible,” said Daniel Vogt, lead Harvard SEAS engineer for Project CETI and first author of the bio-logger paper. “We looked at the state of the art, what was available out there, and there was nothing that really matched what we were looking for. So we made our own.”

A view of the CETI bio-logger’s main components. Credit Daniel Vogt/Project CETI/Harvard SEAS

Open-source technology

The Harvard-designed bio-logger and all its components and software are open-source, available to anyone in the marine biology or scientific communities. The researchers hope this structure will give rise to crowd-sourced innovation and possibly expand to other species.

“This really democratizes and opens up the field of marine science, to biologists across the world,” said David Gruber, founder and lead scientist of the five-year-old Project CETI, a National Geographic Society Program. Gruber’s work on whale communication began when he was a 2017-2018 Radcliffe Fellow.

The non-invasive bio-logger attaches to the skin of sperm whales via suction cups that were also designed by Harvard robotics researchers. It includes three synchronized, high-bandwidth hydrophones – underwater microphones – that can record sound from multiple whales talking to each other at different distances. Other features include GPS logging and transmission equipment, as well as sensors for depth, movement, orientation, temperature, and light.

Lead author and Harvard engineer Daniel Vogt testing equipment in the lab. — Spence Lowell

Whales can dive a mile down and stay underwater for an hour, surfacing for only a few minutes to breathe; the bio-logger is built to withstand those conditions, with battery life of about 16 hours, and audio sensitivity that picks up higher frequencies than humans can hear.

Legacy whale tagging technologies have in the past recorded many whale vocalizations and formed the basis for the field of cetacean communication. The CETI bio-logger builds on those foundational technologies but captures a richer array of data, including the ability to differentiate between different whales speaking by measuring their sounds’ origin. The datasets are helping researchers interpret the sounds and make sense of them, rather than just listening in.

Recent results have already proven out their methods. One published study used data from bio-loggers to show that sperm whales have their own alphabet; another reports a version of vowels and diphthongs in sperm whales’ language, similar to how humans speak.

Whales are tagged using a “tap and go” approach with specially adapted drones. Credit Jaime Rojo

Project CETI goals

Founded in 2020, Project CETI is the world’s largest interspecies communication initiative, involving eight institutions and 50 scientists who work in artificial intelligence, natural language processing, cryptography, linguistics, marine biology, and robotics. Harvard researchers have played key scientific roles; Vogt works in the lab of Robert Wood, the Harry Lewis and Marlyn McGrath Professor of Engineering and Applied Sciences, who leads the robotics component of Project CETI and whose lab designed the clingfish-inspired suction cups that adhere the bio-logger onto the whale’s skin. Stephanie Gil, assistant professor in computer science, designed a reinforcement learning framework with autonomous drones that are used to find whales and predict when they will surface so they can be tagged. In addition, the project’s lead linguist, Gašper Beguš, received his Ph.D. at Harvard.

Adhering a microphone to a sperm whale is a 10-out-of-10 challenge across multiple fronts: Using drones to tag the animals without bothering or hurting them. Getting the tags to stick amidst a salty, wavy ocean. Retrieving the devices. Extracting and interpreting the data. The Project CETI team has reached milestones on each of these endeavors. Bringing the bio-logger to many other labs and teams should edge the project even closer to its goal of understanding how sperm whales and other cetaceans communicate, and maybe one day, answering back in their language.

“This technology could now be expanded to the millions of other species we share the planet with” Gruber said. “I see this as a massive moment, because the field of bioacoustics and artificial Intelligence can now vastly expand.”

Learn more about Project CETI: https://www.projectceti.org/

An open-source bio-logger for studying cetacean behavior and communication, PLOS One (open access)

Over the past decade, bioacoustics associated with diverse marine life has become the focus of increasing research. While fixed acoustic devices play important roles in characterizing localized soundscapes, animal-worn devices that record audio alongside physiological metrics provide richer portals to understanding cetacean communication and characterizing sounds in their environment.

To facilitate scaling the collection of such multimodal datasets for deep learning applications and to encourage rapid prototyping for new recording capabilities, we present an open-source non-invasive bio-logger that can be deployed on marine animals to record high-quality audio synchronized with an extensible suite of behavioral and environmental sensors.

The current implementation is tailored to investigating sperm whale communication and biology. It features four suction cups, three high-bandwidth synchronized hydrophones for audio analysis including directionality, GPS logging and transmission, and sensors for pressure, motion, orientation, temperature, and light.

Its hardware and software are both open-source, with designs, fabrication details, and code available online. Lab-based experiments characterize and validate performance including shear adhesion forces, withstanding pressures equivalent to 560 m depths, battery life up to 16.8 hours, audio sensitivity of –205 dB re FS/μPa with a 96 dB dynamic range, multi-threaded data acquisition, drone-based deployments, and GPS-based recoveries. Field experiments record sperm whale vocalizations and behaviors spanning 10 deployments, 44 hours of recording, 20 dives, and up to 967 m depths.

Altogether, this platform aims to advance the understanding of marine animal biology and communication within the rapidly evolving and intersecting areas of robotics, bioacoustics, and machine learning.

Astrobiology, CETI, SETI, AI, Tricorder,