Gulf of Mexico Interactive Deep-Sea Mosaic Sonification

Original project date: 3/14/2016

An interactive data visualization + sonification project exploring a deep-sea coral ecosystem in the Gulf of Mexico through image, sound, and user-driven navigation.

This project was originally created in March 2016 for Datascapes W16. I am keeping it public because it still reflects a lot of what I value in my work: curiosity, interdisciplinary thinking, data-driven storytelling, and a willingness to build unusual interfaces around complex scientific datasets.

Overview

This sketch turns ecological survey data from a deep-sea coral site in the Gulf of Mexico into an explorable audiovisual experience. The current implementation focuses on mosaic H1, a site in Mississippi Canyon 751, south of Louisiana.

The underlying data come from deep-sea coral research in the Gulf of Mexico and include:

• ecological composition at survey sites
• digitized organism annotations
• dive tracks and related expedition metadata
• imagery collected before and after the Deepwater Horizon oil spill

At the time I built this project, I had already worked with these datasets scientifically as an undergraduate researcher in the Fisher Lab at Penn State. This project gave me the chance to approach the same data from a very different angle: not just as analysis, but as an interactive and aesthetic system.

Project concept

The main goal was to create a soundscape that lets users explore a benthic mosaic visually and sonically.

The interface combines:

• a large mosaic image
• a movable “ROV view” controlled by the mouse
• a digitized overlay of organisms within the selected region
• a set of continuously looping sounds whose amplitudes change based on the organisms present in view

As the user moves across the mosaic, the system measures how much of each organism category is visible and maps that composition to sound. The result is a data-driven ambient soundscape shaped by the ecology of the site.

Why this dataset is interesting

These mosaics were created as part of the Lophelia II project, a four-year effort (2009–2012) funded by NOAA Office of Exploration and Research (OER) and the Bureau of Ocean Energy Management (BOEM), with contributions from multiple academic institutions and the U.S. Geological Survey.

The project is especially compelling because:

• the mosaics were built from images collected in 2009 and 2010
• the Deepwater Horizon oil spill occurred in April 2010
• the dataset therefore captures a before/after frame around a major environmental disaster

The Gulf of Mexico is also a uniquely layered environment: ecologically rich, geologically active, and economically dominated by offshore oil and gas infrastructure. That tension between deep-sea life and industrial presence was part of the emotional and conceptual motivation for this piece.

My connection to the project

This project was informed by my own field experience in the Gulf. In 2014, I was part of the Corps of Exploration aboard Dr. Robert Ballard’s E/V Nautilus. Being surrounded by oil infrastructure while watching life unfold thousands of meters below the surface made a strong impression on me.

I wanted this piece to capture some combination of:

• scientific observation
• environmental context
• exploration
• wonder
• discovery

How it works

The project was built as a p5.js sketch.

Interaction

• A splash screen presents a map of the Gulf region and site markers
• Buttons on the map open a mosaic entry from the list in js/config.js (today both are site H1, as survey vs geo imagery—additional sites or years appear when you add assets)
• The mouse controls the position of a zoomed “ROV eye view”
• Clicking on the mosaic pins that patch in the lower panel, overlays the digitized annotation map there, and updates sonification from the pixels in view (clicking does not switch survey year)
• Pressing any key cycles through loaded mosaics in button order so you can compare change across those views or time periods once multiple years are wired in

Data-to-sound mapping

• Organisms in the digitized overlay are assigned specific colors
• I adjusted color values so each class had a distinct red-channel value
• The selected image region is sampled pixel-by-pixel
• The proportion of each organism class in view is calculated from those pixel values
• Those proportions are mapped to the amplitudes of looping sound elements

Sound design

The sounds were designed artistically rather than biologically “literally,” but with some ecological logic:

• dead Lophelia acts as a low, stable foundation
• coral classes tend to have steadier, lower tones
• mobile fauna tend to be higher-pitched and more active

I used a mix of software instruments and drum/percussive samples to build the sound palette.

Organism classes

Across all surveyed sites, the original dataset includes 32 distinct organism/substrate categories.
The current implementation covers mosaic H1, which includes 13 distinct organism categories and sounds.

These include both:

• substrates digitized as polygons
  (e.g. live coral, coral skeleton, tubeworms, encrusting sponges)
• solitary fauna digitized as points
  (e.g. mussels, whip coral, crabs, fish, anemones, urchins, giant isopods, ophiuroids)

Motivation

This project was shaped by a few overlapping themes:

• Change over time
  The before/after framing around Deepwater Horizon made temporal comparison central to the work.

• Exploration
  I wanted the interface to feel like discovering something, not just reading a chart.

• Micro vs. macro scales
  Inspired partly by Alien Ocean and discussions in Datascapes, I was interested in connecting large environmental systems to smaller biological detail.

• Scientific data as artistic material
  This project came from the unusual position of already knowing the dataset scientifically, then returning to it through visualization and sonification.

Technologies

• p5.js
• custom image / pixel-based class detection
• digital audio assets and looping amplitude control
• ecological annotation data derived from deep-sea survey mosaics

Adding another mosaic

1. Add the RGB mosaic and matching digitization PNGs under assets/ (same pixel dimensions as each other).
2. Append one object to the MOSAICS array in js/config.js with id, label (button text), rgbPath, and digiPath.
3. Serve the site over HTTP (for example python3 -m http.server) and confirm the new button appears, the view scales correctly, and key-cycling includes the new entry.

Species classes and sounds are shared across mosaics (SPECIES in js/config.js). If a future mosaic used a different legend, you would extend the data model (separate palette or per-mosaic species tables) before wiring it in js/palette.js.

Current limitations

This repository preserves the project in a largely historical form. A few limitations:

• only site H1 has mosaic assets checked in—two related views (survey mosaic vs geo mosaic), not multiple survey years until more PNG pairs are added
• UI/navigation is minimal
• sound mappings are handcrafted and exploratory
• the project does not yet include a built-in key for all organisms/classes
• some ideas described in the original proposal were never completed

Future ideas (original + still interesting)

Some directions I originally wanted to pursue:

• support for all 24 surveyed sites
• site buttons / richer map navigation
• smoother “underwater” camera motion
• an in-app legend and organism guide
• more robust time comparison tools between 2009 and 2010
• incorporating DNA-level sonification for species represented in public sequence databases

Why I still keep this online

Although this is an old project, I still think it represents something real about how I work:

• I like crossing disciplines
• I like unusual interfaces
• I like turning complex datasets into something explorable
• I am comfortable mixing technical, scientific, and creative approaches

It is not a polished production system. It is a 2016 student project. But it is also a genuine early example of data storytelling, scientific visualization, and interactive design in my work.

Acknowledgments

The ecological datasets used in this project were provided courtesy of the Fisher Lab.
The broader research context comes from deep-sea coral work in the Gulf of Mexico, including the Lophelia II project.

References

• Helmreich, Stefan. Alien Ocean: Anthropological Voyages in Microbial Seas. University of California Press, 2009.
• Fisher, Charles, Harry Roberts, Erik Cordes, and Bernie Bernard. “Cold Seeps and Associated Communities of the Gulf of Mexico.” Oceanography 20(4): 118–129.
• Prouty, Nancy G., Charles R. Fisher, Amanda W.J. Demopoulos, and Ellen R.M. Druffel. “Growth Rates and Ages of Deep-sea Corals Impacted by the Deepwater Horizon Oil Spill.” Deep Sea Research Part II (2014).
• Friedel, Robert, and Edward R. Tufte. “Envisioning Information.” Technology and Culture 33(1): 143.