Adaptive Art Technologies – Audio Synthesis via Accessible Controllers
📹 Video presentation · 📖 User manual · 🎛 Max patches
AAT Synth is an accessible musical instrument that generates music through movement and gestures. The project consists of two main components:
- An iOS application written in SwiftUI using Xcode. The instrument runs on iOS devices due to the platform's popularity in the disability community. Additionally, the device's embedded sensors and peripherals provide a feedback loop between the performer's actions and the instrument's generated sounds.
- A series of Max patches designed for behavior exploration, serving as a starting point for developing the iOS app. The patches can be found in the MaxPatches folder.
Designed by David Cardona, Assistant Professor at Berklee College of Music, as the capstone project for the Digital Media Design program at Harvard Extension School. Awarded the Dean's Prize for Outstanding ALM Capstone.
The project has been presented at:
- AES International Symposium on AI and the Musician — "AI, Electronic Music, and Creativity: Exploring the Future of Electronic Production and Design," with Akito van Troyer, Ryan Page, Rachel Rome, and Timo Preece. Berklee College of Music, Boston, MA, June 2024.
- ABLE Assembly, Berklee Institute for Accessible Arts and Education — "Designing for the Margins," Boston, MA, April 2024.
- ACM SIGGRAPH 2022 — "Adaptive Art Technologies: Enabling Creative Expression," with Robert Fischer. Diversity, Equity, and Inclusion program. Vancouver, BC, August 2022.
Current sensing technologies, such as IMU sensors and cameras, allow the collection of data to describe movement. AAT Synth is an accessible musical instrument that relies on these technologies to create interactive musical experiences. The instrument was designed around accessibility, resulting in flexible interactive approaches to make music. This project stands at the intersection of music, technology, and accessibility, and introduces emerging technologies into creative expression. AAT Synth is inclusive by design, allowing users with varying degrees of musical and technical proficiency, as well as physical capabilities, to engage in music performance. The project features a custom-built iOS application and a series of Max patches. The application is a versatile and interactive musical instrument that relies on motion sensors and computer vision data. The Max patches serve as building blocks allowing people working in Max to incorporate these types of data into their designs.
- Facilitate creative expression for people with disabilities by utilizing inertial measurement unit (IMU) sensors and computer vision (CV).
- Build a musical instrument that enables creative expression by people with varying degrees of musical and technical proficiency.
- Consider numerous types of physical and cognitive disabilities to inform the development of the design.
- Process raw data captured from 9-DOF IMU sensors, leaning on quaternions, Euler angles, and polar coordinates to describe movement and control parameters of the audio DSP engine.
- Invent meaningful relationships between movement and sound, providing new approaches for musical expression using current mobile devices.
- Develop an iOS application in SwiftUI using Xcode that provides real-time audio experiences that maximize the available input mechanisms of iOS devices.
- Design implementations of advanced computer science and computer architecture concepts to optimize real-time interactivity, guaranteeing intuitive and efficient workflows between the UI, input management, sensor-related services, data management, and output content.
- Explore approaches to maximize graphics processing in sensor fusion and audio-related workflows, serving as the foundation for a future graphics synthesis engine implementation.
- Deploy the iOS application to users via TestFlight and potentially via Apple's App Store.
- Leverage Apple's Continuous Integration and Continuous Delivery (CI/CD) tools, such as Xcode Cloud.
- Create prototypes in the form of Max patches that interface with the mobile applications that provide access to the device's various sensors, to serve as building blocks to prototype designs that map sensor data to music and audio content.
My capstone project, Adaptive Art Technologies – Audio Synthesis via Accessible Controllers, has given me valuable insights, resulting in meaningful conclusions and ideas to further my field of research. In particular, the development and design journey proved to be a notable experience in interdisciplinary studies, overcoming technical challenges, user-centric design considerations, educational values, and opportunities for broader impact.
The heart of this project is interactivity. My core achievement has been to seamlessly bridge technology, musical expression, and accessibility. By utilizing motion and vision-based controllers, I was able to develop a novel musical instrument that aims to break barriers in creative expression. My hope is for my work to serve as a leap forward in inclusive design, bringing music creation to a wider audience by including people with disabilities and people with varying degrees of musical and technical proficiency.
I found SwiftUI and iOS development to be sophisticated approaches to leverage portable and accessible creative solutions. Pairing the mobile device's input and output systems (e.g., motion sensors, camera devices, and onboard speakers) with its powerful onboard multi-threaded processor, it was possible to execute digital signal processing, computer vision, and mathematical algorithms to convert physical gestures into meaningful musical generation.
By considering accessible user-centric design practices from the outset I was able to tap into future-proof solutions for diverse audiences. This project reaffirmed my belief that accessibility and inclusivity in design not only enhance user experience, but are also the fundamental driving forces behind true innovation. I also believe that this project is a valuable educational resource in digital media design and music technology, serving as a blueprint for others to learn from and build upon. For instance, the Max patches I developed are practical building blocks for people in this field to explore and further innovate.
Throughout this process I faced numerous challenges, especially in defining intuitive and meaningful mappings from physical gestures to sound, as well as implementing smooth real-time interactivity. I now have a more robust understanding of accessible UI/UX design, real-time data processing, and human-computer interaction (HCI).
This project lays a strong foundation for future developments and explorations. I intend to further explore HCI in creative expression, both by expanding upon music synthesis and by immersing myself in video synthesis engines. This opens the door to new possibilities, leveraging graphics processing capabilities to achieve audio reactivity, as well as providing new means of expression for people who are deaf and hard of hearing.
I believe that this project sets a precedent for future developments in multidisciplinary practices. Not only does it contribute to the individual fields of art, technology, and accessibility, but it also leads to societal impact by fostering inclusive practices that have the potential to enhance someone's quality of life.
Adaptive Art Technologies (AAT) is a development initiative seeking to create novel musical instruments through alternative controllers.
The design is based on a modular approach that keeps the media engines separate from the input controllers. This facilitates multiple approaches for creative expression. The main idea is to craft intuitive and accessible mechanisms to interact with the system, opening the doors for everyone to create a meaningful musical experience.
Currently, the project implements an audio synthesis engine, a motion controller that relies on the built-in sensors of your device, and a computer vision controller that relies on feature detection through your device's camera. Future efforts aim to expand this project by including a video synthesis engine.
This project is led by multi-disciplinary artist and designer David Cardona.
The icon of the application was designed in collaboration with Elif Kavuşturan.
The logo of the application was designed using the macOS app Sketch, and the resulting image was converted to a flattened PNG (no transparency) using Adobe Photoshop.
The main product is an iOS application that utilizes a motion and a computer vision controller to operate an audio synthesizer engine. The controllers modify engine parameters to control the musical performance via the performer's interactions.
The following table describes the frameworks used to drive each modular component:
| Component | Framework |
|---|---|
| Motion Controller | Core Motion |
| Computer Vision Controller | Vision |
| Audio Engine | AudioKit |
For the synthesizer architecture, parameter reference, and controller mappings, see MANUAL.md.
Building the application requires macOS running Xcode. The following package dependencies are required:
Note that all these packages should already be included in the Xcode workspace, but if needed, they can be installed using the Swift package manager.
It is necessary to test the application directly on the iOS device, given that motion sensors aren't available in macOS. Additionally, the camera routine for iOS differs from that of macOS.
As of the time of this writing, the app must be installed using Apple's TestFlight. This requires an invitation from the developer. If you're interested in trying out the application, please send an email to david@sound-architect.com.
Two Max patches were created for this project:
- ZigSimConnection brings in video and motion data from ZIG SIM Pro.
- FaceDetection performs face detection on an incoming video stream using OpenCV.
The purpose of these patches is to provide input to Max in order to perform creative explorations based on motion and computer vision.
Installation requirements, usage notes, networking protocols, and the mathematical operations used to process sensor data are documented in MANUAL.md.
- MANUAL.md — user manual and technical reference, covering the audio engine, performance and controller settings, gesture-to-parameter mappings, Max patch installation and usage, networking protocols, and development resources.
Designed by David Cardona.
Application icon designed in collaboration with Elif Kavuşturan.
Software — the iOS application and its source are released under the MIT License. See LICENSE.
Max patches — the patches in MaxPatches are dual-licensed under MIT or Creative Commons Attribution 4.0 International (CC BY 4.0), at your option. They were built as shared teaching and prototyping resources; CC BY is offered so that reuse in papers, performances, and coursework carries citation.
Application icon — designed by Elif Kavuşturan and used here with permission. The icon is not covered by either license above and is not licensed for reuse, modification, or redistribution. All rights reserved by the designer.