Skip to content

index.md

Latest commit

 

History

History
85 lines (64 loc) · 5.69 KB

index.md

File metadata and controls

85 lines (64 loc) · 5.69 KB
Raw
layout default
title Serial Communication
description Build end-to-end interactive systems connecting Arduino with the web: serial communication, Web Serial, p5.js, and machine learning with ml5.js, BodyPose, and HandPose.
nav_order 4
has_toc false
has_children true
comments true
usetocbot true

{{ page.title }}

{: .no_toc }

Table of Contents

{: .no_toc .text-delta }

  1. TOC {:toc}

In this lesson series, we will learn about Arduino-to-Computer interaction. More specifically, serial communication, web serial, and using p5.js to communicate bidirectionally with Arduino. By the end of this module, you'll be able to build full end-to-end interactive systems that combine the power of web-based tools—including machine learning—with the physical interactivity of Arduino. 🔌

{: .note }

What you'll learn in this module:

  • How asynchronous serial communication works and how to use it for bidirectional Computer ↔ Arduino communication
  • How to use the Web Serial API to build browser-based apps that talk to Arduino
  • How to use p5.js for creative coding with serial input and output
  • How to build complete interactive systems with bidirectional communication
  • How to integrate machine learning (ml5.js) with Arduino for gesture- and pose-controlled physical computing

L1: Intro to Serial

In this lesson, we'll dive into asynchronous serial communication and how we can use it for bidirectional Computer ↔ Arduino communication. We'll show example serial communication clients using terminal programs and Python.

L2: Web Serial

In this lesson, you'll learn about the Web Serial API and how to build simple web apps that communicate with Arduino directly from your browser.

L3: p5.js Serial

In this lesson, you'll learn about Processing, p5.js, and how to use p5.js with Web Serial (with a focus on processing incoming serial input data).

L4: p5.js Serial I/O

In this lesson, we look at creating more complicated p5.js + Arduino applications that bidirectionally communicate together both Arduino to p5.js (Arduino → Computer) as well as p5.js to Arduino (Computer → Arduino).

L5: Paint I/O Example

In this lesson, we will bring everything together thus far to build a fully interactive p5.js + Arduino painting application, called PaintIO, with bidirectional serial communication and a custom paintbrush controller using an accelerometer to control brush location, a force-sensitive resistor to control brush size, and bimanual interaction using both Arduino + the mouse simultaneously for drawing.

L6: ml5.js Serial

In this lesson, we'll introduce machine learning frameworks like ml5.js and show how to use them with Arduino. Specifically, we'll use ml5's BodyPose model for real-time pose estimation and send body keypoint data to Arduino via Web Serial.

L7: HandPose Serial

In this lesson, we'll explore ml5's HandPose model for real-time hand tracking and use it to control a servo motor on Arduino. We'll also build a fun physical construction (a cardboard figure that waves back at you!) to demonstrate the connection between ML-based input and physical output.