Echoes of Motion

Beyond Explicit Control: Implicit motion and readiness signals for spatial surfaces

2025 | Personal Research Project

Echoes of Motion is a design research inquiry into how bodily motion in human-computer interaction becomes legible, interpretable, and meaningful through mediated representation. Through computational mediation, visual abstraction, and temporal smoothing, the project examines how representational choices shape users’ perception of movement and their mental models of system responsiveness.

The project examines how representational choices shape users’ perception of movement and their mental models of system responsiveness. Rather than treating movement as a command, the system establishes a reciprocal feedback loop in which bodily motion and system response continuously inform one another, allowing users to perceive their own movement through the system’s interpretive lens.

Early development: Gestural input and ink-like response forming a continuous feedback loop.

Intention

Echoes of Motion began with a question: How can an environment recognize when a person is ready to interact—before any explicit command is given?

In everyday life, humans rarely initiate interaction through precise instructions. Instead, we signal readiness through the body by turning toward a space, slowing down, making eye contacts, or raising a hand. From greeting a passerby to engaging with a responsive environment, interaction often begins as an embodied invitation rather than a deliberate control action.

The project situates this question within the architectural context of a wall. Walls are among the most familiar spatial boundaries in human environments. They separate spaces, regulate access, and quietly signal where interaction is not expected to occur. In cities especially, shared walls line streets and public corridors. They are present, visible, yet rarely addressed as interactive surfaces.

Echoes of Motion asks what might happen if a wall were no longer treated solely as a boundary, but as an interface. By transforming a static wall into a responsive surface, the project explores how an architectural divider might become a site of connection rather than separation. Interaction shifts away from personal screens and toward shared spatial experience, inviting passersby to engage with a surface they would normally ignore or avoid.

Drawing inspiration from everyday gestures, the project treats movement not only as expressive output, but as a language through which humans communicate readiness, curiosity, and engagement with their surroundings. In this context, the wall becomes an active participant—sensing bodily cues and responding in ways that feel intuitive rather than instructed.

How can an interactive system recognize a user's readiness for interaction without requiring explicit commands?

Research & Inquiry

I approached the wall as a spatial interface and asked how gesture might function as sensing rather than control. Rather than treating interaction as a sequence of commands, I was interested in how a system could detect a user’s readiness to engage.

Gestures as Readiness Signal

In human communication, certain gestures function less as explicit commands and more as signals of readiness and intent. An open palm, for example, commonly appears in moments of greeting, pause, or hesitation. Rather than prescribing a specific action, it communicates availability, caution, or a desire to regulate engagement.

Across everyday interactions as well as their representations in media, such gestures often mark transitions in interaction states. They appear at moments when an encounter is about to begin, shift, or momentarily stop. What is communicated is not instruction, but presence: an indication that interaction is possible, negotiable, or being recalibrated.

Drawing from perspectives in embodied interaction and social signaling, I interpret the open palm not as a control gesture, but as an example of a broader class of readiness signals. These gestures carry ambiguity by design, allowing participants to modulate interaction without committing to a predefined outcome. This quality makes them particularly relevant for designing open-ended, exploratory interaction systems, where sensing readiness and intent matters more than recognizing explicit commands.

Result of gesture exploration in space — Spontaneous hand movement and interaction exploration before defining mappings.

Refining palm openness with Mediapipe — Feature study: defining palm openness with Mediapipe.

Ink diffusion triggered by open palm — Early mapping: open palm gesture → ink trigger.

Conceptual Frame

Echoes of Motion draws from embodied interaction and ecological perception, treating movement as a form of sensing. It aims to build continuous, expressive, and context-dependent interaction rather than discrete commands. This way, gesture becomes both expression and perception shaped by rhythm, hesitation, and attention.

From Result to Questions

The prototypes above helped me identify which qualities of movement feel meaningful in interaction, and how a wall might respond as an active partner in a shared environment.

Design Mapping

Visual Medium

The visual system draws inspiration from shuimo (水墨), traditional Chinese ink-wash painting, where ink interacts dynamically with water and paper rather than remaining fixed or contained. In these works, a single mark can spread, bleed, or dissipate over time, shaped by material conditions rather than precise control. Echoes of Motion adopts digital ink as an interactive medium to mirror this logic. Bodily cues act as triggers on a virtual wet canvas: when activated, ink emerges, diffuses, and leaves traces that persist and evolve like ripples. This material metaphor supports interaction as gradual, negotiated, and irreversible, aligning visual response with embodied presence rather than discrete commands.

Stroke Mapping

In shuimo painting, stroke behavior varies according to the degree of brush wetness, allowing material conditions to shape how ink spreads, bleeds, or settles on paper. Echoes of Motion adopts a similar principle by modulating the canvas’s response behavior through feedback loops and noise-diffusion logic, enabling visual outcomes to emerge from interaction dynamics rather than fixed mappings.

Technical Approach

To explore how an environment might recognize readiness for interaction, I designed a three-layer pipeline that treats gesture not as a command, but as a signal of intent.

Sensing → Interpretation → Rendering

Sensing captures bodily movement from the environment. Interpretation translates expressive qualities of gesture into interaction states through custom-defined logic. Rendering maps interpreted states into real-time visual behaviors that respond back to the user. Together, these layers form a continuous feedback loop between human motion and the spatial interface.

Why these features?

Instead of mapping raw position directly to visual output, the system interprets higher-level expressive qualities—such as rhythm, hesitation, and continuity—so interaction unfolds as an ongoing process rather than a discrete event.

This approach allows the wall to follow the user’s embodied gestures over time, supporting open-ended exploration and improvisation through movement.

Feature signals are streamed via OSC into TouchDesigner, where they drive shader parameters controlling ink propagation, diffusion (wetness), flow bias (gradient field), and color modulation. This supports material-like behavior while maintaining low latency.

Feature Mapping (Gesture → Ink Behavior)

Hand openness → stroke activation

Mapping rhythm to diffusion timing — **Rhythm & continuity** → propagation speed / bloom timing

These mappings were iterated through rapid prototyping: each feature was tested for stability, perceptual clarity, and how well it invited exploration. The goal was not one-to-one control, but expressive coupling—where the wall’s response remains interpretable yet surprising.

Development

Static stroke color baseline — Noise-based stroke color generation

Here are some of the iterations, prototypes, and early mistakes. Each experiment nudged the system a little closer to a fluid, expressive interaction.

Iteration 01 — Noise-Modulated Color Field

Approach: Using RGB masks driven by simplex noise to let color drift and fluctuate, allowing each stroke to carry subtle unpredictability and material presence.

Result: Noise modulation added brush stroke color variation, resulting a more stochastic ink selection. This gave more color depth variation for diffusion.”

Close-up of pigment-like color variation — Local pigment variation

Noise-modulated RGB mask color dynamics — RGB masking for stroke color

Iteration 02 — Ink Propagation & Diffusion Control

Problem: Immediate diffusion made responses feel abrupt and less exploratory.

Approach: Separated ink behavior into propagation delay, wetness-driven diffusion, and gradient-biased flow (slope field).

Result: Slower bloom encouraged sustained gestures, while wet/dry variation help gives nuanced texture for alteration. This gave the ink controllable parameters with temporal, diffusion, and directional variations.

Fast spread wet ink behavior — Variant A: fast propagation + high wetness

Slow bloom delayed ink behavior — Variant B: delayed propagation (slow bloom)

Dry edge crisp ink behavior — Variant C: low wetness (crisp edges)

In basic shader terms, ink can be treated as a density buffer that accumulates over time. A wetness map modulates diffusion strength (bleed vs crisp edges), while a gradient field biases motion direction (drift), producing material-like spreading without sacrificing responsiveness.

Iteration 03 — Sensing Stabilization & More Interactive Gesture Mapping

Problem:The interaction lacked fluidity for two reasons. First, webcam-based Z-depth sensing was noisy, causing unstable approach and retreat detection. Second, the interaction vocabulary was limited to triggering and dragging within a single stroke, restricting expressive control.

Experiment: I addressed sensing stability and interaction expressivity in parallel. To stabilize depth input, I applied a Kalman filter to smooth noisy Z-depth measurements and maintain a predictive state estimate. To expand gestural control, I introduced additional gesture interpretations beyond basic triggering, including approach/retreat, hand openness, and hand rotation.

Observation Depth smoothing significantly reduced jitter and improved the reliability of proximity-based interaction. At the same time, mapping multiple gesture attributes created richer, more continuous control: hand openness naturally modulated stroke size, while rotation influenced stroke orientation. These mappings encouraged users to explore variation within a single gesture rather than switching between discrete modes.

Kalman filter smoothing plot — Smoothing z-depth noise with Kalman filter avoids jitter in approach/retreat detection with webcam

Different Variations

By combining sensing stabilization with expanded gesture mapping, interaction becomes fluid, layered, and responsive to subtle bodily variation. Together, these iterations shaped the system into a more stable, expressive feedback loop.

Outcomes & Reflection

What kinds of interaction emerge when movement is allowed to remain fluid and exploratory?

When movement is not reduced to discrete commands, interaction becomes gradual and adaptive. Participants explore through pauses, repetitions, and shifts, allowing meaning to emerge through continuous motion rather than explicit control.

How can body and computation co-adapt through continuous feedback rather than instruction?

The system responds to how gestures unfold over time, not just where they occur. In return, the evolving visual response influences subsequent movement, creating a shared rhythm in which body and computation adjust to each other.

Where does the system fall short, and how might it evolve into richer forms of interaction?

The current system is limited by single-user input and camera-based sensing. Currently the system relies on camera-based sensing. This makes the interaction highly dependent on environmental lightning conditions. For example, in projection-based installation, insufficient ambient light led to increased sensing noise and frame instability, resulting in noticeable interaction delay. Future iterations of this project will explore light-invariant and multimodal sensing, to reduce latency and increase adaptability across diverse installation environments. For multi-user interaction, future iterations could expand toward learning-based interpretation, and additional sensory modalities to support more complex and collective forms of engagement.

Future Opportunities

Pathways for further exploration:

From Individual Gesture to Collective Expression
Extend the system for multi-user interaction to turn individual gestures into shared expression, raising questions of coordination, authorship, and social dynamics in responsive environments.

Learning-Based Gesture Interpretation
Introduce learning-based models to adapt to personal movement styles over time, shifting from predefined mappings toward personalized, evolving responses.

Architectural and Multimodal Extensions
Scale the system into architectural or public contexts and integrate sound or haptic feedback to expand the wall into a multimodal interface for perceiving movement across space.