Tangible quests
ROLE
Lead Interaction Designer
YEAR
2024
TOOLS
Unity, Teachable Machine, Wekinator,
Processing, P5.js, Figma, Bezi,
Adobe Creative Suite, Ideogram.ai
PROJECT TYPE
Group project
context
Tangible Quests was a classroom project at OCAD University exploring how augmented reality and machine learning could expand traditional play. Inspired by Keiichi Matsuda’s “Hyper-Reality” and Tilt Five’s AR board games, we set out to experiment with hybrid objects that blend the tactile qualities of board games with immersive digital overlays.
exploration
Our opportunity was to reimagine the cultural and tactility of the board game Ludo in an AR-driven future. Rather than solving a business problem, the challenge here was speculative: How might we preserve the tangibility of physical objects while layering digital capabilities that enrich play and social interaction?
contribution
Led ideation and development of all the prototypes. Prototyped four pipelines exploring communication between machine learning models (Teachable Machine, Wekinator) and Unity. Designed and implemented gameplay logic for token movement in unity, dice recognition, and AR overlays. Collaborated to build a proof-of-concept video using Bezi, Figma, and Adobe Creative Suite to visualize multiplayer setup, scoring UI, and immersive animations.
This project helped me understand how speculative design can explore new ways of experiencing social play
Overview
Tangible Quests is a speculative design project that asks how emerging technologies might reshape the way we play together. The focusof the project was on exploring possibilities rather than solving a traditional problem. Using the classic Indian board game Ludo as a foundation, we created hybrid prototypes that blended physical tokens with augmented reality overlays. Through these experiments, we tested how tactility and immersion could co-exist in future play.
The proof-of-concept video demonstrates a multiplayer AR scenario, where players interact through digital enhancements layered over the board. Early gameplay dynamics such as token movement, scoring, and visual feedback were prototyped to imagine new ways physical and digital play might merge.
Future Scenario
Exploring how physical and digital realities can merge.
Idea Generation
Exploring with how augmented features could enhance real world objects
Prototyping
Iteratively built pipelines connecting various platforms
Key Learnings
Highlighted challenges and opportunities for designing hybrid experiences
Process
We followed the Double Diamond framework of speculative design from the Fountain Institute, moving from exploration to provocation and then into prototyping future scenarios.
Future Scenario
Keiichi Matsuda’s Hyper-Reality presents a provocative and kaleidoscopic vision of the future where physical and virtual realities fully merge. Hyper-Reality is six-minute video exploring concept for a future where augmented reality has been carried to an extreme, with interactive virtual interfaces saturating the urban environment.
"Our physical and virtual realities are becoming increasingly intertwined. It will be the glue between every interaction and experience, offering amazing possibilities, while also controlling the way we understand the world"
— Keiichi Matsuda, Hyper-Reality (hyper-reality.co)
Source: http://hyper-reality.co/
This became our design provocation. It pushed us to ask:
What if reality itself became the playground?
What if the tactile and the digital were not separate, but seamlessly merged?
How might we design enhanced interactive environments without losing the tangiblility of objects?
Idea Generation
We explored a range of future scenarios using Ideogram.ai to visualize how digital interactions might seamlessly merge with tactile objects. These speculative renderings allowed us to imagine playful environments serving as both inspiration and provocation for our prototypes.
From these explorations, we chose to reimagine the classic board game Ludo, inspired by our childhood memories of playing it. With its simple components a board, dice, and player tokens. Ludo offered the perfect foundation for testing how augmented features and interactive digital elements could enhance the inetraction of a traditional game without losing all its tactile essence.
Prototype 1
Prototype 1 Pipeline
This prototype established the core path logic for the Ludo game using Unity, combined with Google’s Teachable Machine to recognize dice values through a camera. In Unity, we created two scripts: one to define the board path and another to move the token along it. Initially, the token movement was triggered through numpad inputs, while Teachable Machine was used to detect the rolled dice and output the corresponding number to advance gameplay
Challenge:
The main challenge was integrating Teachable Machine, which runs on JavaScript, with Unity, which uses C#. This required building a custom pipeline to enable reliable communication and data exchange between the two.
Positive Outcome:
The prototype successfully established the board path, with tokens moving the exact number of steps based on keyboard input. Teachable Machine was also successfully trained to recognize the dice rolls from the camera feed, establishing the foundation for gameplay.
Prototype 2
Prototype 2 Pipeline
After the initial prototype, we shifted to using Wekinator, which allowed machine learning models to interface more effectively with Unity. The integration was managed through Processing and the UniOSC plugin, which successfully sent data into Unity. For this test, we used multi-colored dice, which provided better training data for Wekinator’s model.
Challenge:
Wekinator was not well-equipped to train on traditional dotted dice, limiting its ability to handle more nuanced visual patterns.
Positive Outcome:
The projected board worked successfully, and the tokens moved based on both the color and the number the system was trained to recognize — marking a more robust interaction between physical dice and digital gameplay.
Prototype 3
Prototype 3 Pipeline
For this iteration, we transitioned from Wekinator back to Teachable Machine combined with p5.js, which offered a simpler way to train and integrate dice recognition models. The system used a webcam feed to detect the number on a physical dice and sent this value to Unity through a WebSocket server. This allowed us to test a more direct pipeline between machine learning, browser-based scripting, and the Unity environment.
Challenge:
The model struggled with reliably detecting standard black-and-white dice, leading to inconsistent recognition results
Positive Outcome:
Despite recognition issues, the prototype successfully established an end-to-end communication pipeline: physical dice → Teachable Machine → p5.js → WebSocket → Unity. This proved the feasibility of connecting browser-based machine learning directly to Unity for real-time gameplay.
Learnings
Bridging multiple platforms: Working across Teachable Machine, p5.js, WebSocket, and Unity was both eye-opening and demanding. Coming from a non-coding background, I was able to learn and adapt by experimenting with different integration methods, demonstrating the value of multidisciplinary design practices.
Collaboration and teamwork: Overcoming integration challenges required a mix of design thinking, coding expertise, and peer support showing how interdisciplinary collaboration can make technically complex prototypes achievable.
Exploring augmented reality through HMI: This prototype offered insight into how different forms of interaction within human–machine interfaces can enhance immersive and cognitive engagement.
Acknowledgement
I would like to thank my group members Aishwariya BhattBhatt, Kunal Devi, Samarth Reddy and Rutooja Khamkar for their contributions and collaboration throughout the project. Special thanks to our instructors Alexis Morris and FIdelia for their guidance and feedback, which helped shape the direction of each prototype. I am also grateful to Juan Sulca for his coding support, all of which was instrumental in getting the system to run smoothly.