Type
Master's Graduation Project
Year
2025
Jan - Apr (4 Months)
Individual + 2 Supervisors
Generative AI, Generative UI, Design Futures, Design Fiction, Multi-Agent Systems.
The Context
As the culminating project for my Master of Science in Interactive Arts and Technology at Simon Fraser University, "Genie" addresses a decades-long stagnation in personal computing interfaces. For years, our experience on smartphones and desktops has been defined by a static grid of application icons. This rigid, app-centric model forces users to manually navigate various programs to complete a single task, creating cognitive overload and a fragmented digital experience. The rise of powerful multi-agent AI systems, however, presents a compelling opportunity to fundamentally challenge this and move towards a more fluid, adaptive, and user-centered interaction model.
Objectives
This project's objective was to move beyond theory and demonstrate a tangible alternative to the static, app-based interface. Employing a Design Futures methodology along with Design Fiction, my goal was to bridge the gap between current technological limitations and a preferred future for user interaction. This dual-focused approach led to two primary deliverables:
A Functional Proof-of-Concept: A working, voice-driven prototype that validates the core interaction model by interpreting user intent and dynamically generating a UI using current technology.
A Diegetic Prototype Video: A key Design Fiction artifact that communicates the long-term vision of a seamless, instantaneous, and truly adaptive interface, unhindered by present-day constraints.
Is this the best we can do?
For decades, home-screen interfaces have remained fundamentally unchanged: a passive, app-centric model that forces users to adapt to the system.
For over forty years, the way we interact with our digital devices has remained largely the same. We navigate through a static grid of applications, forcing our needs to conform to a rigid, app-centric model. In fact, our home screens have looked essentially the same since Xerox invented the graphical interface Alto and the System 1 by Apple. Yes, we have come a long way when it comes to what our OS's could do. However, what hasn’t changed all that much is how we interact with it.
This realization hit me during my master's program at Simon Fraser University, leading to Genie, my exploration of what interfaces could become. This project explores an alternative by reimagining the home-screen as a dynamic, responsive environment.
Identifying the problem
This approach works for many good reasons, which is why we've used it for 40 years. But it assumes users know exactly which apps to use, how to use them, and in what order.
The current paradigm led me to believe that there is a "crisis of imagination," a stagnation in design where we've accepted a passive, fragmented user experience. The process of performing a simple task like checking flight times or creating a calendar event requires multiple manual steps and navigation across different apps, creating unnecessary cognitive load.
It’s also inaccessible for people who aren’t tech-savvy, or who have physical or cognitive disabilities as they find these systems rigid and unforgiving.
And even when the user knows what they’re doing, there are so many options. There are multiple apps that all do the same thing slightly differently. You end up spending time downloading and comparing tools to find the right one, only to find out they are locked behind a subscription.
Designing for the future?
So then how do we go beyond incremental improvements and rethink the way we interact with our devices?
So acknowledging this crisis of imagination led me to ask a broader question:
How do we go beyond incremental improvements and rethink the way we interact with our devices?
To tackle that, I needed a methodology that wasn’t just about fixing what’s broken, but one that could help me imagine what should be and what could be. I employed a dual methodological approach, using Design Futures and Design Fiction.
Design Futures merges Design Thinking's problem-solving with Futures Thinking's foresight. The (modified) diagram above by Elliot P. Montgomery shows where these methods lie in this spectrum between the Unconstrained & Imaginary and Constrained & Applied design practices.
I looked at what is making the headlines these days, AI, LLMs, agentic frameworks and tools and designed for probable futures i.e. scenarios that are likely enabled by near-term technological advancements in other words, what could we realistically build now?
However, designing only within today's technical boundaries would limit our understanding of this technology's true potential.
Following Dunne and Raby's approach in Speculative Everything: Design, Fiction and Social Dreaming, I created speculative artifacts as tools for critical inquiry. Their work demonstrates how design can interrogate possible futures rather than simply solve present problems. For this project, I developed a diegetic prototype, what David A. Kirby in Lab Coats in Hollywood describes as a prototype existing within a fictional narrative context.
This speculative vision video explores how intent-driven interfaces might evolve in a more technologically mature ecosystem, similar to how Apple's 1987 "Knowledge Navigator" concept helped audiences imagine computing's future possibilities. By situating the technology within a plausible future scenario, the video becomes a platform for examining not just what this technology could do, but how it might reshape our relationship with digital devices entirely..
Design × Futures
Design Futures is a fairly new industry practice; it's a hybrid speculative design methodology combining elements of Design Thinking and Futures Thinking. Design Futures, as defined by Jod Kaftan (Head of Product Design & Research at Oracle), begins with identifying drivers and signals that shape our technological landscape. Drivers are "tides" or underlying forces in the present with deep roots in historic patterns of change. This can be cultural, economic, technological, or environmental. Signals are "waves" or early indicators: small innovations (a new product service, initiative, policy, data, social convention or technology) happening today that hint at larger transformations ahead.
For this project, I identified two primary drivers reshaping interface design:
Generative AI advancement: Moving from content creation to functional interface generation
Multi-agent systems: Enabling AI to perform complex, collaborative tasks autonomously
My signals came from cutting-edge AI that, while still experimental, pointed to a clear future. Products like Vercel's v0 proved functional UIs could be generated from text, and collaborative AI frameworks like AutoGen showed how complex problems could be solved. Seeing major corporations like Microsoft and Google pushing these technologies confirmed that intent-driven interfaces were not just possible, but inevitable.
Phil Balagtas, Experience Design Director at McKinsey and the founder of the Design Futures Initiative emphasizes that the future should not be seen as a straight line but as a cone of possibilities (many "futures"), encompassing probable, plausible, and possible outcomes. I chose to focus on two:
The Probable Future: What's most likely to happen given current technological constraints and adoption patterns. My functional proof-of-concept operates within today's technical limitations while demonstrating the core value proposition of intent-driven interfaces.
The Preferred Future: The ideal outcome we should strive toward, guided by user values and needs rather than just technical possibility. This vision pushes beyond current limitations to imagine interfaces that truly serve human intentions rather than forcing users to navigate rigid app structures.
What could this mean for user interfaces?
My research identified a convergence of key technological drivers. Academic literature shows that multi-agent systems are shifting development from coding to orchestration (Li et al., 2024), with AI agents already collaborating to generate coherent UI prototypes (Yuan et al., 2024a). At the same time, LLMs are finally overcoming the documented failures of early voice interfaces (Corbett & Weber, 2016; Myers et al., 2018) by enabling flexible, conversational interaction.
This led to my core insight: LLMs provide the reasoning, agents provide the structure, and voice offers the ideal, low-friction channel to state what you need.
I can't be the first one to try, right?
Before I got to start working on my solution, I analyzed some tools and devices that have attempted to change how we interact with interfaces. I chose three interesting different products that target different needs to understand the strengths they offer and the limitations they faced.
Vercel’s v0 can transform plain-text descriptions into working React components, making it a powerful tool for developers. Yet its focus remains front-end prototyping, not complete end-to-end experiences, and it still assumes a developer in the loop.
Natural by Brain Technologies Inc. takes a more radical stance, abandoning the app grid entirely in favor of dynamically generated interfaces from natural language. But its closed architecture and limited extensibility make it feel less like a platform and more like a specialized search engine.
Rabbit R1 positions itself as an AI-powered personal assistant with a “Generative UI” that adapts to the user. It’s impressive, but its scope is bound to a fixed set of predefined tasks, leaving no room for dynamic, extensible orchestration.
Taken together, these tools hint at something larger, a future where the interface itself is ephemeral, summoned only when needed. But none fully bridges the gap between intent and execution.
Creating Genie
Genie is a proof-of-concept that reimagines the home screen as a dynamic, responsive environment. Instead of hunting for the right app, users simply state their needs in natural language, and Genie instantly generates a tailored, context-specific interface. This project was a deep dive into how generative AI and multi-agent systems could orchestrate a fluid user experience that is proactive, intuitive, and efficient.
Squashing early assumptions
The objective was to challenge preconceived notions, refine the project direction, and build a stronger foundation before moving into development.
The project applied my supervisor’s 5 Intelligent Failures exercise, grounded in Research Through Design, to surface and test foundational assumptions. I decomposed the work into five components and deliberately induced small failures using sketches, mockups, and simple code prototypes. Each failure was tested with peers and external observers to gather feedback and iterate quickly.
Early Development
This approach has many good reasons, which is why we've used it for 40 years. It's the least-worst design method. It assumes users know exactly which apps to use, how to use them, and in what order.
Preprototyping
Before writing production code I built two parallel, low-cost artifacts
tldraw sketches that explored layout and interaction states (how a generated UI collapses, expands, or hands control back to the user)
n8n workflow prototypes that encoded voice → action sequences.
The tldraw frames validated layout affordances and interaction hand-offs; the n8n prototypes validated orchestration: intent parsing, worker sequencing, and the structured JSON the frontend would consume. These artifacts reduced risk before investing in the full stack.
Development Phase
Genie is built with:
n8n, implementing LangChain-style workflows for agent coordination
OpenAI, for natural language understanding and task parsing
Supabase, as the backend to store data
React, to generate real UI components in response to user intent
n8n
Multi-Agent AI workflow. n8n implements LangChain JS.
Supabase
A PostgreSQL based database to store widget metadata
OpenAI
Large Language Models used: o3-mini & 4o.
React
React.js framework to generate widgets.
System Architecture
Genie uses a small network of specialized agents coordinated by a Supervisor. The Supervisor parses user input (via LLMs) and orchestrates Worker agents, for example: Weather, Stocks, Calendar, and a Coder agent that produces React widgets. Orchestration is implemented as visual workflows in n8n, which maps well to the agent metaphor: its node-based editor makes decision logic explicit, enables LangChain-style nodes, and integrates with many LLMs and external APIs. The Supervisor uses GPT-4o for intent reasoning, a memory buffer that maintains recent context, and Worker agents executes API calls and return structured JSON that the Coder agent translates into components which are then pushed to GitHub. This then fetched by the frontend and rendered in real-time.
Frontend Architecture
Genie's user experience centers around intuitive voice interactions. The system employs the WebSpeech API18 for accurate speech transcription. The speech recognition component is configured to display interim transcriptions, providing users real-time feedback as they speak. Upon completion of speech input or manual termination, a final transcription is sent to an external webhook service (n8n) for further processing.
The frontend interface of Genie adopts a minimalistic approach, featuring an empty dashboard screen with a central clock to mimic familiar desktop home screens. The interface is rendered in React, using GridStack.js to manage layout, resizing, and drag-and-drop interactions. The renderer interprets the JSON specification coming from the orchestration layer and maps each component definition to a corresponding widget. The Supabase backend is used to store widget positions and configurations between sessions.
Backend and Multi-Agent Architecture
The multi-agent architecture in Genie is built entirely within n8n. The Supervisor workflow receives the parsed user voice input and decides which Workers to activate. For example, a travel-related request might trigger Calendar, Flight Search, and Weather Workers in sequence.
Genie employs specialized worker agents that interact with specific third-party APIs. Due to time constraints during development, only the following agents were fully implemented:
Calendar Agent: Manages Google Calendar interactions, allowing users to create and retrieve calendar events.
Stock Agent: Utilizes the Alpha Vantage API to fetch real-time stock information, providing timely financial insights.
Weather Agent: Fetches current weather information via OpenWeatherMap API.
Following task completion by the Supervisor Agent and worker agents, the Coder Agent which is powered by OpenAI’s o3-mini model, receives structured task outputs and generates the appropriate React widget code. The generated React code adheres to a standardized structure for dynamic content and interactive elements, such as icons from Lucide, and visually appealing styles using TailwindCSS. After code generation, the Coder Agent automatically commits these new widget files directly into the GitHub repository, triggering frontend updates.
Results and Limitations
Design Fiction
Given these practical limitations, it became necessary to shift perspective toward speculation, narrative, and design fiction as a way of imagining what Genie might become once freed from current constraints. Following Dunne and Raby’s (2013) approach to speculative design, the design fiction video serves as a provocation, encouraging reflection on future possibilities in user experience and broader societal implications of adaptive generative UIs. It challenges conventional interaction paradigms and prompts viewers to critically assess what desirable computing futures should look like, ensuring these visions align with user needs.
Ultimately, this approach aims not merely to illustrate technological capabilities but to provoke deeper dialogues about the role of technology in everyday life and the principles that should guide future computing systems.
