Overview
EchoAI Avatar is an interactive conversational AI voice assistant featuring a custom-rendered animated avatar on HTML5 Canvas. It integrates voice input, dynamic LLM responses streamed via Server-Sent Events (SSE), and speech synthesis with visual avatar lip-sync animations.
Problem & Approach
Conversational voice assistants often feel disconnected due to static visual representations and high round-trip latency pauses before voice responses start playing.
Engineered an HTML5 Canvas rendering engine for the avatar's facial expressions, connected to a FastAPI backend streaming LLM tokens via Server-Sent Events with approximately 1 second initial response latency.
System Architecture
- Frontend Canvas & UI
- React, TypeScript, and HTML5 Canvas API with procedural animation ticks.
- Voice Interface
- Browser Web Speech API (SpeechRecognition and SpeechSynthesis).
- Backend Streaming
- FastAPI server generating LLM responses streamed via Server-Sent Events (SSE).
Features & Deliverables
- 01Animated Canvas Avatar EngineImplemented
Custom procedural Canvas rendering animating facial micro-expressions and speech lip sync.
- 02Speech-to-Text & Text-to-SpeechImplemented
Integrated Web Speech API for low-friction voice capture and synthetic audio playback.
- 03Low-Latency SSE StreamingImplemented
Server-Sent Events streaming delivering immediate response tokens with ~1s latency.
- 04Reactive State Micro-InteractionsImplemented
Visual cues indicating listening, thinking, speaking, and idle states.
Technical Challenges
ProblemSynchronizing avatar mouth animation frames with speech synthesis audio playback without audio stutter.
Resolution:Designed a lightweight amplitude estimator tied to the Web Speech utterance boundaries, modulating Canvas mouth geometry smoothly.
Contributions & Takeaways
My Contributions
- Engineered the procedural HTML5 Canvas avatar renderer and animation state machine.
- Implemented the Web Speech API voice capture and playback pipelines.
- Integrated FastAPI SSE stream listener in TypeScript for real-time response rendering.
Key Learnings
- First-token latency matters more for conversational perceived speed than total generation time.
- Procedural 2D canvas animation provides a lightweight alternative to heavy 3D avatar meshes for web browsers.
Interested in discussing this project or working together?
Get in touch