SHUBHAM SINGH BISHT
← Back to writing

How I Built an AI That Teaches a Lesson on a Whiteboard in 6 Days

A hackathon build where a language model plans a lesson and the whiteboard performs it, writing, pointing and narrating, with four layers of checks so the model can't break the app.

By 6 min read

  • Android
  • LLM
  • On-device AI
  • Architecture

The question

Our whiteboard app already had the tools a teacher uses: streaming text, tables, flowcharts, images, a laser pointer, highlights and page turns. In February 2026 I wanted to know whether a model could use them. Give it a topic, and the board teaches the lesson: writing, pointing, pausing and narrating, like a teacher at the front of the room.

My initial mental model

The tempting version is to have the model generate the whole thing: text, positions and timing, in one pass, played back as it streams.

The problem is that a model is good at deciding what to teach and bad at deciding how many milliseconds a laser should stay on a word, or where a table fits on a 4K page. Asking it for both puts the parts it’s good at and the parts it’s bad at into the same output, with nothing between the model and the screen.

What actually happens

The design I sketched on day one borrowed from MCP, the tool-calling pattern for language models. Expose each whiteboard tool as a typed action, have the model emit a script of those actions, and let the app perform it.

topic ──► LLM ──► TeacherScript (JSON)
                    SPEAK, DRAW_STREAMING_TEXT, DRAW_TABLE,
                    DRAW_FC_ELEMENT, IMAGE, LASER,
                    HighlightText, DELAY, PAGE
                          │
                          ▼
          validate ──► heal ──► orchestrate ──► whiteboard
              │
              └── rejects what can't be fixed

The script is the contract. The model never touches the canvas. It describes a lesson, and every action it can name is one the app already knows how to draw.

The prompt teaches pedagogy, not pixels

The system prompt is mostly about teaching. Each section follows a rhythm of hook, explain, reinforce. Each action carries a teaching intent, and the intent decides the visual method:

Intent Used for How it’s shown
PROCEDURE derivations, algorithms streaming text, no highlight until the end
CONCEPT definitions, comparisons static text, laser while narrating
MEMORY formulas, exam facts static text, permanent highlight, slow pace
RECAP reinforcement quick laser sweeps over existing marks

It also sets budgets: four to six text elements per section, at most one table or flowchart, never both. And it sets pacing rules borrowed from cognitive-load research:

  • 2-second perception rule: pause after every major reveal.
  • 7-second focus window: add motion or a question before eight seconds of static content.
  • Modality balance: never introduce new visuals, new speech and motion at once.

Layout is left out of it. The model uses semantic roles (HEADING, BODY, HIGHLIGHT) instead of colours and sizes, and omits coordinates so the engine stacks elements itself.

Let’s look at the implementation

Four layers between the model and the screen

Model output is a draft. Before anything is drawn, the script goes through a validation pipeline:

  1. Structural: malformed JSON, empty actions, unknown action types.
  2. Content integrity: sentences that stop mid-word, unbalanced brackets, text too long for its role. Narration should end like a sentence.
  3. Synchronisation: does the narration finish when the streaming text does? Is there a pause after each page turn? Does speech overlap the next action when it shouldn’t?
  4. Healing: repair what’s safe to repair. Fix punctuation, stretch streaming durations to match their narration, insert missing delays after page turns, and clamp delays into range.

Anything structural that can’t be healed is dropped, and the run keeps both the raw and the healed script so failures can be compared afterwards. Together the four layers are about 1,000 lines of code, and it’s the most important code in the project.

Intent in, timing out

An ActionOrchestrator turns the model’s intent into concrete actions. A laser with an emphasis level becomes an effect, a duration and a jitter. A streaming text gets a duration from its word count, slower for MEMORY and PROCEDURE, faster for RECAP:

// Simplified from TimingPolicy.
private const val BASE_MS_PER_WORD = 150L

fun getStreamingDuration(text: String, intent: TeachingIntent?): Long {
    val words = text.split(Regex("\\s+")).count { it.isNotBlank() }
    return (words * BASE_MS_PER_WORD * intent.paceFactor()).toLong()
}

Every timing constant lives in one TimingPolicy, not in the prompt and not in the playback code. If a lesson feels rushed, there’s one file to change, and the model doesn’t need to be re-prompted.

The surprising part: the model was the smallest piece

The prototype runs on a cloud model or on the panel, with the same script contract either way. On-device was the part I expected to be hardest. It turned out to be one decision.

The first on-device model was Gemma 3n E2B, multimodal and 3.4 GB. On the RK3576 panel’s GPU it took about 38 seconds to initialise and about 2 GB of RAM. Switching to Gemma 3 1B, text-only at 530 MB, brought init to about 5 seconds and RAM to about 500 MB. The download got 85% smaller. Because the model only writes a script, losing multimodality cost almost nothing. Rockchip’s NPU wasn’t an option, because the on-device runtime didn’t support it.

The parts that took the time were the deterministic ones: the validators, the timing policy, and a Bluetooth remote so a teacher can pause, skip and point from their phone.

The remote had its own lesson. The first version put the panel’s Bluetooth MAC address in the pairing QR code. Android doesn’t give apps the device’s real Bluetooth address, and BLE addresses can be randomised. The same day, pairing switched to scanning for the app’s GATT service UUID plus a board name in the QR code. That’s how BLE pairing should have worked from the start.

Practical consequences

It was a hackathon build: six days, about 18,800 lines of Kotlin across 100 files. It was built for a demo, and it never shipped to production.

I built it with AI coding tools in six parallel feature branches, one each for the Bluetooth remote, local AI, the 1B model swap, UI, tables and animation. They merged into one branch at the end. That’s only possible when the pieces meet at a contract, and here the contract was the script format.

What would need to happen before production:

  • Evaluation: the validators measure whether a script is well-formed, not whether the lesson is good. That needs teachers in the loop.
  • Tuning the pacing: the timing rules come from cognitive-load research as defaults. Nobody has tested them with students.
  • Bounding the content: the model can only use our tools, but it can still say something wrong. A lesson needs the same grounding as any AI answer for a classroom.

What I would remember

  1. Make the model write a plan, not pixels. A typed script is reviewable, testable and replayable.
  2. Put deterministic rules in code. Timing, layout and budgets belong in a policy file the model can’t argue with.
  3. Validate, heal and keep the raw output. Most bad output is fixable, and the rest should be visible.
  4. A contract lets a small model do the job. Gemma 3 1B was enough once the model’s job was only the script.
  5. Never trust a hardware identifier that Android may hide or randomise.

Further reading