How I Built a Private AI That Writes Up My Workday, Fully Offline

Overview

Most of my workday happens in conversations: standups, design discussions, “can you look at this” at my desk. Most of it is in Hinglish, a mix of Hindi and English, often in the same sentence. By evening I remember the shape of the day, but not who promised what.

Daybook is a macOS menu-bar app I built for myself in October 2026. It listens all day, transcribes what’s said, knows who each voice is, and at night writes a report judged against my own goals for the month: commitments, decisions, meetings, what I shipped, and who I spent time with.

It has one hard rule. No audio or text ever leaves the Mac. Every model runs locally.

The problem

Off-the-shelf meeting tools fail all three requirements at once:

• Language: stock speech-to-text handles Hinglish badly. It either translates or switches script mid-sentence.
• Continuity: a workday isn’t a meeting with a start and end. It’s hundreds of short exchanges, many overlapping, some heard twice.
• Privacy: recording colleagues all day is only acceptable if nothing is uploaded anywhere.

Constraints

• Hardware: one laptop, an Apple Silicon MacBook, with every model on-device. That rules out the cloud APIs that are good at this.
• Battery: the processing modes defer heavy work on battery and catch up on AC power.
• Honesty about quality: I had no human-corrected transcripts, so every model comparison had to say what it could and couldn’t prove.

My approach

Swift menu-bar app                  Python worker (local, MLX / MPS)
  mic capture                         VAD ─► diarization ─► speaker identity
  stage-1 voice gate ─► chunks ─►        ─► Hinglish Whisper ─► SQLite
  UI (reads SQLite)                   episodes ─► extract ─► check ─► records
                                      daily report · recall index (embeddings)

Android companion (optional): records while I'm away from the desk,
uploads over Wi-Fi when I'm back.

• A cheap voice gate in Swift keeps silence from ever reaching disk.
• The worker is the only writer. The app reads SQLite read-only and changes anything through a local HTTP control API, so the UI can’t corrupt the journal.
• A recall tool. The journal is exposed as read-only MCP tools, so I can ask from my editor what was decided about a repository last week. Search is by meaning, using a local multilingual embedding model, so “pricing” finds “kitna charge karenge”.

Important decisions

Pick models with my own audio, not leaderboards

The first bake-off used a synthetic two-voice Hinglish clip:

ASR model	Error rate	Speed
Trelis Hinglish fine-tune (large-v3)	26%	7× real time
Deepro Hinglish turbo	35%	23× real time
Stock large-v3-turbo	57% (all Devanagari)	10× real time

Then I ran eight models on twelve windows of my own recordings. Some failures only showed up on real audio. Two models looped or translated instead of transcribing. One returned nothing on 4–5 of the 12 windows. A small Hinglish model invented text. The Trelis model had the best precision, with Qwen3-ASR-1.7B as the robust alternative.

I recorded what this didn’t prove. The reference transcripts were never corrected by hand, so these are comparisons between models, not true word error rates.

One job per LLM call

The first version sent one big prompt per hour and one for the whole day. Inputs were small, at most 30k characters against a context of more than 200k tokens. Size wasn’t the problem. The problem was that one call was doing a dozen jobs: segmenting topics, classifying them, extracting commitments, deciding who owes whom, deduplicating across hours, matching goals, and writing prose. One day’s report listed a goal twice with contradicting bullets, and filed one commitment under both “you owe” and “owed to you”.

The rebuild splits that up:

transcript ──► episodes (code: gaps, place, speakers, 45-min cap)
                  │
                  ├─► extract  (LLM: records as JSON, validated in code, one retry)
                  ├─► check    (LLM: is each record supported by the transcript?)
                  ▼
           aggregate in code ──► report writers (LLM, structured input only)

Code finds conversation boundaries, an LLM extracts records, and a second LLM call checks every record against the transcript. Unsupported records are dropped, and the checker can correct who a commitment belongs to. Aggregation is code, including who owes whom, which comes from speaker identity rather than the model’s guess. The report writers only ever see checked, structured facts.

Rebuilt over four real days, it ran in 3.8–7.4 minutes per day, with no failed extractions. Each day had 118–215 verified records, and the checker dropped 1–6 as unsupported.

A local model that fits the job

For the daily report, a 4-bit Gemma 4 26B mixture-of-experts model took 4 minutes against 9 for Qwen3-14B on the same day, and produced far richer commitments and meetings. It became the default.

What went wrong

The ASR model stopped after the first sentence. On 28-second windows, the Trelis model silently dropped about 35% of the speech. It transcribed the first sentence of a long input and stopped, and trimming silence didn’t help. Windows of about 8 seconds, plus a per-phrase redo when the words-per-second rate looks too low, recovered 529 of 537 words at about 3× real time.

“Unidentified voices” were a mixed bag. On 1,842 real lines, the speakers the system couldn’t name turned out to be faint or distant speech, not new people. Their voice-consistency scores were 0.23–0.46, against 0.54–0.71 for real people. “Sounds like X” suggestions came from one averaged fingerprint per speaker and were often wrong.

A speaker-model bake-off showed no model clearly beat pyannote’s (equal error rate about 20%). Line length mattered more than the model: 15% on lines of 6 s or more, against 27% on lines of 1.5–3 s. So I moved to per-line voice fingerprints with loudness features, and suggestions are now made by line votes. After a backfill of those 1,842 lines, live data showed 0 unidentified speakers and 0 wrong suggestions.

One room, two microphones, 235 conversations. When the phone and the Mac both heard the same meeting, the pipeline treated them as two places and shredded one day into 235 episodes. Matching the two recordings, by cross-correlating their audio envelopes with Bluetooth proximity as a prior, fuses them into one scene. The clearer microphone wins per speech window. That day became 62 conversations.

The network. Model downloads from Hugging Face hung forever over IPv6 on my network: connections sat in SYN_SENT. Forcing IPv4 fixed it. A new transformers release crashed Whisper on Apple’s GPU backend until I added a workaround. Neither had anything to do with machine learning, and both cost more time than most of the ML.

Result

Daybook runs on my laptop through the workday. It produces a nightly report and supports questions from my editor, like “what did I promise this week?” or “what was discussed about this repo?”, all answered locally.

I built it in six days, 115 commits, with AI pair-programming throughout. That pace is the reason the evaluation discipline mattered: every model choice above has a measurement behind it, and every measurement says what it doesn’t prove.

What I learned

1. Evaluate on your own data early. Synthetic clips ranked the models sensibly, and the real recordings showed failure modes the synthetic ones never could.
2. Input shape can matter more than model choice. Window length decided ASR recall, and line length decided speaker accuracy.
3. Give each LLM call one job, and have a second call check it. Validation in code plus a checker role turned “plausible prose” into records I can trust.
4. Let code do what code is good at. Boundaries, aggregation and direction of commitments are deterministic. The model’s job is reading, not bookkeeping.
5. Privacy is an architecture, not a setting. “Nothing leaves the Mac” decided every model and every dependency.