How I Cut Lost Lecture Recordings by 94%

Overview

Teachmint’s interactive flat panels are large classroom boards, almost all on Android 16, including Rockchip RK3576 models. Teachers record their lessons from the whiteboard app. The recording captures the screen and the room’s microphone and uploads when the class ends. Often that file is the lesson’s only record. If it is lost, the lesson can’t be recorded again.

I led the rewrite of that recorder from April 2026. I owned its design, the server-controlled rollout next to the old engine, and most of the field debugging.

The problem

The original engine wrote one MP4 per lesson. An MP4 is only playable once its index box (moov) has been written, and that happens when MediaMuxer.stop() runs at the very end. Anything that ended the process first left a file with no index, and nothing could recover it: a crash, a swipe-kill, or the panel losing power at the end of the period.

The old engine also had a race in its stop path. muxer.stop() could overlap muxer.writeSampleData() on another thread, which produced index-less or corrupt files even when nothing crashed.

The fleet data made the cost concrete. Over the same four weeks of the rollout, next-launch recovery deleted hundreds of unfinished recordings from the old engine, because none of them had an index. The losses were invisible to teachers until they went looking for the lesson.

Constraints

• Long sessions: recordings of two hours and more, as H.264 video and AAC audio.
• A short shutdown window: when a teacher swipes the app away, onTaskRemoved runs on the main thread with roughly a 5-second ANR budget. Whatever must be durable has to fit inside it.
• MediaProjection rules: screen capture requires user consent per projection. On Android 14 and later, a projection may create exactly one virtual display. A second attempt is refused and revokes the projection.
• Vendor codec HALs: the hardware encoder is shared with the rest of the system and not always gracefully.
• No way to ship and pray: a regression here destroys data that can’t be recreated, so the new engine had to run next to the old one and be switchable without a release.

My approach

The new engine records in five-minute segments and treats every one as a complete, playable MP4.

MediaProjection ─► VirtualDisplay ─► H.264 encoder ─┐
                                    (surface input) ├─► codec thread
AudioRecord (mic) ─────────────────► AAC encoder ───┘   (every muxer call)
                                                          │
                 rotate on the first keyframe after 5 min │
                                                          ▼
                                 seg_001.mp4  seg_002.mp4  …
                                                          │ stop
                                                          ▼
              box-level concat → validator → MediaStore → upload queue

The design rests on a few rules:

1. One thread owns the muxer. The video drain, every muxer.* call and the teardown all run on one dedicated codec thread, and stop() is ordered after the drain completes. The race from the old engine can’t happen because there is no second thread to race with.
2. Deterministic end-of-stream. Video ends through signalEndOfInputStream(). Audio ends with an EOS buffer queued by the finalizer itself, not by a producer that might be paused at that moment. The drain stops when both tracks report EOS, and a watchdog exists only for the case where EOS never arrives, such as a revoked projection.
3. Segments bound the loss. If the process dies without warning, at most the segment in progress is lost. Every finished segment is already a valid file waiting in the recovery folder.
4. Validate before promoting. A temp file that didn’t finalize cleanly is never copied to the gallery or deleted. It stays, with a small metadata sidecar, for the next launch to repair.
5. Two engines, one switch. A server-side flag, bucketed per user, chooses the old or the new engine for each session. That let us ramp the new one, compare the two in the same weeks, and fall back without a release.

Important decisions

Segments instead of one file

Segmenting moves the hard part to the end: N files have to become one before upload. The obvious way is to replay every sample through MediaExtractor and MediaMuxer. That took 253–277 seconds for a two-hour benchmark on the panels, about 0.45 ms of fixed overhead per sample.

So I wrote a container-level concatenator. It copies the encoded bytes of each segment unchanged and rewrites only the index tables. In production, a 113-minute lecture (23 segments, 322 MB) stitched in 2.4 seconds. That is a different file from the benchmark, but the old path scales with the number of samples and the new one with the number of bytes. The full story, including three wrong assumptions about real MP4s, is in How I Cut a 4-Minute Video Merge to 2.4 Seconds on Android

A slim shutdown path

Before the rewrite, the swipe-kill path tried to stop the muxer, copy the file into MediaStore and insert the upload row, all inside onTaskRemoved. For hour-long files, the copy alone overran the budget and the recording was lost. I split it. Only the durable step happens on shutdown, which is draining the encoders and writing the index. The gallery copy and the upload row are deferred to recovery on the next launch, and the session’s metadata is persisted in a JSON sidecar next to the temp file. With the slow work gone, the shutdown cap dropped from 8 s to 5 s.

Fail closed, then fall back

Every fast path has a slower, proven path behind it. Box concat rejects any input it doesn’t fully understand, and its output must pass the same validator as every recording. If either check fails, the per-sample muxer path runs instead, and the event records which mechanism ran and how long it took.

What went wrong

The new engine fixed the losses it was designed for. Then the field showed problems nobody had designed for.

Audio that drifted one buffer at a time. Video timestamps came from the surface clock, but audio timestamps came from a sample counter. The counter advanced by a full buffer even when a read came back short. Over a long lesson, audio slid behind video. The fix anchors audio to the shared recording epoch, so a lost capture becomes a short silent gap instead of drift that accumulates. At every seam, the stitch now pads whichever track is short, so both tracks reach the same time at each boundary.

Recordings with no audio at all. Fleet analysis found about 3% of new-engine recordings uploaded with no audio track, on panels of every brand. The stitcher’s fallback path took its track formats from segment 0 only. If the first segment lacked a track, the whole lesson lost it: the sound, or in the reverse case the video. Each track’s format now comes from the first segment that actually has that track.

Video holes exactly as long as the pauses. One teacher’s recording was missing 164 s and 103 s of video, with the audio intact. Those numbers matched the lesson’s two pauses to within 0.1 s. The drain loop read the isPaused flag twice per iteration. If a resume landed between the two reads, the in-flight frame was stamped one pause-length late. The monotonic-timestamp guard then dropped every honest frame after it until real time caught up. The fix was to read the flag once per iteration.

A capture that went black but kept “recording”. The same lesson’s video turned solid black (luma 16) at 73 minutes and stayed black. The encoder kept receiving frames and the teacher kept writing. Nothing in the logs failed. I couldn’t reproduce that exact trigger, but I found a close relative. Starting Android Studio’s device mirroring during a recording froze the capture silently: the picture stopped changing, the projection stayed alive, and no callback fired. I built a detector for it. Every ten seconds it decodes a keyframe and compares the result with a PixelCopy of the app’s own window. If the window is changing and the capture isn’t, the capture is broken. Field tests then settled what recovers it. Re-attaching the surface didn’t, and neither did rebuilding the encoder. A projection owns its one virtual display, so only a new projection with fresh consent recovers. The recorder now keeps the class going as audio-only and asks the teacher to save and start a fresh recording.

A profile photo that killed the recording. On RK3576 panels, a teacher’s HEIC profile photo (usually uploaded from an iPhone) is decoded through the hardware HEVC decoder. Allocating that decoder while the recording’s hardware AVC encoder was live crashed the vendor codec HAL. Both codec components died, and the recording with them, within two seconds. It reproduced 3 out of 3 times, triggered by anything that recomposed the toolbar’s avatar chip. Avatars are now transcoded once to a cached JPEG, and never while a recording is live.

Debugging and investigation

Most of these bugs were invisible in Crashlytics. They produced valid-looking files. Finding them needed a view of each recording from start to upload, not crash counts.

I built that view from the recording events in our analytics warehouse:

• Deduplicate by recording, not by event. One recording that retried its upload six times used to count as six errors. Deduplicating by session cut the error count by about a third, and changed which problems looked biggest.
• Count devices exactly. Masking serials to their last six characters made different panels collide, which undercounted the fleet by about 8.5%. Hashed keys used only for counting fixed it, and the masked suffix is still what the UI shows.
• Per-recording timelines. Every event for a failing recording, in order, made patterns visible that aggregates hide, such as an upload retry loop that ran for two days.
• Download the actual file. For the black-video lesson, the telemetry was clean. Parsing the uploaded MP4’s tables and decoding its frames showed both tracks at the full 113 minutes and every seam aligned to within 25 ms. That cleared the stitcher and pointed at the capture.

The telemetry also showed its own gaps. Every stitch event lacked the session id, so a stitch couldn’t be joined to the recording’s outcome. Some events reported the wrong engine, because they fired in a fresh process. Those were fixed as part of the work.

Result

• In the same four weeks, on fleets of similar size under the server switch, the new engine lost 94% fewer recordings per panel than the old one, and it recovered more than twice as many as it lost.
• Stitching went from minutes to seconds, with a validator in front and the old path behind.
• By early October the new pipeline was recording on thousands of panels.

The honest part: I measured whether the September fixes moved the fleet numbers, and mostly they didn’t. Across three releases, the share of recordings that auto-paused for lack of audio stayed at 10.4–10.9%, and the capture-repair events never fired in that window. The dominant remaining failure is a microphone that produces nothing. It hits about 30% of recordings on one panel model against 9% on the others, and that work continues as mic-inventory logging rather than more recorder changes.

What I learned

1. The fix for data loss is to make every intermediate state valid. Segments, sidecars and validate-before-promote did more than any single bug fix.
2. Single-thread confinement is a design, not an optimization. The worst race in the old engine disappeared because only one thread could make the call.
3. Platform limits decide the recovery story. One virtual display per projection meant no amount of rebuilding could save a frozen capture, and knowing that early would have saved days.
4. Count recordings, not events, and keep the counting keys collision-free. The first version of the dashboard was confidently wrong twice.
5. Measure whether your fixes worked. Several of mine fixed real bugs that weren’t the bugs teachers hit most.