How I Cut a 4-Minute Video Merge to 2.4 Seconds on Android

The question

Our classroom panels record lessons in five-minute MP4 segments. When the teacher presses stop, the app stitches the segments into one file and uploads it. On a two-hour lesson, the stitch took over four minutes on the panel. The teacher waited, or walked away, and the upload sat behind it.

The obvious suspect was MediaMuxer. The real cause turned out to be how we were using it.

My initial mental model

The stitch was the textbook Android approach: open each segment with MediaExtractor, read every encoded sample, and write it into one MediaMuxer. No decoding, no re-encoding.

// The shape of the original concat, simplified.
for (segment in segments) {
    extractor.setDataSource(segment.path)
    while (extractor.readSampleData(buffer, 0) >= 0) {
        info.presentationTimeUs = extractor.sampleTime + timeOffsetUs
        info.flags = extractor.sampleFlags
        muxer.writeSampleData(trackFor(extractor.sampleTrackIndex), buffer, info)
        extractor.advance()
    }
    timeOffsetUs += segmentDurationUs + seamGapUs
}

Because nothing is re-encoded, I assumed this was basically a file copy with some bookkeeping. If it was slow, I thought the disk was slow, or the muxer was writing inefficiently.

What actually happens

I measured it on a real two-hour session: 24 segments, 1.03 GB.

Panel	Concat time	Per sample
TeachmintX 75 Pro (Android 16)	252.9 s	0.45 ms
Teachmint_X (Android 16)	277.2 s	0.49 ms

The number that matters is the last column. That session holds 567,487 samples: 258,031 video frames and 309,456 AAC audio frames. Every one of them crosses JNI twice, once out of the extractor and once into the muxer, and goes through MPEG4Writer’s bookkeeping on the way. Copying the same gigabyte as plain bytes on the panel takes about 7–10 seconds, because storage moves roughly 150 MB/s. So about 95% of the stitch was fixed per-sample overhead, not I/O.

The muxer loop itself was clean. The problem was the cost of the per-sample API, and no amount of tuning inside the loop removes that. To get faster, I had to stop handling samples one at a time.

An MP4 is bytes plus an index

A progressive MP4, the kind MediaMuxer writes, is a few top-level boxes:

ftyp   file type
free   padding
mdat   the encoded samples, back to back (≈ all of the file size)
moov   the index: for every track, where each sample is and when it plays

Inside moov, each track has a sample table (stbl) of small arrays:

stts   sample durations (run-length encoded)
stss   which samples are keyframes
stsz   size of every sample
stsc   how samples are grouped into chunks
stco   byte offset of every chunk in the file   (co64 when offsets exceed 4 GB)
stsd   codec configuration (SPS/PPS for H.264, AudioSpecificConfig for AAC)

Two segments from the same recording session share the same codec configuration. Their mdat payloads can therefore be copied back to back unchanged, and the only real work is producing one moov that describes the combined bytes. That work is integer arithmetic on arrays of a few hundred thousand entries, which takes milliseconds.

segment 1: [ftyp][mdat A][moov A]
segment 2: [ftyp][mdat B][moov B]          →   [ftyp][mdat A|B|C…][moov merged]
segment 3: [ftyp][mdat C][moov C]

Let’s look at the implementation

Mp4BoxConcat does this in four phases.

1. Parse. For each segment it scans the top-level boxes and reads every track’s tables into IntArray and LongArray. Even for 567k samples that is a few megabytes of heap.

2. Gate. Before writing a byte, it checks that the segments really are compatible: same track count and handler types, the same timescales, and the same codec configuration. Anything it doesn’t understand ends the attempt: fragmented MP4, more than two tracks, a missing table, or an unexpected edit list.

3. Merge. For each track, in segment order:

stsz: concatenated.
stss: each keyframe index shifted by the number of samples in earlier segments.
stsc: kept per segment, with chunk numbers shifted.
stco: every chunk offset moved by one per-segment delta, which is where that segment’s payload now starts in the output minus where it started in the source.
stts: concatenated, with one twist at each seam (below).

// Output: ftyp, then a fixed 16-byte mdat header, so each
// payload's new position is known before anything is written.
val mdatPayloadOutStart = first.ftypRaw.size + 16L
var running = mdatPayloadOutStart
val deltas = parsed.map { p ->
    val d = running - p.mdatPayloadStart
    running += p.mdatPayloadEnd - p.mdatPayloadStart
    d
}
// later, per chunk:
chunkOffsets[chunkPos + c] = off + deltas[k]

4. Write. It copies ftyp, writes a 64-bit mdat header, streams each payload with FileChannel.transferTo in 16 MB windows (which also drives the progress bar), then appends the generated moov. The output has the same layout MediaMuxer produces.

RandomAccessFile(p.file, "r").channel.use { src ->
    var pos = p.mdatPayloadStart
    while (pos < p.mdatPayloadEnd) {
        val window = min(COPY_WINDOW_BYTES, p.mdatPayloadEnd - pos)
        val n = src.transferTo(pos, window, out)
        if (n <= 0) throw Mp4ParseException("stalled at $pos")
        pos += n
        copied += n
        onProgress?.invoke(copied, totalPayload)
    }
}

Seams are just durations

An MP4 has no per-sample timestamps. A sample’s time is the sum of all the durations before it. To join two segments, there’s no timestamp to rewrite. You set the duration of each segment’s last sample to the gap you want before the next segment starts:

if (k < segs.size - 1 && t.sampleCount > 0) {
    // nominal gap + A/V equalization padding
    val seamDur = gap.toLong() + seamExtras[k]
    durations[samplePos - 1] = seamDur.toInt()
}

Then track duration is just the sum of the merged stts, with no separate seam accounting. The seamExtras term is how the stitch keeps audio and video aligned across seams. That is its own story, about audio that drifted one buffer at a time. It isn’t needed for the speedup.

The surprising part

The plan was approved and the code passed its tests the same day. Then real segments arrived, and each assumption failed in turn.

A 4-byte offset. The movie header (mvhd) stores the total duration at a fixed offset that depends on the box version. In version 0 it sits at byte 24. In version 1, where the timestamps are 64-bit, the earlier fields are wider and it moves to byte 32. My first version wrote it at 28, which looks plausible and is wrong. An 8-byte write at 28 lands on the timescale (bytes 28–31) and half of the real duration field. Nothing crashes, and the box still parses. It just describes a different movie. A single-segment round-trip test now pins it.

          size type ver+flags created modified timescale  duration
mvhd v0:   4    4      4        4       4        4       @ byte 24
mvhd v1:   4    4      4        8       8        4       @ byte 32

“Same codec config” is not “same bytes”. The design said the codec description (stsd) must be byte-for-byte identical across segments. Then real segments were rejected even though they came from one encoder session, because muxers write advisory bitrate metadata (btrt, and the bitrate fields inside esds) that differ per segment. The gate now compares a key built from the parts that decide whether samples are interchangeable. That means the sample-entry fields, avcC, and a normalized esds, with btrt dropped. A real codec change, such as different SPS/PPS or a different AAC config, is still rejected, and a test covers each case.

Edit lists I didn’t know we were writing. Segments recorded through our swipe-kill recovery path came back from the panel rejected with track 0 elst offset 800ms > 100ms. Our segments weren’t supposed to have edit lists at all. Dumping the boxes showed the shape MediaMuxer writes when a track’s first timestamp isn’t zero: an empty edit (a leading gap) followed by the media. The offset was 800 ms in the first segment and grew to 8,454 ms in later ones. The recorder was carrying one timestamp origin across every rotation, so each new segment started later than the one before, with pause time and drift built in.

There were two fixes. The concat now recognizes that exact two-entry empty-edit shape and normalizes it away, while still rejecting every other kind of edit list. The recorder now resets its timestamp origin at each rotation, so new segments start at zero.

Shipping it without trusting it

A stitch that is wrong but fast is worse than one that is slow, because the result is a lesson the teacher can’t re-record. So the fast path never decides on its own:

segments
   ↓
Mp4BoxConcat ──── rejects input ─────┐
   ↓ writes output                   │
RecordingValidator ── fatal error ───┤
   ↓ passes                          ↓
done                     per-sample MediaMuxer path
(box_concat)             (media_muxer)

Box concat fails closed: on any surprise it deletes its partial output and returns false. Its output then has to pass the same validator as every recording. Only then is it accepted, and each stitch reports which path it took and how long it ran. It shipped behind a rollout flag, with the muxer path still in place, and went on for production the next day. I removed the third-party MP4 library I had used for comparison, so nothing new ships in the APK.

I checked correctness at three levels:

Unit: 17 JVM tests. They cover seam bridging, keyframe shifting, offset rebasing, co64 promotion past 4 GB, the edit-list policy, and every rejection rule.
On the panel: the real 24-segment session went through the debug stitch screen, next to the shipping path.
Off the device: I compared packet counts of the output with the sum of the inputs using ffprobe, plus playback spot checks at the seams.

The off-device check also cleared the stitch of something it didn’t do. The output’s audio ended 14.7 seconds before its video. Before blaming the new code, I measured the inputs. The last segment had already arrived 14.7 seconds short, so the stitch had carried the problem through rather than caused it. That gap pointed to an audio problem upstream.

Practical consequences

In production, a 113-minute lecture (23 segments, 322 MB) stitched in 2.4 seconds on the panel. That’s about 134 MB/s, which is the storage speed we measured for a plain copy.

That isn’t the same file as the 253-second benchmark, so I don’t claim an exact multiple. The shape is clear, though. The muxer path scales with the number of samples, and the box path scales with the number of bytes and runs as fast as storage allows. For long recordings, that is the difference between minutes and seconds.

The options I ruled out:

isoparser is unmaintained.
ffmpeg-kit is retired and adds megabytes to the APK.
The NDK would not help. Once the work is a bulk copy, the bottleneck is storage, and transferTo already runs at storage speed.

What I would remember

Measure per-operation cost, not just total time. “0.45 ms per sample” pointed at the API and away from the disk. “Four minutes” pointed nowhere.
Containers are bytes plus an index. When inputs share a codec config, the cheapest correct operation is often to copy the bytes and rewrite the index.
Specs describe the files you expect. Real files are different. Byte-equal codec boxes, an edit-list-free input, and the obvious header offset were each wrong once real segments arrived.
Make the fast path fail closed and get a second opinion. A validator gate and the old path as fallback turned a risky optimization into one we could ship the next day.
Measure the inputs before blaming the new code. The 14.7-second audio gap was already in the source segment.