SHUBHAM SINGH BISHT
← Back to writing

What I Learned Building a Floating AI Assistant on Android

Four bugs from an assistant that draws over other apps: a check that runs a frame late, a keyboard that wouldn't open, windows test tools can't see, and code that ran before the device unlocked.

By 6 min read

  • Android
  • WindowManager
  • System apps
  • Testing

The question

Our AI assistant for classroom panels isn’t an app you open. It’s a floating button, a radial menu and a chat window, drawn with TYPE_APPLICATION_OVERLAY windows by a Service, on top of whatever the teacher is using. Everything an Activity gets for free, such as focus, the keyboard, lifecycle and test tools, an overlay has to arrange for itself.

These are the rules I wish I’d had on day one. Each one came from a bug.

My initial mental model

An overlay is a view you add with WindowManager.addView() and remove with removeView(). Make it FLAG_NOT_FOCUSABLE so it doesn’t steal input from the app underneath, and it behaves like any other view, just higher up.

Rule 1: isAttachedToWindow is a frame late

Split screen sometimes showed two floating buttons, one per pane, where there should have been one. The extra one never went away.

Our “safe” window helpers checked isAttachedToWindow before removing a view, to avoid removing something that was never added. But that flag only becomes true one frame after addView() returns. When an add and a remove landed in the same UI frame, as they could during a split-screen change, the remove saw “not attached” and skipped. The window stayed on screen, orphaned, with nothing holding a reference to it.

WindowManagerGlobal tracks a view from the moment addView() is called. So the gate was removed: removing or updating a view whose add is still pending is valid, and a view that was never added (or was already removed) throws, which the helper reports instead of hiding.

Don’t use isAttachedToWindow to decide whether WindowManager owns a view. Track your own adds and removes.

Rule 2: a keyboard in a window that can’t take focus

The overlay must be FLAG_NOT_FOCUSABLE, or it steals focus and the keyboard from the app underneath. But the chat window has a text field, and a non-focusable window can’t show the keyboard.

The answer is to change focusability per gesture:

ACTION_DOWN inside the window
  ──► clear FLAG_NOT_FOCUSABLE, update the window layout
  ──► a beat later: show the IME, if a text field took focus

ACTION_OUTSIDE (watched)
  ──► set FLAG_NOT_FOCUSABLE again, hide the IME

FLAG_WATCH_OUTSIDE_TOUCH delivers ACTION_OUTSIDE when the teacher touches the app underneath, so the overlay can give focus back.

Two details made it work in practice:

  • Wait for the window manager. Clearing the flag is a layout update that the window manager applies asynchronously. If the keyboard is requested immediately, Compose can ask for it before the new flag takes effect, and nothing happens. The explicit show is posted a moment later.
  • Only show it for text. ACTION_DOWN fires for taps on buttons too. An unconditional show flashed the keyboard open and shut whenever a teacher tapped an option. The show is gated on onCheckIsTextEditor(), which is true only while a text field has an editing session. The delay is long enough that a tap on the field has taken focus by the time we look.

The posted show also re-checks that input mode is still on and the view is still attached, because a hide can land in between.

Rule 3: test tools can’t see your overlay

QA’s Appium tests couldn’t find the floating button or the radial menu, even though both had proper view ids. UIAutomator2’s default hierarchy dump skips TYPE_APPLICATION_OVERLAY windows that are FLAG_NOT_FOCUSABLE, so they simply weren’t in the tree.

It took two changes, and neither works alone:

  • App side: give each overlay root an accessibilityPaneTitle and importantForAccessibility = YES, so the accessibility service registers it as a navigable pane.
  • Test side: enable multi-window in the Appium session (appium:settings[enableMultiWindows] = true).

Rule 4: nothing about boot is free

An overlay assistant wants to be there from the moment the panel turns on. Each way I tried to get there had a cost.

directBootAware started the whole app before unlock. Marking the accessibility service direct-boot-aware made Android create the Application before the user unlocked the device, and onCreate read SharedPreferences from credential-encrypted storage. That storage doesn’t exist before unlock, and the read threw IllegalStateException. The flag was removed, with a comment in the manifest saying not to add it back until the Application and preferences are direct-boot-safe.

Sticky services get intents after an early return. The overlay service’s onCreate has early exits for missing overlay permission, a refused foreground start, or the feature being turned off. Android still redelivered sticky intents to onStartCommand, which then touched views that were never created. Now onStartCommand checks, stops the service, and returns START_NOT_STICKY.

BOOT_COMPLETED kept moving. Installing a pending update at boot changed shape several times between April and July:

  1. a boot receiver that started the overlay service
  2. a small dedicated boot service, because the overlay service needs overlay permission and a boot task shouldn’t
  3. the check moved to app launch, and the receiver went away
  4. in July, a boot receiver that had come back was removed again

Each step did less at boot.

A refused foreground start still has to go foreground. If the service is told to start in the foreground and then decides to stop, for example because the overlay permission is missing, it must still call startForeground() before stopSelf(). Otherwise Android kills it with ForegroundServiceDidNotStartInTime.

Practical consequences

For anything built on overlay windows:

  1. Track your own add and remove state. isAttachedToWindow lags.
  2. Toggle FLAG_NOT_FOCUSABLE per gesture. Watch outside touches to give focus back. Show the IME only for a real text session, and only after the flag has applied.
  3. Make overlays visible to accessibility and test tools, with pane titles and multi-window support.
  4. Assume boot is hostile. Avoid direct-boot awareness unless every startup path is safe before unlock, guard onStartCommand against a half-built service, and keep boot-time work to the minimum.

What I would remember

  1. isAttachedToWindow becomes true a frame after addView().
  2. A non-focusable overlay can host a text field if it changes focusability per gesture.
  3. UIAutomator ignores non-focusable overlays unless you mark them as panes.
  4. directBootAware on one component starts your whole Application before unlock.
  5. Every early return in onCreate needs a matching guard in onStartCommand.

Further reading