Six ways Android will eat your background service, and what actually worked

The SMS gateway this article is about does something Android has supported since the beginning: it reads an incoming text message and posts it somewhere. That part took an afternoon.

Everything else took much longer, because the app has to run on a cheap phone sitting on a shop counter, unattended, for months, and it has to not miss a single M-Pesa payment confirmation while doing it. The phone is not a server. It reboots. It runs out of battery. Its manufacturer ships a battery manager whose entire purpose in life is to kill exactly the kind of process this app needs to be.

Every failure mode below actually happened. Each one cost real time, and each one has a fix that is now in the shipped code.

Failure 1: a BroadcastReceiver alone is a lie

The obvious design is to register a receiver for SMS_RECEIVED and do the work inside onReceive. Parse, store, POST, done. It works perfectly in development, on a phone that is plugged in with the app open.

It fails in the field for two separate reasons. The first is that onReceive runs on the main thread with a short execution budget and no permission to start long work; anything you kick off from there is a background process, and background processes are exactly what Doze and OEM battery managers exist to stop. The second is worse and has no workaround at all: after a force-stop, the receiver will not fire. The package is in a stopped state and the system will not deliver broadcasts to it until a human opens the app. That is Android-wide behaviour. There is no manifest flag, no permission, and no clever intent filter that changes it.

So the receiver's job shrank to almost nothing. It is manifest-registered with android:priority="999" and guarded by android:permission="android.permission.BROADCAST_SMS", and all it does is parse, reassemble and hand over:

override fun onReceive(context: Context, intent: Intent) {
    if (intent.action != Telephony.Sms.Intents.SMS_RECEIVED_ACTION) return

    val sms = SmsParser.fromIntent(context, intent)
    if (sms == null) {
        Log.w(TAG, "SMS_RECEIVED but no parsable message")
        return
    }

    val svc = Intent(context, SmsForegroundService::class.java).apply {
        action = SmsForegroundService.ACTION_INCOMING_SMS
        putExtra(SmsForegroundService.EXTRA_SENDER, sms.sender)
        putExtra(SmsForegroundService.EXTRA_BODY, sms.body)
        putExtra(SmsForegroundService.EXTRA_TS, sms.timestampMillis)
        // ...
    }
    ContextCompat.startForegroundService(context, svc)
}

Two details in there earn their place. The parser concatenates every segment of a multipart message before handing it over, because M-Pesa confirmations routinely exceed 160 characters and a half-message is worse than no message. And the sender and PDU timestamp are taken from the first segment, which is what makes the content hash stable across redeliveries.

Note also what the receiver does not do: it does not apply the allowlist. The allowlist configuration lives in the encrypted Dart layer, so the raw body crosses one in-process channel and is dropped on the other side if the sender does not match. Duplicating the filter in Kotlin would mean two copies of a privacy rule that must agree, which is one copy too many.

Failure 2: the process dies, so make the process a citizen

The fix for "background processes get killed" is to stop being a background process. SmsForegroundService runs with foregroundServiceType="dataSync", stopWithTask="false", returns START_STICKY, and posts a low-priority, ongoing, alert-once notification. On Android 14 and later this needs the typed FOREGROUND_SERVICE_DATA_SYNC permission alongside the general one.

The service is where the Flutter background engine lives. It starts a FlutterEngine, runs the backgroundMain entrypoint, and then does the one thing that is easy to forget:

val engine = FlutterEngine(applicationContext)
val entrypoint = DartExecutor.DartEntrypoint(
    loader.findAppBundlePath(),
    BACKGROUND_ENTRYPOINT,
)
engine.dartExecutor.executeDartEntrypoint(entrypoint)

// Make pub plugins (sqflite, secure storage, dio's deps) available
// to the background isolate.
io.flutter.plugins.GeneratedPluginRegistrant.registerWith(engine)

Without that last line the isolate boots and then fails on the first plugin call, which means no database, no secure storage, and no HTTP. The symptom is an isolate that appears to start correctly and then does nothing, which is a miserable thing to debug.

The service also owns the durability contract for everything downstream, because it is the process that must not die. It writes an "I am alive" timestamp every 60 seconds, in both wall-clock and elapsedRealtime form, so the dashboard can tell the difference between "the service stopped" and "the clock jumped".

Failure 3: the warm-up race, and the one window we admit to

Starting a Flutter engine is not instant. It takes on the order of a second. An SMS that arrives inside that window has nowhere to go: the service is up, but the Dart code that knows how to persist it is not.

The fix is an in-memory queue with an explicit boundary. The service holds an ArrayDeque and only drains it when Dart calls back over the method channel to say it is ready:

private fun handleIncoming(intent: Intent) {
    val map = mapOf(/* sender, body, timestampMillis, subId, slot */)
    if (backgroundReady && methodChannel != null) {
        invokeOnMain("onSmsReceived", map)
    } else {
        synchronized(pending) { pending.add(map) }
        Log.i(TAG, "SMS queued (engine warming up)")
    }
}

And on the Dart side, the very last thing backgroundMain does before starting its timers is announce itself:

// Tell the service we are ready so it flushes any SMS queued during warm-up.
await channel.invokeMethod('backgroundReady');

This is the only place in the entire system where a message exists in RAM and not in the encrypted database. I would rather it did not exist, and I am not going to pretend it does not. It is bounded to roughly one second after a cold service start, it only matters if the process dies inside that second, and the inbox backfill in failure 6 recovers the message if it does. But an honest architecture diagram has this window on it.

Failure 4: WAL journalling wedged openDatabase

This one cost a weekend. On some OEM storage, specifically OnePlus running OxygenOS, setting journal_mode = WAL was observed to hang openDatabase outright. Not fail. Hang. Which meant the whole app hung at launch, because the very first thing it does is open its database.

Write-ahead logging is normally the right answer for a multi-reader SQLite workload, and I reached for it without thinking. The fix was to stop reaching: stay on the default rollback journal, which has been entirely reliable here.

There is a second trap in the same three lines, and it is one everybody hits eventually. On Android, a value-returning PRAGMA must go through rawQuery. If you call execute, sqflite maps it to execSQL, which rejects it:

Queries can be performed using query or rawQuery methods only

Which, because it happened inside onConfigure, made every single database open fail. The current configuration is deliberately dull:

onConfigure: (d) async {
  // Value-returning PRAGMAs MUST use rawQuery on Android: execute()
  // (execSQL) rejects them with "Queries can be performed using
  // query or rawQuery methods only", which previously made every open
  // fail and hung the whole app. (journal_mode=WAL had the same problem
  // and is intentionally left at the reliable default.)
  await d.rawQuery('PRAGMA busy_timeout = 5000');
},

Two more hardening measures came out of that incident, both of which are about refusing to hang. The open is time-boxed, 12 seconds for the open and 8 for reading the key out of the Keystore, so a wedged native call surfaces as an error you can see instead of a spinner forever. And if the open fails for any reason, the database files are deleted and recreated exactly once:

try {
  return AppDatabase._(await _open(path, key), path, key);
} catch (e) {
  // The DB may be locked, half-created, or corrupt (a crash mid-write, or a
  // stale WAL file from an older build). Wipe the files and recreate once so
  // the app recovers instead of hanging/erroring on every launch.
  AppLog.w(_tag, 'open failed ($e) - recreating database');
  await _deleteDbFiles(path);
  return AppDatabase._(await _open(path, key), path, key);
}

The deletion covers the bare path plus the -wal, -shm and -journal siblings, because a stale WAL file left behind by an older build is one of the things that can poison the open in the first place. Yes, this discards queued messages in the worst case. That is the correct trade: a gateway that self-heals and loses the tail of a queue beats a gateway that is permanently bricked and needs someone to drive to the shop.

Failure 5: database_closed, or distributed systems inside one phone

This is the best bug in the project, and the symptom was maddening. The gateway would run fine for hours and then silently stop capturing SMS. No crash. Notification still showing. Service still alive. Restart the app and it worked again.

The cause is a single sentence that I now think everyone writing multi-isolate Flutter should have tattooed somewhere: sqflite shares one native database handle per path across every isolate in the process.

The WorkManager backstop was a well-behaved citizen. It ran its sweep, then tidied up after itself in its teardown by closing the database. In doing so it closed the database for the always-on foreground service and for the UI as well. Every subsequent write from the service threw database_closed, and because those throws happened deep inside a background timer, nothing visible ever happened. The gateway just went quiet.

The first part of the fix is a deliberate non-action. The background runner's dispose() does not close the database, and there is a comment in the code explaining why so that nobody helpfully adds it back:

// Deliberately do NOT close the database here. sqflite shares one native
// handle per path across every isolate, so closing it (e.g. from the
// WorkManager backstop's teardown) would close it for the always-on service
// and the UI too - the root cause of the database_closed failures. The OS
// reclaims the handle when the process exits.

The second part accepts that the first part is a convention, and conventions decay. Every repository call goes through a guard that catches the error, reopens once and retries once:

Future<T> _guard<T>(Future<T> Function() op) async {
  try {
    return await op();
  } on DatabaseException catch (e) {
    if (_isClosed(e)) {
      AppLog.w(_tag, 'database_closed - reopening and retrying once');
      await _appDb.reopen();
      return await op();
    }
    rethrow;
  }
}

Three things make that safe rather than a source of new bugs. reopen() is idempotent, returning immediately if the handle is already open. It is serialised through a stored future, so if the sweep and the UI both hit the closed handle in the same millisecond they share one reopen rather than racing to open two. And the retry is exactly once, so a genuinely broken database surfaces as an error instead of spinning.

The general lesson is bigger than sqflite. Anything the platform shares process-wide, and there is more of it than you think, turns your single app into a small distributed system with all the coordination problems that implies. It is worth asking of every handle you hold: who else has this, and what happens when they let go?

Failure 6: the message that simply never arrived

Even with all of the above, there is a residual set of messages the live path never sees. It arrived while the database was briefly closed. It arrived while the service was dead and before the boot receiver brought it back. It arrived during the warm-up second and the process died.

So the app checks its own work. Every 60 seconds the Dart side asks Kotlin to query content://sms/inbox and ingests anything missing. The scan is not the interesting part; the bounds are.

Bound	Value	Why it is that value
Furthest lookback	2 days	Bounded well under the 14-day retention window, so a message that was synced and then purged can never be resurrected and re-sent.
Overlap before the high-water mark	10 minutes	The inbox date column and the PDU timestamp can disagree. Re-scanning a little is free because content dedup catches it.
Messages per scan	200	Keeps a single scan bounded on a phone with a busy inbox.
High-water advance	Only to messages actually scanned	So a capped scan never skips past a message it has not looked at.

The dedup rule is the subtle one, and it is different from the rule the live path uses. The live path deduplicates on message_hash, which is a hash over sender, body and the PDU timestamp. The backfill cannot use that, because the inbox date may differ from the original PDU timestamp, which would produce a different hash for the same message and cheerfully insert a duplicate. So the backfill deduplicates on exact sender plus exact body:

// Already captured by the live path or an earlier scan?
if (await repo.existsByContent(sender, body)) continue;

The lookback bound is worth staring at for a second, because it encodes a rule about time that is easy to get wrong. Retention purges synced messages after 14 days. If the backfill ever looked back further than that, it would find a message in the phone's inbox that the gateway had already delivered and then forgotten about, and it would deliver it again. Two days is comfortably inside the window. The two numbers are related and a change to one is a change to the other.

The four clocks

All of that produces a system with four independent periodic timers, which is worth drawing because the cadences are chosen and not arbitrary.

People are usually surprised that the sweep runs every 15 seconds. The reasoning is that the two most common outage shapes are completely different animals. A backend deploy or a container restart is over in twenty seconds, and during it every in-flight send fails; a slow sweep would leave those messages sitting for minutes for no reason. A genuine outage lasts hours, and there the thing that must not hammer the server is the individual message, which it does not, because each message carries its own growing backoff. Fast sweep, slow per-message retry. They are different clocks solving different problems.

Backoff, and why "full jitter" is not a detail

The retry delay is computed like this:

/// Exponential backoff with full jitter, capped.
Duration _backoff(int attempt) {
  final exp = K.baseBackoff.inMilliseconds * pow(2, attempt - 1);
  final capped = min(exp.toDouble(), K.maxBackoff.inMilliseconds.toDouble());
  final jittered = _rng.nextDouble() * capped; // full jitter
  return Duration(milliseconds: jittered.toInt());
}

Note the third line. The delay is a uniform random value between zero and the capped exponential. It is not the exponential plus or minus a small wobble, which is what most people write when they say they added jitter, and the difference is the whole point.

The schedule is a 15-second base doubling per attempt, capped at one hour, over a maximum of eight attempts. Doing the arithmetic reveals something mildly embarrassing and worth admitting: the eighth attempt's ceiling is 15 seconds times two to the seventh, which is 32 minutes. The one-hour cap is therefore a guard that the current retry count can never reach. It is not wrong, and it costs nothing, but it is a constant that looks load-bearing and is not. If maxRetries ever rises above 9 it becomes real.

Knowing it broke

The last failure mode is not technical. It is a gateway that died at 11am and was discovered at 6pm.

Three mechanisms cover it. BootReceiver listens for BOOT_COMPLETED, the two QUICKBOOT_POWERON variants including HTC's, and crucially MY_PACKAGE_REPLACED, so that updating the app also restarts the gateway rather than leaving it stopped until someone notices. The service publishes its alive timestamp every 60 seconds and the dashboard flags it stale after three minutes. And the five-minute heartbeat means the server knows too, which is the one that matters, because the person who needs to act is rarely holding the phone.

The FastAPI reference backend exposes this as a single endpoint answering the only question anyone asks: GET /api/devices/offline?threshold_minutes=10.

The scoreboard

Failure mode	What actually worked
Receiver never fires after a force-stop	Accept it. Make force-stop unlikely with a foreground service, battery exemption and OEM autostart, and recover with the inbox backfill.
Background process killed by Doze or an OEM manager	Typed dataSync foreground service, ongoing notification, START_STICKY, stopWithTask="false".
SMS arrives during engine warm-up	In-memory queue drained on an explicit backgroundReady callback. Documented as the one memory-only window.
WAL wedges openDatabase on some OEM storage	Default rollback journal, rawQuery for PRAGMAs, time-boxed open, wipe-and-recreate once.
One isolate closes the shared handle	Never close during normal operation, plus a guard that reopens once and retries once.
A message the live path never saw	60-second bounded scan of content://sms/inbox, deduplicated on sender plus body.

What I would do differently

I would make the silent failures loud earlier. The database_closed bug survived as long as it did because a background timer swallowing an exception looks exactly like a quiet afternoon. The counters were on the dashboard the whole time and nobody was looking at them, because there was no reason to look. What was missing was not a metric but an alarm: a heartbeat that carries "my last successful database write was 40 minutes ago" and a backend that says so out loud. The health payload has the fields for it now. The alerting on top is still a thing I check rather than a thing that pages me.

I would also stop treating the WorkManager backstop as a safety net and start treating it as a test. It runs the same code path as the live sweep, roughly every 15 minutes, and if that path is broken it fails silently in exactly the same way the live path does. A backstop that reported "I ran, and here is what I found" would have caught the shared-handle bug within fifteen minutes instead of within a day.

And I would write the OEM autostart walkthrough sooner. Every reliability mechanism in this article is defeated by a phone whose manufacturer has quietly decided the app should not run, and there is no API to ask. That rabbit hole gets its own article, because it is a genuinely different kind of problem: not code, but a dozen undocumented settings screens and a user who has to be walked through them.

If you want the integrity half of this story, how a message stays trustworthy once it leaves the phone, that is the byte-exact contract article.