Emulators, simulators and real devices: building a mobile automation strategy that won't fail in production

There's a scene that repeats in almost every mobile project: the regression suite goes green in the pipeline, the release ships to production, and two hours later the reports start coming in. Crashes on Samsung with Android 13. The camera that won't open. Biometric login returning a generic error.

The tests weren't wrong. They were running in the wrong place.

This is one of the technical decisions with the biggest impact on an app's quality and yet it's usually made by default: where you run your mobile automation.

First, a distinction that matters

They're used interchangeably, but they aren't the same thing.

An emulator (Android) emulates the full hardware: CPU, GPU, sensors, radio layer. It runs a real OS image on a virtual machine. That's why it's slow, and that's why it's more faithful.

A simulator (iOS) doesn't emulate hardware. It runs your app compiled for your machine's architecture inside an environment that mimics iOS behavior. It's fast, lightweight and considerably less faithful: the binary you test isn't exactly the binary that goes to the App Store.

A real device is a phone. With its processor, its battery, its manufacturer-modified Android build, its antenna and its impatient user.

The practical consequence: a test that passes on the iOS simulator tells you far less than you think.

What an emulator catches well

They aren't a necessary evil. Used well, they handle most of the testing volume:

• Business logic and functional flows. If the cart totals correctly, the emulator tells you just as well as an iPhone would.
• Navigation and UI states. Screens, transitions, form validation, error messages.
• OS version coverage. Spinning up Android 11, 13, 14 and 15 images in parallel costs minutes, not money.
• Resolutions and densities. Testing layouts on small screens and tablets without buying hardware.
• Regression on every commit. This is where they shine: they're disposable, reproducible and always start from a clean state.

That last point matters most. An emulator can be thrown away and spun up again. A real device accumulates state, runs out of battery, drops the wifi and gets taken to a demo by someone.

What you'll never catch on an emulator

And here's the list that explains the production bugs:

• Manufacturer fragmentation. Android isn't Android. It's Samsung's One UI, Xiaomi's HyperOS, Oppo's ColorOS. Each layer touches permissions, background memory management, notifications and keyboard behavior. The emulator runs pure Android (AOSP), which almost nobody is holding.
• Real performance. The emulator uses your machine's or your CI runner's CPU and RAM. A low-end device with 3 GB of RAM and the system killing background processes behaves differently. Load times, scroll jank and ANRs only show up there.
• Hardware and sensors. Camera, NFC, GPS with real signal, accelerometer, fingerprint, Face ID, QR scanning. They can be mocked, and the mock validates your code, not the integration.
• Network conditions. Intermittent 3G, switching from wifi to mobile data mid-transaction, real cellular latency. An emulator has your office fiber.
• Battery, temperature and throttling. No banking app fails in the lab; it fails on the customer's phone at 8% battery.
• Third-party app integrations. Payment tokenization, deep links into banking apps, system biometrics, Google Pay. Flows that leave your app and come back almost never behave the same in an emulated environment.

The hybrid strategy: not “one or the other”, but “which one at each layer”

The right answer isn't to choose. It's to distribute execution according to the point in the cycle and the cost of each run.

The logic is economic and risk-based. The emulator gives you cheap volume and fast feedback. The real device gives you certainty where being wrong costs money: payment, login, onboarding, account opening.

A simple criterion for the real-device subset: use your own app's analytics. In Latin America, covering 70% of the installed base usually takes between 6 and 10 model and OS version combinations. You don't need a catalog of 200 devices; you need the right one.

How this translates into the framework

Having the same suite run on a local emulator and on a real device in a farm isn't magic: it's design.

• Parameterized capabilities, not hardcoded. deviceName, platformVersion and udid come from configuration or environment variables. One test, five execution targets.
• Abstraction of the hardware layer. If your test needs to scan a QR code, the action is defined in an interface. One implementation injects the image into the emulator; the other uses the real device's camera. The test case doesn't change.
• Explicit waits, always. This is the number one source of flakiness when moving tests between environments. An emulator on a loaded runner can be three times slower than a physical phone. If your test depends on a sleep(2), it will break. Wait on UI state conditions, never on the clock.
• Controlled state from the start. Language, timezone, permissions, screen density and session data set by configuration before every run. Tests that assume the environment's state aren't tests, they're luck.
• Animations disabled on CI emulators. It cuts execution time and removes an entire category of intermittent failures.

The mistakes we see most often

1. Automating everything on emulators and calling it coverage. It gives false comfort: the report is green and the bugs keep coming.
2. Automating everything on real devices. Expensive, slow, with queues at the farm and feedback that arrives when nobody is looking anymore.
3. Measuring performance on an emulator. The numbers mean nothing. Not even as a baseline.
4. Not versioning the environment configuration. If every machine spins up its own Android image, results aren't comparable.
5. Starting with the UI. Mobile automation is the most expensive and most fragile layer of the pyramid. If the API layer isn't automated, the UI ends up papering over holes that aren't its job.

In summary

The emulator doesn't replace the real device and the real device doesn't scale. The question isn't which to choose, but what runs at each layer, how often and to answer which question.

A healthy mobile strategy looks like this: volume on emulators, certainty on real devices, and a single maintainable suite that runs on both without touching a line of code.

At Qualis Lab we automate mobile regression with Appium and in-house frameworks, running on emulators and on physical devices through BrowserStack and Sauce Labs. If you want to know where the weak point of your current strategy is, book a 20-minute assessment.