CI environments are performance constrained by nature, and the Android emulator is unfortunately very resource-intensive both in terms of CPU and memory. This slowdown can also surface race conditions and other strange bugs in the application and test code.
Here are some general tips that can help to narrow down the problem:
-
Make sure individual test cases are isolated and don't depend on the state of a previous test case or test run. Cases are executed in random order, so sometimes a test only passes if a given test case was run before, setting up the app state in a certain way (database, SharedPreferences, caches).
-
Run your test suite multiple times and try to notice a pattern in the failing test cases. Is it always the same cases that fail, or the failure happens in a random way among cases?
-
Do you make real network calls in your tests, or you have mocked APIs? Real network calls can take longer (or fail for various reasons) in a CI environment, so these tests will always be flakier than mocked ones.
Tips specifically related to the [BETA] Android Virtual Device Testing step:
-
Set the Use Orchestrator input to true in the Step's Instrumentation Test section. This improves isolation between test cases, making your tests less likely to fail. You can see the benefits and drawbacks of Orchestrator here.
-
You can also set the Number of times a test execution is reattempted input in the Step's Debug section. If this value is higher than 0, then failed tests will be reattempted this many times before marked as failed.