I tried to push a plan for bootstrapping an automated integration test system for our Android and iOS applications. The plan was based on similar strategies I’d used, or seen used, at other companies. It didn’t fit well with the current situation and workflows. I failed to take those differences into account and the initiative failed.
Developers never saw the value of using their time and workflow on the automated integration tests. Test engineers were overwhelmed by the amount of manual regression tests required for each release. Even with the manual tests, we have gaps in regression coverage leading to some severe defects shipped to users. We are wasting valuable manual testing time on hundreds of manual regression tests that rarely break, when we should be focusing those people on new feature and exploratory testing.
Looking Forward
We want to be able to do automated testing on our iOS and Android client apps. From the simplest type of “does the application start” smoke test to more complicated tests around critical features and functionality. More test automation means:
- Finding bugs faster
- Focusing manual testing on high value tasks (new features and exploratory testing)
- Shipping releases faster with higher quality
Test engineering is highly motivated to do more integration testing as a way to reduce the number of manual regression test cases. Though they don’t have development experience, those folks want to start creating the tests. So we want to keep the barrier to writing tests very low. As we automate regression tests, we want to focus on manual testing on new features, exploratory, and ad-hoc edge case testing.
Objectives for our automated integration testing reboot:
- Doesn’t require knowledge of building applications or the languages used to develop the applications.
- Requires little knowledge of the structure used to build the UI of the applications.
- Reuse integration testing framework, code, and knowledge across all application platforms.
- Reduce the amount of manual integration testing as much as possible.
Approach
We intend to use a black-box approach to installing, launching, and driving the applications. The plan includes:
- Using python-based Appium scripts as the framework for integration tests. Python is a good entry-level programming language and Appium has capabilities to black-box test Android and iOS clients. We’re leveraging the same language and framework for both mobile platforms.
- Using emulators & simulators to run smoke and integration tests. Easy to setup and run locally, while also capable in CI.
- Run the tests several times a day using CI, but not on each PR. The focus is on reducing manual regression testing, while not adding friction to developer workflows.
- Only send consistently failing tests to QA for manual verification and ticket filing.
Milestones
Milestone 1 is about creating a solid foundation for the approach. We’ve completed the proof of concept:
- Test engineers are building scripts using cross-platform tools — and learning to code.
- Developers have added a few testing hooks into the clients to allow faster, more robust tests.
- Python scripts have been created for over 200 integration tests using Appium
- Tests are running in CI several times a day.
- Test engineering created a simple system to send consistent failures to QA.
- Reliability is better than the previous Espresso/XCUITest test suite.
We’ve already saved several tester-hours a day from manual regression testing.
Milestone 2 is really just expanding the test coverage from only high priority test cases to medium and even low priority test cases. We’re also expanding the tooling to support running the Appium tests on both Alpha and Beta channels, as well as self-service support for running on pull requests. Some additional tasks:
- Get better at controlling feature flags for more deterministic test flows
- Start mocking API responses for faster testing and less variations due to live data
- Intercept outgoing requests to track and verify more analytics
- Create a smaller, faster suite for PR testing