Why Mobile App Testing Requires a New Approach and How We’re Doing It
Today, as we were delivering our live webcast to the business press, industry analysts, technology bloggers, and enterprise customers, I was watching the mobile testing demo with great interest. After spending over half my career in automated testing (12 years at testing pioneer Mercury Interactive), I couldn’t help notice the similarities with our old 90’s era WinRunner with its nifty record/replay technology, verification points and context-sensitive test scripts.
But then it also struck me that there are some fundamental differences between the testing of the business processes of applications like SAP and Oracle for enterprise (desktop) users, and today’s mobile apps. Sure, the underlying technology today is different, as is the delivery mode (now via the cloud) and the user experience, but there are more substantive ones too:
1. Testing vs. Quality
Testing in those days was never considered a glamorous discipline. QA used to be considered low on the corporate totem pole; the engineer wannabees, the naysayers, the ones who invariably stood in the way of releasing new products and services to market. The business would commonly shove in some more features to the release expecting that users would thank them for giving them so much functionality! “If it doesn’t work, we’ll patch it in the next release” was the mantra.
The CEO certainly didn’t know much about Testing either – if anything, it was just an impediment standing in the way of delivery. So are we really surprised that over 50% of all IT projects failed to be delivered on time, on budget and on requirements?
Today, I guarantee you that every CEO whose company has one or more mobile apps (whether for customers or employees or partners) is keenly aware that the quality of those apps is of paramount importance, and can make the difference between successful adoption and failure. They may still not know much or care about testing, but they certainly do care about Quality. The quality of the user experience, the design, and the function is now the overarching theme that takes precedence over everything else. Better to keep the app brain-dead simple with high utility, than a monster app with low usability. Quality is now an integral part of the delivery lifecycle, and QA (Quality Assurance) has evolved to QE (Quality Engineering), with a direct impact on the success of the business.
2. Release Cycles: Months vs. Days
When enterprises rolled out new versions and modules of their ERP/CRM systems, release cycles were literally measured in many months, even years. It wasn’t uncommon for a U.S. rollout to take 18 months, followed by another 18 months for subsequent regions. Changing timelines from the business to roll out new versions, coupled with engineering feature-creep meant that testers were constantly struggling to figure out what to test and when to test it.
In a previous blog (“Nothing is certain except death, taxes and a short mobile app lifespan”) we discuss how mobile enterprise apps need to be focused on a small feature set targeted around a specific set of needs to be successful and that the best approach is to write apps that users want and need right now. Most likely these apps won’t even live more than a year or two in total, and that’s after going through multiple iterative releases in between. So from a testing perspective, the feature set is much more defined.
Today, with agile development practices release cycles are down to days or even hours. With cloud development and DevOps teams, the line between development and testing has become blurred. Everyone is developing and testing at the same time, and the traditional model of testing purely as a discrete event prior to release is no longer valid.
3. Desktop vs. Mobile Devices
How simple life was when we had a corporate standard for the enterprise desktop; one brand of manufacturer (Dell, HP), one operating system (MS Windows), and one certified browser (Internet Explorer) – all governed and controlled by IT. Worst case was that there were a few supported browser versions, but by and large the number of end user platforms on which the apps would run was very small. This made the testing far more manageable since the number of permutations was so small.
With mobile today, there are literally hundreds of permutations to test for based on device manufacturer, platform, O.S., versions and display size. Factoring in the Android fragmentation (iOS fragmentation is also starting) and pretty quickly the list of testable permutations starts to get messy. On top of that, enterprises require a legitimate solution for testing that doesn’t involve jail-breaking or backdoor tomfoolery.
Requirements for Mobile Application Testing
The automated testing of today’s mobile apps therefore needs to be suited for this new world. The key requirements include:
- Automated capture/replay (without jail-breaking)
- Support for multiple platforms and device types
- Integration with the development environment for rapid and continuous testing
Today
Appcelerator announced a partnership with SOASTA the leader in cloud and mobile testing as well as a joint integration between Titanium and SOASTA’s TouchTest. We’re also now selling this solution under the name
Appcelerator Functional Test as a new part of our platform. This, we believe, will provide our enterprise customers with a strategic advantage in their pursuit of business-transforming mobile apps.
Here’s what George Mehok, CIO of Safeguard Properties, the largest mortgage field services company in the U.S, had to say about this new solution and why they’re adopting it now:
“We’ve adopted mobility as a key enabler to providing productivity enhancing technologies to our nationwide network of contractors and inspectors, in addition to delivering superior quality to our clients. As an innovator in mobility, we’re actively implementing the integrated solution between Appcelerator Titanium and SOASTA TouchTest to further optimize the quality of our apps, reduce development cycle time, and accelerate adoption.”
If you’re interested in learning more about this, watch the recent webinar.
