Wayne makes a key point about testing: “it’s not just about finding bugs in the software, but to really making it better.”
Wayne presents testing as a series of “zero sum games.” Budgets and deadlines mean you can’t test everything, so it’s important to decide the kinds of testing that are going to be the most effective. In other words, testing itself needs to be optimised. It should get the most ‘bang for your buck’ and help to actually fix bugs, rather than just collect them.
Top 4 Recommendations for Testing
Since not everything can be tested or fixed, Wayne introduces a few scientific approaches to help pare down testing to make it more manageable. Wayne’s recommendations for testing include the following:
1. Have a Plan: When you decide to test, it’s important to have a plan. Organize your limited resources, and schedule your testing to be effective as possible. It’s a good idea the spend time identifying ‘high-yielding’ tests and prioritizing ‘high-yield’ areas of code or features before setting up testing environments and thinking about test coverage.
2. Variety is Essential: Don’t always use the same data. Use different types of tests to compensate for each test’s weaknesses.
3. Knowledge is Power: Learn from other testers’ experience and pay attention to what is happening in the development practices, then ask questions, read the bug reports, and see what is breaking for others, so you don’t repeat the same mistakes.
4. Be Results-Oriented: Create priorities about what needs to be tested and work on the goal of choosing quality over quantity when testing. Avoid default test data, target critical bugs in main features, reduce and eliminate redundant tests, and generate ONLY quality bug cases.
Continuous Integration is just what it sounds like: a development process in which team members’ work is integrated continuously. Integration happens multiple times per day, and is tested with an automated build to find errors whenever they crop up. Wayne believes that continuous integration is a key aspect of modern agile development practices, since constant iteration requires constant testing to make sure each version holds together.
Wayne uses the analogy of Lego blocks: When building, you want to make sure that the individual blocks and the way they are connected work, and not wait until later to see if your construction falls apart. Under older integration processes, developers might spend as much time trying to integrate the unwieldy pieces of their creations as building them. Continuous integration is about avoiding that problem by identifying and squashing bugs as soon as they occur. By reducing the risk of unpleasant surprises later in development, it also allows you to get a good idea of the quality of your code and what kind of progress you are making.
Wayne sounds a note of caution: Automated testing will not identify all bugs, log issues, or help to debug software. Think of it as a part of a developer’s repertoire, rather than a replacement.
The Pitfalls of testing
Wayne outlines three pitfalls of testing: Human Bias, Group Think, and Inference and Assumptions.
Human Bias: We all have biases due to our backgrounds, and these may influence our decisions. While transcending bias is easier said than done, we all benefit from becoming aware of our bias and how it might affect our testing and objectivity. If you are aware of your bias, you might be able to make choices you otherwise wouldn’t have.
Group Think: Teams have their own biases as well. If everyone is just agreeing with each other, we might be creating ‘yes teams,’ instead of creating teams that test assumptions.
Inference and Assumptions: There is an additional pitfall: Obvious patterns can fool us into jumping to erroneous conclusions. Indicators are not results: They just indicate those areas where we should test for results. For any testing, past experience should be only used as a guide and not as a conclusion.
UPDATE since Wayne’s presentation at MEIC!
Since Wayne’s presentation on testing at MEIC, Steve Krugh has been recently acknowledged Wayne in a new chapter on mobile in his 3rd edition of “Don’t Make Me Think,” a very popular and must have book on UX. Steve Krug is one the leading authorities in research in human-computer interaction and web usability. Wayne highly recommends Steve’s book for everyone who builds mobile apps to buy this book.
Steve’s website: http://www.sensible.com/
Video 1: Introduction to testing
Video 2: Mathematics of Testing
Video 3: Continuous Integration
Video 4: Pitfalls of Testing
Workshop 4: Testing
Thursday, February 28, 2013
205 Richmond St. W., Room 7311
11:30 a.m. – 3:30 p.m.
Testing is vital for both the creative and technical aspects of a mobile project. This is how you make sure your product is hitting audiences in the right way.
– Recommendations for testing
– Possible outcomes
– How to make the most of user testing
11:30 a.m. Networking Lunch
12:00 p.m. Welcome & Introductions with Wavefront
Anthony Thomas, Business Development, Wavefront
12:30 Testing Your Way to Success!
Wayne Pau, SAP
– The Art & Science of Testing
– Common Misconceptions of Testing
– CI (Continuous Integration) & Automated Testing
– Testing Tours & Exploratory Testing
– Tips & Tricks
Wayne Pau is a well rounded, multi-disciplined member of the newly formed Emerging Technologies group with SAP. Educated as a Systems Design Engineer from the University of Waterloo, he has become a relative “veteran” of mobile devices since the first set of handhelds over 15 years ago.
Wayne has a strong commitment to building the next generation of quality systems and applications. Over the years he’s been involved with various parts of the development process, including application development, design, UX and testing. On top of mobile apps, Wayne has had the opportunity to work with other technologies, including RFID, Embedded Devices, IoT and Bio-Med.
Wayne also has great passion for lifelong learning. When not actively involved in building apps, he loves to engage with others to help promote adaptive innovation and building user-centric solutions using such approaches as Design Thinking and Agile/SCRUM.
3:30 p.m. Networking