On Monday the 13th of January 2020, TestHuset hosted the seminar ‘Putting AI to the Test’ for 50 people, who got the answers to 5 crucial questions from Roland Møller, CTO TestHuset, Viepul Kocher, AiU and Stefan Veis Pennerup, Trifork:
- Does AI mean the end of manual testing?
- Does AI require a thinking (human) tester?
- What are the vulnerabilities of AI?
- What type of business will AI generate in the future?
- Is this a defining moment in testing history?
Does AI mean the end of manual testing?
Absolutely not! AI will be implemented as a supplemental tool to manual testing and a human touch. Some of the existing test forms will be transformed, where humans are still involved, but more as a validation of the test result. Testers must acquire an understanding of what AI/ML is and the special challenges it poses, as well as techniques, such as metamorphic testing and the ability to understand bias in the data.
Does AI require a thinking (human) tester?
Yes, the thinking tester is back! In the past, due to agile philosophy of generalist rather than specialist testing and agile = need for speed, there’s been a focus on automation and unit testing, which diminished the role of the thinking tester. The testing of AI cannot be completely automated because a human must think and take precautions to avoid possible skews, scenarios, corner cases and bias in the data. Testing AI requires skills at a different level, where the tester must consider and offer advice on bias, moral and ethics. It’s testing and quality assurance at a whole new level. As this involves values, the testing and quality assurance of AI solutions becomes a potential branding platform for companies and organizations. However, it could also turn into a major crisis, if testers and companies do not consider their own bias and fail to pay attention to the shifting social reality of bias. As a tester you must be aware of and challenge not only your own bias, but also the bias of the users in a region, a population, a gender or a work culture, because it will affect the quality of the solution if you don’t. So, always think and beware of bias!
What are the vulnerabilities of AI?
Until recently AI has mainly generated excitement about its vast possibilities. But since 2015, where some of the first research into the weaknesses and ‘cheating opportunities’ that AI is vulnerable to was published, there has been an increased focus on its weaknesses, which most definitely will become a key topic in the future. The big difference between quality assurance and testing before AI and now, is that it used to be enough to test the software and algorithms; now you must test the data/input on top of that. And there is no oracle, no definite fact sheet telling you an objective truth. Before AI, the result of a performance test could be either good or bad. Now, you’re dealing with probability. AI will never be 100% accurate, but it can be more accurate than a human.
What type of business will AI generate in the future?
AI will free us from tedious, laborious tasks and create more quality and knowledge intensive jobs. AI will also give us a lot more time on our hands to be entertained and creative. Although this all sounds great, it obviously must be dealt with at a political level to counter social imbalance.
Is this a defining moment in testing history?
Yes! This is a defining moment in testing history indeed. It’s time for testers to reinvent themselves, to understand technology and re-apply their skills of critical thinking and asking questions. There’s also a great deal of research opportunity available, because the field of testing AI is at a very early stage.
Feel free to send your comments and questions at info@testhuset.dk
About the people giving the answers
Roland Møller, CTO and Partner at TestHuset, has extensive experience with Business Development, DevOps and Quality Assurance from major IT and Financial corporations in Denmark and abroad. Roland ensures TestHuset stays at the forefront of the Tech and QA frontier.
Vipul Kocher (to the left), Roland Møller (in the middle) and Stefan Veis Pennerup (to the right)
Stefan Veis Pennerup is the ML and Google Tech lead at Trifork in Aarhus, Denmark, with a background as a Mechatronics Engineer. He has recently developed a habit of using Brazilian flip-flops during presentations and has a hypothesis that Machine Learning and flip-flops are related – although correlation does not automatically imply causation. Introducing new customers to Machine Learning on a regular basis has prepared him with a long list of easy digestible examples.
Viepul Kocher is an ex-Adobe engineer and IIT alumnus with a 25-year experience in Software Development and Testing industry. Besides being the creator of AiU – the world’s first AI testing certification, DoU – DevOps united – world’s first DevOps and testing certification, Viepul is also President of the Indian Testing – ISTQB Board, Convenor of STeP-IN forum and National Convenor of Indica Academy, to name a few.