login
Blurts on the Art of Software Development

Today | RSS | RDF | Atom | Other Tags
Categories : All | All | CI | .NET | General | Humour | Java | Personal | Reviews | Ruby | SW Eng

Jonathan Kohl blogs about the dangers of reckless test automation. He's right on the money and I strongly recommend reading what he has to say.

One thing I'd like to highlight is that the source of the problem is not "100% test automation" itself but the interpretation of it. I haven't heard a single agilist I consider knowing his (or her) stuff say that the goal should be 100% test automation and 0% manual testing. What I have heard people say (and what I've said myself as well, from time to time) is exactly what Jonathan is suggesting: that all regression tests should perhaps be automated but that context-driven, risk-based, exploratory manual testing should be carried out in addition to the automated tests.

Computers are good at finding bugs we told them to look out for. Humans are excellent at finding those we couldn't see coming.


Thanks for this, Lasse. Good and thought provoking.
Hey, thanks for the kind words! Nice post. I do have a question here though: "I haven't heard a single agilist I consider knowing his (or her) stuff say that the goal should be 100% test automation and 0% manual testing. " What room in 100% test automation is there for any other percentage? Isn't 100% inclusive of all? How can I squeeze in any manual tests if 100% of the tests are automated? Now, I know what you mean, but the language we use is important. I run into this frequently with decision makers - recently with a CEO who had heard this figure and wanted us to stop doing manual testing. When he heard "100% of tests should be automated", the corollary is naturally that there is 0% left for any other tests. If he had heard "80% of the tests should be automated", or "most of this kind of test should be automated", it would have saved him from some confusion. In this case, this team had a history of reckless test automation, and the manual exploratory tests we were executing found 5 showstopping, expensive, embarrassing bugs that were in production, missed because of so-called "Agile Testing" thoughtless automation efforts. He felt we would be more effective if we dropped the manual tests altogether, not realizing that the balance of both was key. If the CEO comes and tells an employee you can't have manual tests, and your performance review is tied to automated tests, etc. etc., what are you going to do? In the end, the CEO said he cared about results, and thanked me for clearing the confusion. I run into this frequently, which is why I rant. Good blog by the way - I appreciate what you're doing. :)

That's a good point, Jonathan. "Most" would likely be a safer choice of words in many situations. Especially in those where the medium used for transmitting the message is the mostly-one-way-with-delayed-interaction kind such as weblogs or email.

Missing the slight movement of the eye-brow when you've said something can mean the difference between the recipient getting your message mostly correct or mostly incorrect.

Oh, and thanks for your own kind words as well. Much appreciated.



Add a comment

Title
Body
HTML : b, i, blockquote, br, p, pre, a href="", ul, ol, li
Math Quiz 6 + 3 = (Helps stop blog spam)
Name
E-mail address
Website
Remember me Yes  No 

E-mail addresses are not publicly displayed, so please only leave your e-mail address if you would like to be notified when new comments are added to this blog entry (you can opt-out later).

TrackBack to http://radio.javaranch.com/lasse/addTrackBack.action?entry=1144492933692