Category: Test Automation

Today, Elisabeth Hendrickson has a blog post out titled How Much to Automate? Agile Changes the Equation. In this post, she points out the value of timely, reliable automated test results:

But teams that do practice TDD and ATDD wind up with large suites of automated tests as a side effect. (Yes, it’s a side effect. Contrary to what some people think, TDD is a design technique, not a testing technique. We do TDD because it leads to clean, well-factored, malleable, testable code. The automated tests are just a nice fringe benefit.)
Moreover, the resulting set of automated tests is so much more valuable because the test results are typically reliable and trustworthy. When the tests pass, we know it means that the code still meets our expectations. And when the tests fail, we’re genuinely surprised. We know it means there’s something that broke recently and we need to stop and fix it.

Upon reflection, I realize that I may have assumed too much in my recent discussion (see here and here) with a colleague about his company’s automated tests. I explained the cost/benefit of creating and maintaining different types of automated tests, but I did not discuss the cost and benefit of the information that they provide; I made some assumptions in this regard.
If I had it to do over again, I would have asked him about the value that his group receives from having and running the automated UI tests. From there, we could have discussed more explicitly the value they would derive from having a robust set of unit tests and running them regularly.
As it was, I assumed that the relative worth of unit tests was much higher than the GUI tests. For reasons that I’m unaware of, that may not be the case; Or, more likely, my colleague may not have thought about the value derived from different types of tests and my exploration of that aspect would have lent additional credibility to my recommendations.

I had a conversation recently with a colleague whose small company had contracted out their testing. One of the (several) problems of this arrangement, he said, was that the testers had created automated UI tests, but they were having trouble keeping the tests up to date. The colleague asked me what I thought they should do about it.
I gave him my standard spiel about the cost/benefit of automated testing: as a general rule, UI testing is the least cost effective automated testing that you can undertake. The main reason is its high maintenance cost; you have to update your tests pretty much every time you change the UI. And in my experience, the UI changes a whole lot more than the underlying layers.
After delivering this sermon, I asked my colleague what other types of automated testing they have. He said that they have a few unit tests that are not run on a regular basis and do not all pass anyway (so, basically, no other automated tests).
Based on this information and other things that my colleague told me, my conclusion was that the automated UI tests were not their real problem. I recommended that they give up on the automated UI tests altogether and focus on developing a comprehensive suite of lower level tests that run and pass on a daily basis, starting with unit tests. Once they have that base well covered, then they should start thinking about automated UI tests.
I got the distinct impression that my solution was not at all what my colleague wanted to hear. He wanted to solve the specific problem of the automated UI tests using their existing QA resources (the contract test team). I told him instead that they actually have a different, and broader, problem and that his company needs a more comprehensive approach to solving it.
Unfortunately, we tend to get so focused on specific problems that we forget to step back and look at the bigger picture. We miss the forest for the trees.

John Overbaugh offers some advice for finding the right testing tool. In addition to listing factors to consider in evaluating tools, John also offers some tips on how to elicit advice from others. Good stuff.

Recently, I’ve had several conversations at work about the cost/benefit analysis of performing automated GUI testing (in our case, using Borland SilkTest, of course) in our agile environment.
In general, automated GUI testing is most useful in the following situations: when the UI of the application under test (AUT) does not change much, and when there is significant regression testing to be done–where the time commitment of creating and maintaining automated tests is lower than the time commitment of repeating manual regression testing.
I’ve been working with an agile team that is in the first few sprints of a new product; this product’s UI is still changing frequently at this point and the product does not yet have much functionality for regression testing.
So, we have a resource allocation dilemma: this team would not benefit particularly from automated functional testing at this time. However, in, say, a year, when this product is more mature, automated UI tests would be very beneficial. But if we wait until then to start automating, we’ll never get caught up, so we need to start devoting steady effort to the automated testing soon.
In an agile environment, it’s difficult to get resource commitments for efforts that only have long-term results.
I’d love to hear whether others have faced this same dilemma and how you dealt with it.