Is your beta effective?

December 21, 2011

Your boss is a little nervous about the upcoming release and wants some reassurance that things are going to go fine. You tell her, “We had 15 beta sites and they had the product for 4 months!  It is just fine.”  Should your boss be happy?


A good boss might ask how many bugs the Beta’s found? What they did with the product when they had it?  How many Beta sits were active?  But too many times I find people really think that just having Beta sites is enough. It is not.  They have to do real things with the software.  You may have to guide them and make sure they are really working with your capabilities, if they are to be worth the cost of having a Beta test.


What is Beta?


Beta is when people not involved in building the product get to try out the product to determine if it is ready to be released to the large population. End users, people not part of the building or testing team, will generally approach the product from a very different perspective than the people involved in building a product. We get a whole different kind of testing from people trying to use a product than we do from people just testing.  Testing is clearly important, but Beta is also important.  That feedback is essential.


Isn’t it just a matter of numbers?


Most beta situations do not gain from the law of large numbers.  The goal of Beta is to have customers really push the product around for real and in so doing bump into the limitations and problems that the engineers and the QA people did not find.   If you have enough people, maybe they will trip across these things, but you have to have a lot of people for a pretty long time (e.g. Windows7, Chrome). But most Beta tests do not have a lot of people and often do not last long enough.


To have a decent Beta, you need to know what it is that your customers are going to be doing with your product.  Chances are they will not be putting into production. Chances are they will play with it for a little while and lose interest.  The customers love for the shininess of having a Beta wears off very quickly when it turns to work.


A few products ago I found someone in another company to Beta test my product who gave me a 10 page report after a three month Beta.  The report was awesome.  He never bought the product, as his company would not give him the money. When it came time to Beta test the next release I called him and asked them if he would Beta test again, he did.  This was the best Beta tester I ever had and I gave him the product for free.  It is hard to get a good Beta tester.


Unfortunately, Beta customers follow the following profile:

  1. Sales wants them as a Beta to get in good graces.

  2. The customer wants to be the first to have the new features.

  3. The customer is well intentioned but has no plan.

  4. The customer did not get approval for the Beta testing and nothing happens.

  5. IT will in no way support testing what you are trying to do.

  6. The Beta customer thought others would love trying to test the new thing but was wrong.

You really have to convince yourself that the Beta customer is going to do something useful with your product.  They cost time and money to have as a Beta site and do not let them be a Beta site if they have not committed to doing testing and showed you what they are going to do.


“But I just need some sites, why are you making this so hard?” you might say.  It is because lying to yourself (and others in the company) that you actually have a Beta test going is not a good idea. They are either set up to do something useful or you should dump them.  Even then, many of those you signed up will not be able to perform the task they have signed up to.


Beta tests are too important



© 2011-18 Westerly Consulting LLC, all rights reserved


Share on Facebook
Share on Twitter
Please reload

Featured Posts

I'm busy working on my blog posts. Watch this space!

Please reload

Recent Posts

November 23, 2015

September 11, 2014

Please reload

Please reload

Search By Tags
Follow Us
  • Facebook Basic Square
  • Twitter Basic Square
  • Google+ Basic Square

© 2017-18 by Westerly Consulting, LLC

Westerly Consulting, LLC

All photographs by Fred Engel