Home › Blogs › Humphrey's blog

What is a Beta testers role?

Fri, 04/14/2006 - 10:50am — Humphrey

What is a Beta Testers job in the world of Game development?
Are we nothing more than Glorified stress testers?
Are we essential places for information on the public? Basically a free marketing survey?
Are we places for a outside point of view?
Extra Minds to find bugs?
Replacements for QA?
Free Advertisement?
Etc.

Or are we all of the above?

I have been a Beta Tester for a varying amount genres. From You Don’t Know Jack to Auto Assault. From Planetside, to After Dark Games. And each in my view have had a few things in common, the list above.

As a beta tester I feel that my job, my responsibility is to fulfill a contract I signed when I signed up for beta testing. I believe that when you sign up for a beta test, and participate in one you sign an unwritten contract to do the following things:

  • Submit concise yet explanatory reports of bugs and anomalies
  • Report all bugs and exploits found
  • Submit suggestions on game improvements

You are not there for a free demo, and not there to play a game for free until you are required to pay for it.

As a beta tester I feel those are the “required” items that you must do. In return for allowing the public to become beta testers, the Game company gets the following information:

  • Bug reports
  • Server stress information.
  • an outside perspective
  • Free marketing
  • Free advertisement

The first Two are obvious, but the final three I feel need a bit of explanation.

Anyone who has done some creative work and writing knows that the greater the amount of time you spend on a project, the deeper you get into it and the harder it is to step outside and view from a fresh perspective.
You know what it is like to play character “X”, you know their future, and all the game play mechanics. You know what missions do what and send you where.
However what works for you might not for the general public. A fresh perspective on the game play mechanics, the storylines, the missions, the characters, the weapons, etc. All of that needs to periodically be seen by a fresh pair of eyes who have not been up to their armpits in coding and writing those items.
A beta tester can do this. They can see a game play mechanic from a new perspective and see if it actually works in the real world.

A beat test can also be seen from a marketing perspective. You do have niche markets out there for certain games. Allowing beta testers in and getting basic information on what market niche they belong to can give you valuable information on how the general public will react to your game. Instead of pulling random players form all over, it might be nice to be selective and choose random players from each niche market. One from the teenagers, one from the 20-something, from northerners, southerners, etc.
This way that you can form an informal survey of possible reactions by the general public.

Finally you get free advertisement.
Some of the greatest games out there travel by word of mouth. [A friend tells a friend who tells a friend…]. Letting players beta test the game, and making a good one will allow word of mouth to spread.

But a game company cannot sit back and enjoy the ride. They have an unwritten contract also when they allow testers to come into the game.

The must:

  • Read and follow thought with the bug reports
  • listen to the feedback. [Note: not required to follow through, but they should read it all]
  • Accept Testers opinions as valid and equal to their own.
  • Post information and reply to feedback

All are pretty much self explanatory except I would like to expand upon:
“Accept Testers opinions as valid and equal to their own.”

What I mean by this statement is to not get an ego about your work. It is precious and we know that a game company has put countless hours into a mission, graphics, storyline, etc. But if a majority of testers find fault with it, it should be changed. A mechanic you are inherently familiar with might be too complicated for the average player to “get”. Or a storyline that makes sense when taken in context with the overall world storyline, might not when taken as what part of the storyline as visible form the game itself.
Again I will say that you do not have to follow through on all, or even close to all suggestions. But the ones that are most requested, and the ones that most people talk about and complain about should be given serious consideration.

Several big name game companies have been faulted several times for over and over again not listening to their testers, not taking their feedback to heart, and like above, having a opinion that their ideas for the game are better than the average testers.
These are your future customers, and their friends are too. If you do not keep them happy and enjoying the game, you will quickly run into a problem of lack of people buying the game at retail.

So in essence, I feel that both require each other and both have a contract with each other. They are not solitary beings who do their own thing. They must rely on the opinions and relationships with each other to make a great game.

[edit:] Many of you will say "No Duh!" to most of what i said above. However some companies still do not follow any of the standard rules for Beta Testing. From what I have heared S.O.E. and SWG is a prime example of this.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Sat, 04/15/2006 - 2:31am — kaikai

very true!

very true!
i just wish more beta testers realised this, and thus we, the testers rather than free loaders, would get into testing easier, due to a decline in the "freeloaders"

wishfull thinking eh?

Sat, 04/15/2006 - 9:33am — Snipehunter
Snipehunter's picture

Sad but true on that edit Humphrey

I've had the chance to see how quite a few companies handle beta testing over the years. My first few jobs were as a QA tester, and more often than not, the beta testers were considered free advertising while the QA testers were expected to isolate and identify every single problem. I've worked on betas where the developers literally never saw the /bug reports - they left it up to the QA team to wade through them and input bugs into the devs bug tracker based on the /bug feedback. As a result the beta tester's comments and opinions are never seen. Sure you might fix the problems, but what about fixing the sentiment, or identifying the emotional tone of the feedback?

Still, as a developer, I'll tell you: The value of a good beta cannot be understated. Look at AA. We were so absorbed with our work that we were ignoring key factors like AI and environmental effects. I'm ashamed to admit it, but we really did expect that our beta would be short, positive, and then we'd be ready to do.

Some of us had our doubts, of course; but the culture at the studio was that the game was "awesome" and would be a hit. Imagine the surprise at the studio when we got the type of feedback we did at the beginning of the beta. We were fortunate; we were actually looking at the feedback and trying to respond to it. If we hadn't we might have shipped, right then. I can't stress how hard it is to push a game the size of AA back by 6 - 9 months the way we did.

That's part of the problem; the size, scope and length of a beta are all determined at the beginning of the development cycle. Often we developers "eat into beta" because the game is grossly under-scheduled. That's a plague in this industry: No one seems to know how to effectively schedule. I've literally only worked on one project that shipped on time and it was a piece of trash when it shipped. So, we try to make up for the overly-optimistic schedules and lost time by cutting the beta short and starting it later.

The net result is that betas are rushed. When this happens the beta becomes a PR exercise, at best. Worse, because the schedules are already wrong, everyone is rushed and the devs have little or no time to respond to feedback. MMOs are a little different, thankfully. We tend to schedule very long beta tests, at the very least, so eating into them still leaves us some time to see what people think. Eye-wink

For me, beta tests have always been about:

  • Ensuring acceptance of the game by the intended market (responding to feedback)
  • Finding and fixing bugs
  • Pre-building the community of fans by keeping your beta testers engaged and showing them the value of their participation.
  • Word of mouth buzz

I actually list that in order of importance. Bugs can be fixed as the game evolves, if needs be, but if you don't tailor the game to the people you intend to sell it to, what's the point of making the game at all? You can tell when this hasn't been done; the game's community is small and populations online (or sales for single player games) are much lower than expected. When I say much lower, I mean getting, say, 3,000 subscribers instead of the 50,000 or 75,000 you expected.

For an MMO that would be a disaster, both financially and for the community you built in the beta. How rewarding is it to spend months of your time basically working for no pay, only to have no one to share the game you helped build with? It's disheartening.

As a Dev, it's a crushing blow. Nothing is worse then when you see the die-hard fans and supporters you worked so hard to keep, just up and leave because there's no community. By the time you see it happening, it's too late to fix it; the damage was done months beforehand. All you can do is watch the best and brightest leave and promise yourself you won't cry.

- Snipehunter

Sat, 04/15/2006 - 9:39am — Snipehunter
Snipehunter's picture

It doesn't hurt to dream!

I've actually been involved in planning a few betas (as opposed to just being a dev on the team during beta) and the first I was part of the group planning the beta I was shocked by what my producer told me.

In as cynical a voice as I have ever heard he said, "1000 testers is too few, plan for at least 10k. Only 11% or so will ever submit a bug; the rest just want to play for free."

Since then, I've never been able to prove him wrong. Sad Worse, in most betas, at least a third of the bugs submitted are worthless because they're not really bugs; they're just people bitching instead of providing any useful information for finding, let alone fixing, the problem.

- Snipehunter

Sat, 04/15/2006 - 11:05pm — Humphrey

Aye, its a sad world when

Aye, its a sad world when people will sign up for betas only to try out a free game.

What i feel is worse, especially from a Game developers standpoint, is that many of those base their personal "Reviews" of the game based off of that beta.

I have always felt that you can never review a game until it is out of beta. Yet if you looked on EBgames, Gamestop, and even placed like MMORPG.com and Gamespot. So called "Reviews" were coming out months before Auto Assault even released.

A majority of them were coming from players who only had weekend accounts, or only played the tutorials. And their lack of knowledge was evident.

It is a double edges sword dealing with people like this.
Limiting Betas to select groups of people will severely limit your potential testers submissions.
But allowing anyone in also causes what is described above in this post and the ones before it. Freeloaders.
How do you deal with this problem?

I cannot think of a single way to prevent it that will not anger some group.