Suggestions for Testing: The Tester's side
Rule 1: Do not release the beta you’re given.
This should be obvious, but I mention it because it is the most important aspect of the beta-tester’s trust. Also, do not discuss the game you’re testing in public before it is released. And do not, ever, mention the bugs you encountered in the pre-release version, unless you have the author’s explicit permission.
Establish a form for feedback.
If the author doesn’t make this clear, and it’s not an author I’ve worked with before, I usually ask what kind of feedback he would like to receive. There are two things to settle on, form and content.
Content: Some people only want reports on bugs in the function of the game; others want a wholesale critique of the entire gameplay experience. The former comes up especially frequently when the author is re-beta-testing a new version of a game that has already been released — say a competition game that he has reworked a year later. There’s good reason to test again before re-releasing the game, to make sure that no new bugs have crept in while the old ones were being fixed. In this situation, though, it’s too late for remarks about the plot and so on to be really useful. In other contexts, however, authors may welcome feedback on the non-bug aspects of the game. I have in the past gotten comments about puzzle difficulty and cluing; about the efficacy of my UI design; about additional verbs that would be nice to have, or additional comments that might flesh the game out more;
Form: Some authors want to receive transcripts of your entire play experience, while others want just a message summarizing the bugs you found. If I’m beta-testing a game, I play with scripting on — whether or not the author has requested transcripts as the form of his feedback — and then I have the transcript to refer back to. Frequently, I send the annotated transcript along with a few paragraphs of summary that contain my overall feelings about the game and any things I think are most important to change or improve. (I also try for a few lines of encouragement about the parts of the game I did like, but this is not an essential aspect of the beta-testing job. The chief task is to get the game into playable shape; keeping the author happy is sometimes not perfectly compatible with that task.)
When I say that I annotate a transcript, what I mean is that I type my comments directly into the game as I play, usually prefaced (consistency is the key here) with asterisks or brackets, like so:
>* Missing space in the second paragraph.
>* I dropped the featureless cube but it's still in my inventory.
(TADS and some generations of Inform actually provide for this annotation so that no game time will elapse after a turn that starts with *.) Typically I read the entire transcript through at least once, but sometimes later when I want to review specific bugs I find the searchability important.
This is pretty standard beta-tester behavior (inasmuch as anything can be said to be standard in this community). Some of my testers get quite chatty with these comments, and I also get things like this:
>* Yay!
>* Wait, what??
>* ARGH
>* I thought the Marquis was older than that.
>* Rowr.
…etc. It’s sort of as though the tester is having a running conversation with me about the game as he or she plays.
Follow your own style.
What you actually do while testing is largely up to you. As I mention elsewhere, some of my testers play as though they were experiencing the game as regular players — which is useful, and gives them a regular-player sort of perspective on the game. I have also had or heard of testers who
- Tried every possible direction from every possible room, even if there wasn’t supposed to be an exit there.
- Closed every door behind them when passing through an area, and/or otherwise experimented with restoring things to their initial conditions.
- Ruthlessly repeated all NPC-related actions — showing them things twice, asking them the same questions over again, and so on, to see whether critical triggers fired twice or only once.
- Tried to attack, break, cut, burn, or otherwise destroy all important objects to see whether they interacted appropriately and whether it was possible to lose game-critical items.
- PUT ALL into each container item to see whether capacities and sizes were handled intelligently.
- Ran around using any special new verbs I’d written on the most implausible objects — scenery, other characters, the player, etc.
- Disassembled the gamefile in order to find things that should have been accessible, but weren’t.
- Set their monitor or screen size to different settings to see how a graphical UI behaved at different sizes and aspect ratios.
Testing a game often involves deliberately trying to break it. As you play a game, you may get a sense of where the code is likely to be most complicated. Anything that involves an important plot trigger, any complex simulation, any added verbs or language: these are things that can use special investigation. But ultimately, you should do what you like when testing. It is part of the author’s job to find several beta-testers and, if possible, to get several testers of different temperaments. No one tester will do or think of everything — so do what you do best, and trust that someone else will catch whatever you don’t think of.
Send reports in chunks.
Unless the game you’re testing is a short one or the author has made it clear he has a different preference, it often makes sense to send your reports serially rather than in one big batch. You may only have a few hours at a time to concentrate on a game; the author would probably prefer to get that feedback immediately, so that he can start fixing those bugs, and also because waiting in limbo for feedback is nerve-wracking. Also, a huge email containing hundreds of bugs is much more daunting than five emails each with thirty or forty. In my larger games I have fixed thousands of bugs.
The general operating procedure for my testers has usually been that they will play my game until they’ve worked up a few transcripts and have gotten stuck or come to a major plot break-point. Then they send me the files, and ask any questions they have about how to solve puzzles that are daunting them. I never send walkthroughs to my testers in advance, because finding out how they interact with the puzzles, and whether they get frustrated, is part of what I use the testing process for.
Assuming the author is interested, share your reactions.
I always like to know how my testers feel about a game. Sometimes they have specific complaints about the way I’ve arranged the windows, or about my writing, or about the way a puzzle is clued. Sometimes they only have vaguer impressions — a sense that the gameplay is too rushed or too slack, or that the narrative is too confusing. Even that kind of information is useful, especially taken in combination with the feedback from other testers: it gives the author an idea of how the game is likely to be received, and may allow her to diagnose the problem, even if you don’t know what exactly is wrong.
If you have to stop, let the author know.
It’s not at all infrequent for a tester to find herself swamped for time and unable to finish testing a game. If you’re testing, you’re doing so at your own option, and there’s no shame attached to quitting — either because you’re busy, or because the game is not to your taste. (If you really dislike a game, you’re unlikely to be a good tester for it anyway: too much of the process requires goodwill on your part. It’s one thing to find a work flawed but intriguing — that you can work with. If you don’t think you’ll ever like it however well-executed it is, though, you should think about putting it aside and working on something else.)
If you do quit, you should let the author know that you’ve decided not to go on so that he can find a replacement tester. Simply ceasing to send reports may leave the author in an uncomfortable limbo position.
