During
last weekends European Weekend Testing session we tested
Brainworkshop by using tours based on the
FCC CUTS VIDS heuristic.
After starting with some delay we started by extending the timing for testing and reporting to 45 minutes which was quite important to me since I had to spend a few minutes to review the details on the heuristic. We also decided on splitting up the tours to get a better coverage.
My first impuls was to take the "Features" tour, because it gives the possibility to look around without being to specific on what to look for. However during downloading the product I had a quick glance over the website and since it look very clear and detailed in pointing out what the software should do and made clear that there are a lot of options when using the program I decided to go for the "Claims" and "Configuration" tour.
At the end of 35 minutes touring I took some time report the following findings
Claims
- Implements the Dual-N-Back exercise
- Works on XP, Vista, Mac OSX and Linux
- Usage improves working (short term) memory and fluid intelligence
- What is the definition of "fluid intelligence"?
- Keeps n-back level reached, for the rest of the day (restart)
- session stats can be graphed inside the application
- session stats are stored as csv on the filesystem
- Plays music between sessions depending on "your score"
- Which score? The one from the last session or the overall score?
- N-Back modes (audio/position) can be run separately
- Configuration changes (
config.ini) are visible after a restart
- Only if you get to the screen where the option is used.
Configuration
- The configuration can be changed modifying the
<brainworkshop>\data\config.ini or changing options in runtime
- Clarify if the changes in runtime are stored in the .ini
- The keycodes are based on ASCII keys (A-Z)
- What happens on non-ASCII keyboards?
In addition to the tour related information I found the following issues
- http://brainworkshop.sourceforge.net/#tutorial has an
<li> element to much (in Additional Game Modes)
- Pressing on the main screen closes the program but produces and error popup
- Also when using the "Close" button of the window (X top right)
- Possible connected error message (from log):
F:\Downloads\brainworkshop-4.7-win32\brainworkshop\library.zip\pyglet\window\win32_init_.py:552: UserWarning: Could not set vsync; unsupported extension.
- Can't be resized. (Is this really a bug?)
The most interesting point of the discussion was for me that Anne-Marie Charett said that the quick "Structure" tour she took would completely change her approach for testing since the issues found in former versions might well arise again in new parts of the code.
For me an open question after the tour was: Did I get the "right" information accordig to the heuristics that I used? To clarify this in more detail let me quote the tour descriptions:
- Claims tour: Find all the information in the product that tells you what the product does.
- Configuration tour: Attempt to find all the ways you can change settings in the product in a way that the application retains those settings.
Now, if you compare my findings with the heuristics do you think I have found information that fits the respective heuristic or not? I can't really say if this is the case, since these are not my heuristics.
For example I extended the "Configuration tour" to also include settings made in runtime to develop tests that find if a configuration is stored or not. Should this really be part of the tour? Should runtime configurations be tested in case they are unexpectedly stored or not?
I have the strong feeling that if I use heuristics which I haven't developed myself they will not only be less effective but might actually lead me down the wrong road when using them. This can only be corrected in a detailed conversation with the "owner" of the heuristic.
