User testing: Guerilla Methodologies
Over the years, I have become a strong advocate of “guerrilla” user testing. While your testing methodology needs to meet your objectives, the benefits of guerilla testing are obvious. Most importantly you can do more testing more often. This allows the UX designer to present iterations to users and gather feedback through out the process.

Ultimately I like to aim for 5-6 users based on Jakob Neilson’s research into the optimum number of user to test for. This is also a really nice number – because from a management perspective you can test this many users in one day. I like to aim for a 20-30 minute test once an hour. This gives me time to consolidate my notes, check my emails before the next participant arrives. But don’t be fooled even this kind of testing takes time to prepare, recruit users, agree with stakeholders on testing objectives, reporting and respective design iterations.

Why leaving my notes until tomorrow just didn’t work
Despite being a very good note taker, I was finding that by the end of the day, there was a lot of information to digest. Plus the overwhelming sense that I wanted to get as much of the information down while it was all fresh, before sleeping on it. However, we all know that user testing is a long and tiring day. So it became clear to me that there needed to be a better way to streamline this. It seemed clear to me that this would be to do some of the work as I went along.

Introducing the analysis wall
So I created an analysis wall which I would go to and work through after each session while the content was fresh in my mind. The analysis wall, pictured, should be out of the view of your user group. I hid mine behind a white board, but if you have a corridor, or separate meeting room available that would work well. As you can see I’ve simply printed out the interface onto separate A3 sheets with plenty of room to write.

User Testing Analysis Wall

Also importantly, coloured textas….
The first time I created my analysis wall, it became clear that my blue biro for all users was not particularly helpful. It became a mush. It also became really difficult to connect the user to the finding. However, when using coloured textas I was able to highlight that all of the dark green findings are from a 35 year old woman, with plenty of internet experience, but no mobile web experience and no iphone experience. This is a really important aspect of the finding.

Using coloured textas for documenting user findings on the analysis wall

Outcomes/ benefits
The first time I conducted the analysis this way – I was blown away by how much time it actually saved me in pulling together my report. Using the analysis wall had meant all I had left to do is pull the findings together, offer recommendations and report in. The time saving was considerable.

Secondly, I was blown away by how I was able to engage and illustrate the “big things” to the project team immediately by inviting them to come and check out the analysis wall that afternoon. While this doesn’t replace the need for documentation to share with stakeholders, it gives an immediate and powerful overview of the user feedback. In the past I had found that stakeholders tended to observe only one session, leaving them focused on the findings of that one session only.

Most importantly, what you end up with from the analysis wall is a visual illustration of each user’s feedback among the group, and you can easily see where the views of each user align and where they diverge.

Links:
Jakob Neilson: Why You Only Need to Test with 5 Users
http://www.useit.com/alertbox/20000319.html
Andy Budd: Guerrilla Usability Testing
http://www.slideshare.net/andybudd/guerilla-usability-testing

Scott Berkum, who I saw speak at the Web Directions South conference in Sept 07 asks what people would be interested in for the 17th chapter of his book Art of Project Management

Here is my answer:

I’ll have to put my vote in for “Learning from projects after they ship (or your iteration ends)”.

I’m really big on reflecting, learning and getting better at what we do as individuals and teams and am currently lobbying for this where I work. The drive for rapid development in large to small businesses within the interactive industry is leaving this valuable step short. (or for dead)

Where I work we make the time for individual reviews against our job descriptions and KPIs but somehow move on to new projects regularly without looking at what could work better, how we could improve our processes or skills and importantly whether users or stakeholders liked the output. Is the product being used/ meeting performance targets? Is it a good user experience?

Often a project is compromised along the way in some aspect. Sometimes this is ok and perhaps even leads to the “mother of all invention” but other times the root of the compromise is retained by the lack of review. Neglecting this step leaves many of the tough issues unexamined or glossed over, conveniently left behind until another occasion for the compromise resurfaces. Likewise – it would be great to look at what worked well so that we can try and replicate that.

Often I think some of this neglect is human nature. As much as I love a good problem, I love to see the back of a problem that got in the way of outcomes.

In my mind it is important to look at how a product/project is received, how it has performed, what the feedback is, how the process could be improved, what skills and training would be useful for the team, what people have found that they are really interested in that they hadn’t thought of before… just for starters! I think many people don’t know where to start with this sort of assessment – especially in teams, so that the useful stuff can come out and work its way into constructive implementations to the way we work.

A chapter and exercises that extract and separate subjective outcomes and hard evidence (such as metrics) would be really useful.