It is often said that usability testing is a necessary evil of software development, it is said by some to disrupt the development process whereas others find it takes too much work to properly instigate. What is often forgotten is that there is tremendous value in observing real users working and interacting with an application, prior to release.

At Arrk Group, we are firmly of the belief that usability testing can be enormously beneficial for teams at various stages of software development. A recently undertaken customer project further highlighted this belief, with the customer not only finding the outcomes enlightening but the usability issues flagged up during these sessions were immediately prioritised to be taken up in the next development release.

Why Usability Testing?

The simple idea behind usability testing, yet so important for high quality software development, is that if the real people who use the finished product cannot effectively do so then they will not want it, however wonderful and ground breaking it might be. Some of the reasons why usability testing helps during software development:

• Alternative viewpoints. By providing a ‘fresh pair of eyes’ on the application, interacting with the product as an end-user offers a completely unbiased opinion with no vested interest in any element of the design or functionality.
• Unpredicted responses. Usability testing is an excellent way of recording and evaluating any workarounds, hacks and alternative paths that users might try to make tools fit the way they want to use them.
• Playing the trump card. Good quality, rigorous usability testing analysis will trump development team gut feelings and opinion wars, which means informed developmental decisions can be taken quicker and add impetus to the project.
• Search engine performance. For some projects the impact on search engine organic results performance can be assisted through usability testing. While the influence is indirect, a search engine indexes websites by looking for quality content and stacks it based on keywords, links, site structure, linking patterns, user engagement metrics and machine learning. In many cases mimicking a user’s journey through an application.
• Pushing boundaries. Usability testing ensures that the development team constantly places the end user at the centre of its work, and stops teams from straying in a coding comfort zone.

How we Usability Test

At Arrk Group, Usability Testing has been carried out for applications which are in-use but the success with the Usability Testing framework along with our EmbArrk sessions has convinced us that Usability Testing can be successfully performed earlier in development projects, for example as early as when draft UI / mock-ups are produced. Usability Testers are selected from Arrk Group’s systems administration team, with a team of Testers selected on the following criteria:

• No previous exposure to the application being tested.
• Experience of using similar monitoring tools.
• Broad mix of expertise.
• Ability to think like an end user to understand likely scenarios and context.

Usability testing can incorporate both internal and external people, the important aspect is to create an end-user representative sample size to make session outputs meaningful.

The Usability Testing framework

Our framework consists of five sections.

1. Logistics and Test Planning

In this first stage, resource allocation, scheduling, test strategy and Test team recruitment is determined. In case of external participants look to arrange and confirm session dates, with draft session itinerary where possible, also agree any participant incentives for completing usability tests. We work collaboratively with the customer to develop and agree the Usability Testing objectives and prepare detailed participant profiles.

One testing team member is selected as moderator for a specific session and where a project requires multiple Testing sessions an alternate moderator should be selected. We do this because every moderator has their own style of running a session and switching between observer and moderator roles helps to foster learning. An example of Usability Testing objectives might include:

• How many participants successfully complete a task?
• How quickly do the participants complete a task, on average?
• How many errors are made while attempting to complete that task, on average?

2. Test Preparation

Test preparation involves defining tasks for the participant(s) to perform, setting up the test environment and publishing the final schedule to all the involved parties. So what works as a test script?

• The tasks must be easy to understand, self-explanatory and contextual from an end-user point of view.
• All complex and detailed scenarios should be broken down into small and meaningful tasks.
• All stakeholders should be involved creating the test script, including the business analysis team (to ensure real user and business context), the marketing team (so that the selling points to be marketed to the consumer are covered) and the development team (to get a sense of the vulnerability of the application under test, its quirks, technical implementations, limitations, stable areas of the application).

In advance of a Usability Testing session an appropriate environment is required, with all tasks tested to ensure correct behaviour of the application and prevent any surprises. When setting up a test environment, consider:

• No distractions. Ensure the test application is stable and free of any unwanted distractions, such as sudden pop-ups, and conducive to taking the test and recording observations.
• Physical environment. Ensure that the Test room is a welcoming and pleasant environment, using comfortable seating where possible and ensuring the room is clean and well lit. Also make sure that there is sufficient space for the moderator to moderate and observe next to the participant.
• Logistics. Prior to the start of the session make sure that any tools and telephone handsets are working so that observers can follow and accurately record the participant’s actions and feedback.
• Provide adequate refreshments and breaks for all participants.
• Realistic time frames and preparing for the unexpected. The moderator should confirm participation and provide a realistic schedule.
• A contingency plan is also a good idea if some test participants or observers fail to attend.

3. Test Execution

The moderator should take responsibility for ensuring a smooth Usability Testing session, ensuring that all participants are welcomed and thanked, including an explanation of what is involved and how long it should take, as well as obtaining consent to record some or all of the session.

Consideration should also be given to asking participants to sign non-disclosure agreement forms. Remember that the session is not about testing the end user, merely their interaction with the application. Care should be taken in clearly expressing this to remove any apprehension the participant may be feeling.

Example Timeline
The moderator should check the logistics in the test room and the observation room and ensure that observers have what they need to record findings. The moderator should also provide some final tips for the observer team and hand over the task sheet(s).
The moderator greets and escorts the participant(s) to the test room.
The moderator provides a brief introduction, including any health and safety notices, and an overview of the session to the participant.
The moderator asks for consent documents to be signed and non-disclosure agreements (if required), while answering any questions or concerns raised by participants.
The test session starts, each task expressed and handed out individually to the participant.
The participant is asked to think aloud and asked to perform the task with no instruction/minimal involvement from the moderator.
The moderator should ensure the participant is given necessary breaks and refreshments between individual tasks, as required.
The observer team notes down observations and questions for the participant.
Upon completion of the session, the moderator asks participants for further feedback. Someone from the observation team passes a list of questions which the moderator checks with the participant.
The moderator asks whether the participant has anything further to add, thanks and provides the participation incentive (if required).

4. Observation

To ensure observation is successful, Arrk Group operates an ‘active’ observation, which means observers should:

• Try not to interpret, or second guess, the actions/responses of the participant.
• Focus on capturing the details and quotes of the participant, what they say and what they do.
• Remove any potential distractions, i.e. mute/switch off mobile phones and shut down any email programs.
• Not think about possible fixes during the session.

The observer team should also record the length of time taken by participants on tasks, number of misfires, issues with the test itself and other key metrics.

5. De-brief and issue prioritisation

After the completion of any Usability Testing session the moderator and observation team should gather for a de-brief to discuss and consolidate findings. Information gathered should be grouped into common themes, with any suggestions shared to improve subsequent sessions. The Testing team should refrain from providing solutions at this stage as there might be further Usability Testing sessions scheduled and alternative findings might be recorded. Once all Usability Test sessions have concluded a final Data Analysis De-Brief is held where all stakeholders meet to discuss findings and prioritise any usability problems which have arisen from the sessions. Some of the frequently found issues which are raised during Usability Testing sessions include problems with search filters, screen layouts, colours and font usage, help buttons and general sizing issues.

Conclusion

Usability Testing should be considered a key component in software development, if an application is to thrive. Users are diverse and complicated, and if an application fails to meet their needs or is counter-intuitive, it is unlikely to succeed. By sampling end-users before releasing an application, the likelihood of the application being ‘user friendly’ is increased. Furthermore, Usability Testing can also provide highly valuable feedback which can feed back into the development of the application. Usability testing fills a pivotal need and complements the other methods of validation like functional testing, performance testing, and accessibility testing. At Arrk Group, we are convinced of its merit in high quality software delivery.

Appendix

The three roles of Usability Testing:

The Moderator

The driving force behind the Usability Test mechanism and if Usability Testing was a team sport the moderator would be the captain. The effectiveness of a moderator can make or break a test. The moderator needs to understand usability aspects, its rationale, must be communicative but not overly and must be people friendly.

The moderator role consists of the following:

• Managing the Usability Testing session.
• Determine and coordinate the tests that the participant should carry out.
• Selecting, interacting and briefing the participant on what is required.
• Encourage, prompt, observe and probe the participant’s actions in a neutral manner.
• To deal with situations that arise in a composed manner.
• Co-ordinate with the observation team and drive the post-session data analysis sessions.

 

The Observer

The observer is someone who attends the test and records some or all of the participants actions, observers could be members of the development team, managers, and members of other companies who are developing interacting products. The observing team is usually located in a different room and observe the participant’s actions of the test tasks, recording them for future analysis and design decisions

Diversity of roles helps in an observation team. At Arrk Group, we include a mix of lead developers, management and UX experts.

The Participant

The participant takes the role of the end-user, performing set tasks and provides honest feedback on a range of application elements. Participants play a vital role in Usability Testing and their contributions can have a profound effect on the development project and should be carefully selected based on the sample profile of the end-user and their ability to articulate their feedback to the moderator and observers.

At Arrk Group, we often use questionnaires to screen participants on the basis of whether they fit the profile of a real end-user. Characteristics considered can be as simple as age, gender, social indicators and field of work. Reward incentives are often provided to the participants for their time.