Swain
UX Research
Iterative Testing and Design
Project Overview
Many email marketing campaigns center around a given call-to-action (CTA). This project sought to empower users of our WYSIWYG email editor by encouraging the incorporation of calls-to-action such as "RSVP", "Donate" and "Poll", each of which would link email recipients to a unique landing page where they would fill in relevant personal information. In addressing this design challenge, we needed to ensure that email senders understood not only the interactions involved in adding and editing one of our calls-to-action, but also the experience their email recipients would receive on the other end.
Process
This project involved many open ended questions and the development of design patterns that would be entirely new to our user base. Facing much uncertainty and a wide range of possible solutions, we decided to implement an iterative design process incorporating three rounds of usability testing. We decided on this multi-phase testing structure as a means to first narrow our field of options, then refine and ultimately validate our chosen design.
We began the first phase of testing with five candidate designs, each represented by an interactive Axure prototype. To determine which design treatment most effectively promoted user comprehension, I conducted a between-subjects usability test of the five prototypes on usertesting.com. Each prototype was tested with a minimum of ten participants. Remote, unmoderated testing was chosen for the speed with which it would deliver results and allow us to narrow our options. The number of subjects, which is relatively large for an early stage, formative usability test, was chosen due to the limitations associated with recruiting through usertesting.com and the wide range in participant experience with WYSIWYG email editors.
Screenshot 1: Axure prototype of WYSIWG email editor
Success in our first phase of testing was judged according to participant comprehension of the task they were completing and their ability to anticipate the experience an email recipient would receive when engaging with their CTA. One of the five original design treatments stood out distinctly, including self-explanatory interaction patterns and effectively foreshadowing the email recipient experience. This winning treatment was then refined by our design team, eliminating minor usability bugs and enhancing prototype interactivity.
Our enhanced prototype was then subjected to a series of moderated usability tests, conducted with participants drawn from the current active user base of our product. Recruiting and scheduling active users is more labor intensive than outsourcing the recruit, but we considered this investment worthwhile as it helped ensure a representative participant population. In this round we tested the single prototype with nine participants, split between regular users of our product and the most experienced "super-users".
The second round of usability testing confirmed our findings from the first regarding general participant comprehension and highlighted additional opportunities for prototype refinement. Having completed these two rounds of testing and incorporated our newest findings, we felt that our design was ready for validation in a summative usability test. This phase was conducted as an unmoderated test on loop11.com, with 165 product users moving through our finalized interactive prototype. At each step in the process, participants were asked a question designed to determine their comprehension level. In some cases these questions were open ended and analyzed manually, in other cases they were presented with a series of statements and asked to indicate their level of agreement on a 1-5 Likert Scale (Strongly Disagree - Strongly Agree).
Outcome & Reflection
This iterative design and testing process generated and validated design patterns that would eventually make their way into the production WYSIWYG email editor. The early formative phases of testing succeeded in eliminating confusing design patterns and refining those that were most successful. Summative testing gave us the confidence to promote the new design and helped us gain buy-in from key stakeholders.
No process is perfect, and this one certainly could be improved upon. The first phase of testing in this case was decisive in that it determined the "big picture" design patterns that would advance to future phases of testing. If we could have spent more time brainstorming a wider range of alternative patterns to test against one another, we may have ultimately arrived at a stronger final product.
Additionally, the scope of our summative test was relatively narrow. As a result of limitations in prototype functionality and the constraints associated with an unmoderated test structure, we could only reliably gauge participant comprehension of our design and interaction patterns. The usability of these interaction patterns, although supported by qualitative data from our earlier rounds of testing, could not be validated in the summative test.
Screenshot 2: Current production WYSIWYG email editor