Usability Testing of an HPV Information Website for Parents and Adolescents.

OBJECTIVE
Parents make the decisions regarding their children's health care. Unfortunately, many parents are misinformed about HPV and HPV vaccines. In order to help parents make an informed decision regarding HPV vaccination for their daughter, the GoHealthyGirls website was created for parents and their adolescent daughters. Usability testing was conducted with members of the target population to refine the website prior to conducting an efficacy trial.


METHODS
Parents with girls (n=9) between the ages of 11-13 and 11-13 year old adolescents (n=10) were recruited for usability testing. The testing consisted of completing twelve scenarios where participants were asked to find specific information on the GoHealthyGirls site. This was followed by a self-administered system usability scale-to determine ease of use and functionality of the website-and a user satisfaction survey.


RESULTS
Both adult and adolescent participants were able to easily find the requested information and reported an increased positive opinion of HPV vaccines after visiting the website. Both groups of participants reported favorable evaluations of using the website.


CONCLUSION
The GoHealthyGirls website has the potential to help parents of adolescent daughters make an informed decision about HPV vaccination. A large scale efficacy trial will determine its usefulness.


Introduction
Human Papillomavirus (HPV) is the most common sexually transmitted infection in the United States and throughout most of the world (Hariri et al., 2011;Gillison et al., 2012;Mandic 2012). Research has identified about 40 types of HPV that can result in health problems, including low-risk HPV strains that do not cause cancer but do cause genital warts, and high-risk HPV strains which can cause cancer, including cervical, anal, vaginal, oropharyngeal, and others (Schmeink et al., 2013;Chaturvedi et al., 2011;Jayaprakash et al, 2011;Watson et al., 2008). The Centers for Disease Control and Prevention (CDC) indicates that HPV causes about 360,000 cases of genital warts and 10,000 new cases of cervical cancers in women each year in the United States (CDC, 2014d) However, 21,000 HPVassociated cancers could be prevented annually in the Unites States through HPV vaccination (CDC, 2014d) Two vaccines to prevent HPV infection have been approved by the United States' Food and Drug Administration (FDA, 2014). These vaccines (Gardasil® and Cervarix®) have been shown to be very effective against high-risk HPV strains that account for most cervical and anal cancers. Gardasil® also provides protection against low-risk HPVs that account for most genital warts (Koutsky et al., 2002;CDC, 2014a). Gardasil® has been approved for use in both males and females, whereas Cervarix® is approved only for use in females (FDA, 2014). Both vaccines require three doses over a 6-month period for completion (CDC, 2014a; and both the Advisory Committee on Immunization Practices (ACIP) and the CDC recommend HPV vaccination routinely begin for all kids at ages 11-12 so that recipients have time to develop an immune response prior to becoming sexually active (CDC, 2014a;. Healthy People 2020 (U.S. Department of Health and Human Services, 2014a) has set a target to increase 3-dose HPV vaccination coverage to 80% among 13-15 year old females, but has not set an HPV vaccination goal for males equivalent to that for females. However, the 2013 National Immunization Survey (CDC, 2014b) found that only approximately 26% of 13 year old females, 32% of 14 year old females, and 39% of 15 year old females in the United States received all three recommended doses of HPV vaccine. The survey also found that only about 12% of 13 year old males, 14% of 14 year old males, and 15% of 15 year old males received all 3 doses of the recommended HPV vaccine.
Typically, for minors less than 18 years old, health care decisions are made by someone else on their behalf, usually by their parents or a legal guardian (Macina, 2007;Zawistowski & Frader, 2003). Despite an increase in clinician recommendations to parents of adolescents for HPV vaccines Vadaparampil et al., 2014;Hensley, 2012), many reasons have influenced parental decisions against this vaccination (CDC, 2014c;Trim, Nagji, Elit, & Roy, 2012;Gerend, Weibley, & Bland, 2009). For example, parents may believe that their adolescent daughter is too young for the vaccine (Waller, Marlow, & Wardle, 2006;Darden et al. 2013;Jacobson et al, 2013), that the vaccine is not necessary for either boys or girls (CDC, 2014c;Grabiel et al. 2013;Jacobson et al., 2013), or that the vaccination may encourage early sexual behavior (Luedtke, 2008). Other research reveals that parents may be misinformed about HPV and HPV vaccine side effects (CDC, 2014c;Grabiel et al. 2013;Trim et al., 2012;Lai, Tinker, & Cheung, 2013;Rambout, Tashkandi, Hopkins, & Tricco, 2014). Considering the slow uptake of HPV vaccines among both female and male adolescents, and misinformation or concerns associated with the vaccines, health communication researchers need to address information concerns regarding HPV vaccination among parents of adolescents (U.S. Department of Health and Human Services, 2012). Over 1,000 studies have documented problems consumers have understanding health information. Generally these problems occur because the information provided greatly exceeds the comprehension of the average U.S. adult (Neuhauser & Paul, 2011). As such, usability testing-a method of engaging users in a website's design and testing-is essential in order to make risk communication more understandable.
With funding from the National Institute on Allergies and Infectious Diseases, our research team developed an interactive website to be used by both parents of 11-13 year old girls and their daughters to help them make an informed decision about HPV vaccines. At the time the project was funded, the use of Gardasil® had not yet been approved for adolescent males so the focus was placed on adolescent girls and their parents. The process of developing the HPV vaccination website involved conducting separate focus groups with parents of girls and their 11-13 year old daughters to understand their information needs and preferences regarding HPV vaccine information. The results were used to develop the content in the theoretically-guided HPV vaccination website, GoHealthyGirls, which was designed to help parents of adolescent females and their daughters make an informed decision about receiving HPV vaccinations (Nodulman et al., 2014). The project website was then subjected to usability testing with adolescents aged 11-13 and parents of girls aged 11-13 to determine how well the website functioned for members of the target population. In this article, the results of usability testing with these two groups are reported and discussed.

Website Overview
The website contains four main modules. "Get Answers!", discusses parent's questions and concerns about HPV and HPV vaccines, including risks/benefits, and social and health organizations that recommend HPV vaccines. The second module, "Let's Talk!," addresses ways for parents to communicate with their daughter, how to talk with other family members, and advice on how to talk with their physician about HPV vaccination. The third module, "Info for Teens", was designed specifically for adolescent users and contains an interactive quiz show game regarding HPV and HPV vaccines, a texting simulation, and a Frequently Asked Question area specifically for adolescent users. The last module, "Preteen Vaccine", contains an adolescent vaccination schedule, a section where users can put in their zip code to find clinics offering HPV vaccines in their area, and an email reminder system to remind parents of HPV vaccination follow-up appointments. Additional description of GoHealthyGirls may be found in Starling et al., 2014a.

Procedures
The purpose of usability testing the GoHealthyGirls website was to gather opinions and user feedback about the website from members of the target group of decision makers about HPV vaccination: parents of 11-13 year old adolescents. GoHealthyGirls was developed primarily for parents since research shows they generally make health decisions for their children (Macina, 2007;Zawistowski et al., 2003), but includes information targeted to adolescent females; therefore, we included usability testing with 11-13 year old adolescent females as well. Further, we included adolescent males in the usability testing to gain their insight into how a similar website might function for them as well, given that Gardasil® was recently approved for boys. Prior to conducting any research activities, approval was obtained from the University of New Mexico Institutional Review Board (IRB) and informed consent was obtained from adults and informed assent from adolescents. Eligible parents did not have to enroll in the study with their eligible child in order to participate. Participants were a convenience sample recruited through program contacts in New Mexico and Colorado and through a middle school outside of a metropolitan area in New Mexico. Testing was conducted individually between a trained member of the research staff and a participant in Albuquerque, NM and Golden, CO in late 2012.
Usability testing was conducted via 1) protocol analysis (Zimmerman, Akerelrea, Buller, Hau, & LeBlanc, 2003), 2) a self-report on a system usability scale (Bangor, Kortumb, & Miller, 2008;Bangor, Joseph, Sweeney-Dillon, Stettler, & Pratt, 2013), followed by 3) survey questions on user satisfaction. Participants met individually with a trained member of the research project to complete the testing which generally ran for 45 minutes to an hour. Parents were asked two screening questions prior to participating: "Do you have a daughter between the ages of 11-13?" and "Has your daughter ever received any dose of an HPV vaccine?" Only those who responded yes to the first question and no to the second question were allowed to participate. Participating adolescents were asked two screening question prior to participating. They were asked, "Are you between the ages of 11-13?" and "Have you ever received a dose of an HPV vaccine?" Adolescent participants that were between the ages of 11-13 and had not received any HPV vaccine were eligible for participation in the study. Both adult and adolescent participants received a $30 gift certificate for their participation in the study.

Protocol Analysis
To begin testing, the research team member met individually with each participant and explained the HPV informational website project and showed the participant the website homepage. Participants were then provided with 12 scenarios-worded for adults and again for adolescents-related to finding information on the website; e.g., for the parent, "where on the website would you go to find out if your child can get the vaccine?" or for the adolescent, "where on the website would you go to find out if you can get the vaccine?" Participants were encouraged to talk out loud while they searched for the answer to the prompts. The research members wrote down participants' statements, reactions, and how s/he navigated through the website for each of the prompted scenarios. See Table 1 for the list of scenarios.

System Usability Scale and User Satisfaction
Developed by Brooke (1996), the system usability scale (SUS) provides a reliable assessment of subjective usability of various products and services (Bangor et al., 2008;. Containing only 10 items, it is a very easy scale to administer and can be used on small sample sizes to differentiate between usable and unusable systems (U.S. Department of Health and Human Services, 2014b). Questions regarding user satisfaction with the website were developed by the research team members and attached to the SUS.

Measures
Protocol analysis scenarios were developed by trained research staff by adapting usability testing scenarios used in previous projects (Buller et al., 2013;Starling et al., 2014b). The research staff member would count how many sections of the website participants went to in order to complete each scenario. Upon completing protocol analysis testing, the SUS was administered immediately to participants. Participants answered the 10-item SUS, using Likert-type response categories (1=strongly agree to 5=strongly disagree).The SUS has repeatedly been demonstrated as a reliable and valid instrument (U.S. Department of Health and Human Services, 2014b).
In addition to the SUS, participants also provided information regarding their satisfaction with the website. The questions consisted of a 7-point Likert-type question on the userfriendliness of the product (1=worst imaginable to 7=best imaginable) followed by open ended questions regarding what each participant liked most/least about the website, what they learned from the website, and whether the website influenced their opinion about HPV vaccines. The qualitative questions were developed by the research staff for the project by adapting questions used in other website usability testing (Buller et al., 2013;Starling et al., 2014b). Data from the SUS and quantitative portion of the user-satisfaction items were then entered into SPSS statistics software version 19. Information from protocol analysis, SUS, and user-satisfaction questions were used to further refine GoHealthyGirls prior to conducting a beta-test of the website.

Statistical Analysis
Data were analyzed employing SPSS version 19. Basic descriptive statistics of participant's efforts during the quantitative portion of GoHealthyGirls website usability testing were calculated.

Participants
Nineteen people (n=19, 79% female, 21% male) participated in usability testing of GoHealthyGirls; nine parents of 11-13 year old adolescents (n=9) and ten 11-13 year old adolescents (n=10, mean age=12.7). Approximately 79% of the participants self-identified as Anglo, 10%, as Native American, and 10% considered themselves Hispanic or Latino. Specific ages of adult participants were not recorded. Table 2 reflects the ethnicity and gender of participants.
Parents were asked how long they had been using the Internet to which all replied "more than 5 years." They were also asked how comfortable they were using websites, to which all replied "very comfortable." Age and socioeconomic status of parents was not recorded. Among adolescent participants, five participants (50%) reported they had been using the Internet 1-5 years, whereas the other participants reported they had been using it more than 5 years. Six participants (60%) reported that they were "very comfortable" using websites while 4 participants (40%) reported that they were "somewhat comfortable."

Protocol Analysis
Protocol analysis testing, a test to see how participants navigated through the website to find specific content, took approximately 45 minutes for each participant to complete. All parents and adolescents answered each scenario (Table 1), except scenario seven where one adolescent did not complete the task, and scenario 8 ("Imagine you want to talk to your child (or parent) about getting the HPV vaccine but aren't sure how to go about it. Can you find help with this on the website?"), where no adolescents, and only four adults, completed it due to the perceived similarity to scenario 6 ("Where would you go to get help discussing HPV and the vaccine with your child (or parent)?". Scenario one ("GoHealthyGirls is a website designed to help young women and their parents to get vaccinated for HPV. Knowing this, what kinds of information would you expect to find on the website?") did not ask for participants to complete a task, but rather briefly explain what information they believed would be available. Parental participants typically answered with "What the HPV vaccine is for," "Risks and benefits of vaccine," "Where to get the vaccine," and "Ages to get the vaccine." Adolescent participants answered with "What HPV is," "What the vaccine is for," and "How it works." The protocol analysis revealed that participants had little trouble finding information requested in each scenario. Adult participants demonstrated the most difficulty with scenario 4 ("Where would you go to find out more about potential side effects?"). Participants went through almost three sections (M=2.90) in order to find the information. Adult participants were able to find the requested information in other scenarios within two sections of the website. Similar to adults, the majority of adolescents found the requested information within the first two sections they examined. A few problems were uncovered, such as one adult participant discovered that she could not get back to the home page from the FAQ section and another adult participant had trouble locating descriptive pictures in a website area where one may click a picture for additional information. Table 3 shows the scenarios and the mean number of sections participants went through to complete the each requested task.

SUS
After completing the protocol analysis, participants were asked to complete the systemusability scale (SUS). Participants gave favorable evaluations of the GoHealthyGirls website on the SUS (Table 4). Specifically, both parents and adolescents felt it was easy to use, most people could learn to use it quickly, it could be used without technical support from someone else, and they were confident using it. They also perceived the functions of the website were consistent and well integrated.

User Satisfaction with the Website
After completing the SUS, participants were asked to rate the user friendliness of the website on a 7 point Likert-type scale from 1=Awful to 7=Best Imaginable. An average among all participants resulted in approximately a 6 (Excellent; adults M=5.8, adolescents M=6.0). Participants were also asked "Did the website influence your opinion about the HPV vaccine?" Overall, 68% of participants (60% of adults and 89% of adolescents) responded "yes". When asked how it had influenced them, 50% of adults stated that they were more inclined to have their daughter vaccinated after viewing GoHealthyGirls, 33% said they learned more about vaccine safety, and 40% reported learning the vaccination schedule. Among adolescents, 12% said they learned about HPV vaccine safety from the website, 62% said they learned the purpose of the vaccine, and 25% said the website influenced them to tell their friends to get the vaccine.
During the qualitative section of the user-satisfaction portion, participants recommended very few changes to the website. Adolescent participants suggested that the title of the adolescent girls section of the website be changed from "For Girls" to "Info for Teens" so that adolescents would understand that that section of the website is designed for them. Adolescents also stated that sound effects in the website were louder than voice levels of the actors in the website. Adult participants suggested that the Frequently Asked Questions (FAQs) page should include hyperlinks that direct users to additional information within relative areas of the website. They also suggested that the "What is HPV?" hyperlink should be established as "Get Answers" in order to help users better understand that they could find a variety of information about HPV and HPV vaccines in that section.
The results of the usability testing were summarized and discussed by the members of the research team. The research team agreed with the changes requested by the participants and these changes were delivered to the project web-development team. The suggested changes were incorporated into the website by the web-developers, and the research team verified the changes had been successfully made. The information gained from usability testing was used in order to refine the website prior to initiating beta-testing and a later project efficacy trial.

Discussion
Usability testing of GoHealthyGirls found many compelling results. Testing was divided into multiple parts. First, participants completed a protocol analysis which consisted of completing hypothetical scenarios on the website. Next, they completed a system-usability scale for the website. Last, they answered a few brief open and closed-ended questions regarding their satisfaction with the website and influence it had on them. Protocol analysis revealed that participants were mostly able to easily find the requested information with very few problems reported. Overall, adolescent participants were able to find the requested information from scenarios quicker than the adult participants.
The protocol analysis portion of the usability testing of GoHealthyGirls was significant because it demonstrated areas of the website that participants could easily navigate to and those that they had more difficulty finding. Due to the protocol analysis, our research team was able to find what areas proved difficult to participants, and we were then able to make the necessary changes to better the site. It was also interesting that adolescents were able to find information quicker than adults. All of the adult participants reported that they were "very comfortable" using the Internet whereas 40% of teens reported being only "comfortable" using the Internet; however, teens found the information faster. This may be due to parents having more cognitive complexity, therefore, thinking there could be many different options to solve a task, whereas adolescents went with the first option they encountered.
Results from the system-usability scale (SUS) yielded positive evaluations-ease and confidence of use, ability to learn information quickly, consistent and integrated website design-from participants. These findings add to previous research (Atkinson et al., 2009;Pak, Prince, & Thatcher, 2009) that highlight the importance of testing health webpages prior to implementation to make sure they meet the needs of the consumer. By administering the open and closed-ended user satisfaction questionnaire, we were able to supplement the favorable results from the SUS with more specific information about users' opinions of the website. For example, participants' indicated that the website positively influenced their opinion of HPV vaccines. Most participants stated that they learned that HPV vaccines can help prevent genital warts and cervical cancer, in addition to learning the HPV vaccination schedule. Many others stated they learned more about HPV itself and what it does.
The goal of usability testing is to learn both the positive aspects to the design of a site, but also uncover any challenges users faced and learn how participants would recommend changing the site. Participants recommended a few changes to the website. Adolescents suggested changing the title of the section for them and smoothing out the sound levels in another section. Adults suggested adding hyperlinks in the FAQs so that additional information could be easily obtained. They also suggested changing the title of one section. As there were few suggestions for changing the website from participants, making the changes was relatively easy.
The study confirmed the research by Neuhauser et al. (2011) in using usability testing with targeted consumers in designing and modifying health information websites. Many other health information websites that have conducted usability testing confirm the usefulness of this tactic (Hinchliffe & Mummery, 2008;Korn, Murray, Morrison, Reynolds, & Skinner, 2006;Kushniruk et al., 2014;Johnson & Shaw, 2012;Choi & Bakken, 2006). Failure to incorporate usability testing with members of the targeted demographic prior to launching can result users feeling lost when navigating the website (Carusi & Alvao, 2012;de Matos et al., 2013).

Limitations
Although results appeared positive, study limitations should be considered. One significant limitation is sample size, which is often the case with usability testing since it is timeintensive and employs qualitative and quantitative methods. The small sample also limits generalizability, as did lack of diversity in the sample. Participants were selected based on convenience sampling which did not provide for a highly diverse ethnic and/or racial group of participants. Lastly, a possible social desirability effect may have occurred during testing. As usability testing was conducted individually with participants and a research team member, the possibility exists that the participant wanted to respond in a positive manner to the SUS and user-satisfaction questionnaire as it was completed in front of the research team member.

Suggestions for Future Research
The GoHealthyGirls website is now ready for evaluation and the efficacy trial has been launched. We propose to conduct a randomized controlled trial of 550 New Mexico parents of adolescent girls who have not received a dose of HPV vaccine. Half of the parents will receive the website and the other half will receive an electronic PDF about HPV and HPV vaccines. As our trial specifically targets parents of adolescent girls, future research could include using technological methods to address HPV vaccine informed decision making among parents of adolescent males and their sons. As technology becomes more mobile (i.e., smartphones, tablets/iPads), additional research could include using advanced technology to target both adolescent boys and girls, in addition to their parents.
The information gleaned from usability testing of GoHealthyGirls has been incorporated into the website for the upcoming project efficacy trial. The project efficacy trial will examine the impact of the website on actual uptake of HPV vaccines among participants. An opportunity exists to use technology to help parents make informed decisions regarding HPV vaccines for their adolescent children, and this study provides an important glimpse into how researchers and practitioners can test the usability of such technology to better deliver health communication efforts in a mediated context. Starling et al. Page 14 Table 2 Participant Demographics Online J Commun Media Technol. Author manuscript; available in PMC 2015 November 20.