So, your recruit fell apart… now what?

So, your recruit fell apart… now what?

U

BOLD INSIGHT

With the thousands of projects managed over the years, we feel like we’ve “seen it all”. While rare, in spite of the amount of preparation and planning, sometimes things don’t go as expected. Allison details tips on how to recover from a research participant recruit that falls into that category.

You have written the screener, kicked off your recruit weeks before the study, you keep in regular communication with the recruiter and things seem to be going well. You are at the Friday before your study and you only have one slot to fill. Your recruiter is confident and you are feeling good!

Imagine your dismay when testing hits and participants begin dropping like flies. The flu has taken over and those participants who are showing up are less than desired. Participants lied about their background and aren’t eligible to participate. They can’t find a parking space so decided to just not come. All these unanticipated events can take a seemingly ‘easy’ recruit and turn it into a bit of a challenge.

What do you do when you find yourself short on participants?

Over recruit and backups

Okay, so this is a preventative option. Over recruit when possible to avoid coming up short. If you need to have 15 completed sessions, schedule at least 18 participants. Recruiters local to the area often have an idea of how many participants you will need to recruit in order to fill your quota.

On a recent study in Atlanta, the recruiter recommended we schedule 19 participants in order to get 15 completed sessions. Due to the location of the facility, traffic, and parking we experienced a high no-show rate and certainly would have missed our quota had we not over-recruited. Luckily, we were prepared and ended the study on target. Trust your recruiter’s expertise!

When filling the recruit, you could also consider scheduling backup participants, typically referred to as ‘floaters’, that come to the testing facility but do not partake in the session unless there is a no-show, or another participant is ineligible. This may be more difficult for populations that require a higher compensation, such as surgeons, but paying for an few extra participants could save you a lot of time and money in the long run if you end up not filling your quota.

Keep in constant communication with the recruiter

This may seem obvious, but your first step when you notice participants not showing up or filling requirements should be to keep in regular contact with the recruitment company. As recruiters try to fill the empty slots, ask for consistent updates. Try to get to the root of the issue to prevent it from happening in future studies.

Take a closer look at requirements

Now that you are in a bit of a bind, it may be beneficial to loosen some of the screening requirements. Consider lowering your age range or modifying any requirements on demographics, if possible.

On a recent study this approach proved invaluable at filling a recruit that had otherwise stalled. I’d been recruiting users for a new home comfort app, but the study had very narrow demographic criteria and was targeted exclusively at iPhone users. I knew that if we were able to open the recruit up to Android users, we would easily be able to meet our quota. I decided to have a follow-up conversation with my client to better understand why we couldn’t recruit Android users. During our discussion he disclosed that the app would be released on Android shortly following the iPhone release and agreed to open the recruit to Android users which solved our problem.

Engage new resources

There is no shame in reaching out to more recruiters if you are finding the original company just isn’t getting the job done. Engage new resources as early as possible and let them know that you might need some help filling the slots. You can likely pay the recruiter per each session they fill.

If appropriate, ask participants if they know anyone that would be interested in participating in the study. Earlier this year, I was working on a study that targeted specialty pharmacists and pharmacy technicians. During the week of fieldwork, participants started dropping due to conflicts. My client originally asked that we only recruit a certain number of participants from the same workplace, but once we found ourselves in a bind, they released that requirement. After a session, when escorting participants from the testing room, we asked them to reach out to colleagues to see if they would be interested in participating in the study. By the end of the week, we reached our quota thanks in part to their referrals. Just remember to remind participants to keep the contents of the study confidential if they signed an NDA.

Coworkers may also be able to cast a net to friends or acquaintances to see if anyone qualifies to participate in the study. Reach out to them to see if they can lend a hand!

Extend testing hours

Testing often occurs during regular working hours of 9-5. If participants are being recruited on short notice, these hours may not work for them. Consider adding in evening sessions to create more opportunities for participation.

Increase compensation

A little more money can make a world of difference. Give participants incentive to take part in the study!

 

Remember, occasionally losing a few participants is an unavoidable part of doing user research: what matters most is how we respond to such situations. Recognition of what is happening along with rapid corrective action can pull a study back from the brink and help you fulfill your research goals!

Recruiting methods and study logistics for human factors and user research

Recruiting methods and study logistics for human factors and user research

U

BOLD INSIGHT

A stronger recruiting strategy that includes relationships with patient support groups and clinical treatment centers can provide better access to difficult-to-reach patient populations. Being intentional about how you plan the logistics of your human factors and user research can mitigate risks to validity introduced by biases.

Minimizing bias and mitigating risks to validity

Earlier this month we attended the 4th Annual Human Factors Excellence for Medical Device Design. We had a great time connecting with clients and colleagues (and gave away some awesome cooler backpacks). We also enjoyed the conference topics ranging from tactical examples of applying human factors engineering to the development of medical devices, to overcoming common pitfalls, and even best practices to institutionalize human factors and design thinking within an organization.

While many discussions resonated with me, there was one that I wanted to amplify – Strengthening Strategies for User Research through Consideration of User Groups (Tina Rees, Associate Director – Human Factors, Ferring Pharmaceuticals). In this presentation, Tina raised a number of reasons why more attention should be paid to who is recruited for user research and how those participants are recruited. Among those reasons, two stood out to me the most:

  1. The need to consider alternative recruiting methods for very specific or difficult to reach populations.
  2. The need to at least acknowledge, if not mitigate, the limitations sometimes associated with conducting user research exclusively during business hours.

Alternative recruiting methods

Increasingly, we are conducting research with populations who are difficult to identify or reach, and for whom special consideration is necessary. As research with these populations has become more common for us over the years, we have had to adapt and forge relationships with different types of patient support groups and clinical treatment centers to maintain access to the “right” kind of research participants.

Historically, and outside of the healthcare and medical device human factors space, relying solely on “traditional” market research recruiting firms to source research participants has been sufficient to find various types of end users. However, as healthcare-related products and devices are increasingly targeted to very specific use cases, rare disease patient populations, and populations with perceptual, cognitive, or physical limitations, the limits of traditional market research participant databases are tested. Forming partnerships directly with clinical sites has, in our experience, been the best approach to overcome the challenge of accessing these difficult-to-access populations. What’s more, these partnerships have provided us with access to clinicians who can offer the support needed by certain patient populations throughout the research process.

This is not to say that these partnerships can’t be formed between traditional market research recruiting firms and clinical / patient support groups – and there are some firms out there who have specialized in recruiting various patient and clinician populations – but the astute human factors practitioner will ensure that whenever the risk of not finding enough of the “right” kind of research participant is high, the appropriate relationships are in place to facilitate access.

Conducting user research during business hours

Most of the time, when we conduct user research we are doing so during normal business hours (from 9a-5p, Monday – Friday). This is usually practical, and in some cases, it is absolutely necessary (e.g., ethnography or contextual inquiry conducted in the field when end-users use a product during the course of their normal workday). However, it does make for a very clear selection bias when conducting lab-based simulated-use research. If you only plan to conduct this research during normal business hours, you are effectively adding “has no conflict during business hours (i.e., potentially unemployed), or is willing to take multiple hours off from work to participate in research” to your inclusion criteria. (Disclaimer – I am a consultant. My teams are the ones regularly tasked with planning, management, and execution of human factors and user experience research, and we do not want to be regularly conducting research on evenings and weekends either.)

I am not advocating for a wholesale shift to conducting research outside of business hours. That is neither practical or necessary. What I am advocating for is to consider the impact that this selection bias may have on the external validity of research data. In many cases, the impact may be negligible. There may not be any anticipated differences in use-related behavior for those with more flexible work schedules and those not likely to participate during working hours. In other cases, the impact could potentially be significant. If the product for which you are conducting research is intended to be used by (e.g., among others) practicing Rheumatologists – what risk does limiting participation to normal business hours pose, and how can that risk be mitigated?

Some key factors typically within our control when we plan research include: the days of the week, hours of a day, and location in which we conduct the research, as well as how we incentivize the end users to participate. Depending on the sample size required for research, offering a few time slots earlier or later in the day during the normal work week may be sufficient to mitigate selection bias and improve the external validity of your data. Particularly if you plan to conduct the research at a site nearby and convenient to a clinical site with ample Rheumatologists (for example). Further, consider the incentive you offer participants, and whether it is enough to encourage participation from those typically reluctant to take time off from work.

A note on bias and validity

In collecting these thoughts on alternative recruiting methods and limiting research to business hours, the concepts of bias and validity are recurrent throughout. Particularly selection bias and external validity. It is worthwhile to consider the potential limitations of human factors research for medical devices as it is typically conducted. When one considers all the possible biases and threats to validity inherent in research with human subjects, these things can start to add up (Image below).

I make this point to discourage cutting too many corners when stakes are high – for example, in a human factors validation study. There are many variables at play when a validation study is planned, and sadly (but understandably in some cases) robust research design is not always deemed the most important. When, in the name of budgets and timelines, we sacrifice adequate representation of distinct user groups, a logistical plan that minimizes selection bias, or sufficient investment in realistic use scenarios – we introduce bias and detract from the validity of any conclusions we draw. Those who understand where the “minimum of 15 representative users from each distinct user group” requirement comes from know that from a statistical reliability perspective that really is the bare minimum to be confident you have provided ample opportunity for major use-related issues to present themselves for each use case, assuming robust research design. If, on top of going with the bare minimum from a sample size perspective for validation, you also “economize” your research design in one or more of the above ways, you are increasing the chances that your conclusions will be based on invalid data.

Bottom line – be as intentional about the logistical planning for your research as you are about the development of research materials and planned analyses because these logistics can have just as much impact on the validity of your conclusions!

UX project logistics: choosing the right vendors for project success

UX project logistics: choosing the right vendors for project success

U

BOLD INSIGHT

Selecting the correct vendors to support your UX project is critical to success. From facilities to recruiting, knowing the right questions to ask, budget constraints, and client needs and expectations will make the selection process smooth and painless.

When we start planning UX projects for our clients, one of the main things that we keep in mind is that one size does NOT fit all. Every client has different needs due to method, target user population, stimuli, or even personal preferences for where the research should take place. Because of these factors, we carefully think through the best logistical approach for every engagement.

Test lab flexibility

One thing we always keep in mind is to be flexible and think outside the box. While we prefer to use traditional labs that we’ve already vetted for UX research, we recently proposed a project in which a traditional UX studio might not be the ideal place to conduct the fieldwork. Instead, we decided to conduct fieldwork at a hotel because the room and amenities offered more closely aligned with the chosen methodology (the rates were also a better match for our client’s budget). Having tested in many hotels around the globe, we knew we could prepare for the challenges that may arise in that environment, so we were comfortable with that approach.

Responsive vendors

Another consideration is that the lowest-priced vendor is not always the preferred choice. We recently drafted an automotive project plan that required working with a vendor that rents out very specific vehicles with even more specific trim packages (not cars that you can easily find by calling up your local car rental agency). We contacted a few potential vendors – one of them called me immediately to discuss the details of what we needed… another didn’t reply to my email until a week later. At that point, we were ready to go with the vendor that was responsive and eager, even if they didn’t have the lowest bid. We were more comfortable knowing that this vendor was more likely to quickly resolve any potential issues to ensure a smooth project for our client.

Participant access

Finally, one of the most important vendor choices to make when setting up a UX study is who to trust to recruit study participants. The right choice of recruiter can be the difference between a successful and a not-so-successful study. Our team has years of experience working with several recruiters across the US and around the world, and in that time, we’ve come to learn that the best recruiter for finding Type 2 Diabetes patients may not be the best recruiter for finding consumers that use their mobile device to pay for gas at the pump (both of which are populations with whom I’ve done work).

In addition to relying on past experiences with recruiters, study parameters have to be considered, including time line (filling a recruit in one week vs. three weeks), sample size (n=12 is very different than n=72), and even client requirements (some clients have vendors they prefer we use). All of these factors are taken into consideration when we decide what recruiter we’re going to use for any given project.

With thousands of proposals under our belts, we always keep these things in mind (as well as some other project planning tips and tricks) to provide the best experience for our clients and participants. Just as the research stimuli and participant sample change from project to project, so should the logistical approach be flexible and adapt to fit the needs of the client. Knowing what to look for in a vendor is one way we deliver seamless, successful projects.

Bold Insight partners with watchLAB in Chicago loop to deliver exceptional recruiting and state-of-the-art research studios

Bold Insight partners with watchLAB in Chicago loop to deliver exceptional recruiting and state-of-the-art research studios

Bold Insight, a user experience (UX) and human factors (HF) research agency, and watchLAB, a leader in recruiting, facility rental and hosting, announced a partnership to better support organizations conducting user research in Chicago. This partnership provides Bold Insight’s clients access to six advanced and spacious studios and viewing rooms that have been vetted for the specialized needs of UX and HF research.

One reason companies choose to conduct user research in the Chicagoland area is the access provided to a wide variety of consumer types across both urban and suburban settings. Having recently established headquarters in the Chicago suburb of Oakbrook Terrace, Bold Insight sought a like-minded firm to provide the desirable downtown location to its clients.

“We have found over the years that establishing close partnerships with research participant recruitment and research facility providers can dramatically improve the efficiency with which we plan, manage, and execute user research on behalf of our clients,” says Korey Johnson, Managing Director of Bold Insight. “This not only allows us to offer more competitive pricing, it also allows us to more effectively target difficult-to-reach populations and thereby improve the quality of the insights we provide our clients to inform the design of exceptional user experiences.”

watchLAB has flexible spaces that offer the right mix of state-of-the-art technology and modern, comfortable design. Located in the city’s vibrant River North neighborhood, the Studio is ideally located just three blocks from Michigan Avenue.

“The diversity of Bold Insight’s work is very exciting and really allows our facility and recruiting process to shine,” says Heather McEneany, Vice President of watchLAB. “With our partnership we’re able to go deeper to learn their needs and objectives, and ensure the environment and respondents are spot-on for each project.”

 

About watchLAB

watchLAB provides recruitment, facility rental, hosting services, and project management for all qualitative research methodologies, both in Chicago and across the US. With state-of-the-art technology, flexible room configurations, and celebrated aesthetics, watchLAB fosters the ideal environment for uncovering insight. And its innovative recruiting team isn’t confined to the limits of an existing database, but rather networks and does additional outreach to find the most qualified and engaged respondents for each project. Visit watchLAB.com to learn more.

About Bold Insight

Bold Insight helps clients deliver outstanding experiences to customers by converting behavioral observation into actionable insights informing the design of products that fit seamlessly into those customers’ lives. The team has conducted research on hundreds of products and services delivered on a variety of platforms, including websites, software, mobile devices, medical devices, voice assistants, connected devices, and in-car navigation systems. Find out more at boldinsight.com.

Enough about us, let's talk about you!

We’d love to have an opportunity to bid on your project.
Responsiveness is important to us so we promise to get back to you immediately!

+1.630.317.7671

hello@boldinsight.com

Pin It on Pinterest