Minimizing bias and mitigating risks to validity
Earlier this month we attended the 4th Annual Human Factors Excellence for Medical Device Design. We had a great time connecting with clients and colleagues (and gave away some awesome cooler backpacks). We also enjoyed the conference topics ranging from tactical examples of applying human factors engineering to the development of medical devices, to overcoming common pitfalls, and even best practices to institutionalize human factors and design thinking within an organization.
While many discussions resonated with me, there was one that I wanted to amplify – Strengthening Strategies for User Research through Consideration of User Groups (Tina Rees, Associate Director – Human Factors, Ferring Pharmaceuticals). In this presentation, Tina raised a number of reasons why more attention should be paid to who is recruited for user research and how those participants are recruited. Among those reasons, two stood out to me the most:
- The need to consider alternative recruiting methods for very specific or difficult to reach populations.
- The need to at least acknowledge, if not mitigate, the limitations sometimes associated with conducting user research exclusively during business hours.
Alternative recruiting methods
Increasingly, we are conducting research with populations who are difficult to identify or reach, and for whom special consideration is necessary. As research with these populations has become more common for us over the years, we have had to adapt and forge relationships with different types of patient support groups and clinical treatment centers to maintain access to the “right” kind of research participants.
Historically, and outside of the healthcare and medical device human factors space, relying solely on “traditional” market research recruiting firms to source research participants has been sufficient to find various types of end users. However, as healthcare-related products and devices are increasingly targeted to very specific use cases, rare disease patient populations, and populations with perceptual, cognitive, or physical limitations, the limits of traditional market research participant databases are tested. Forming partnerships directly with clinical sites has, in our experience, been the best approach to overcome the challenge of accessing these difficult-to-access populations. What’s more, these partnerships have provided us with access to clinicians who can offer the support needed by certain patient populations throughout the research process.
This is not to say that these partnerships can’t be formed between traditional market research recruiting firms and clinical / patient support groups – and there are some firms out there who have specialized in recruiting various patient and clinician populations – but the astute human factors practitioner will ensure that whenever the risk of not finding enough of the “right” kind of research participant is high, the appropriate relationships are in place to facilitate access.
Conducting user research during business hours
Most of the time, when we conduct user research we are doing so during normal business hours (from 9a-5p, Monday – Friday). This is usually practical, and in some cases, it is absolutely necessary (e.g., ethnography or contextual inquiry conducted in the field when end-users use a product during the course of their normal workday). However, it does make for a very clear selection bias when conducting lab-based simulated-use research. If you only plan to conduct this research during normal business hours, you are effectively adding “has no conflict during business hours (i.e., potentially unemployed), or is willing to take multiple hours off from work to participate in research” to your inclusion criteria. (Disclaimer – I am a consultant. My teams are the ones regularly tasked with planning, management, and execution of human factors and user experience research, and we do not want to be regularly conducting research on evenings and weekends either.)
I am not advocating for a wholesale shift to conducting research outside of business hours. That is neither practical or necessary. What I am advocating for is to consider the impact that this selection bias may have on the external validity of research data. In many cases, the impact may be negligible. There may not be any anticipated differences in use-related behavior for those with more flexible work schedules and those not likely to participate during working hours. In other cases, the impact could potentially be significant. If the product for which you are conducting research is intended to be used by (e.g., among others) practicing Rheumatologists – what risk does limiting participation to normal business hours pose, and how can that risk be mitigated?
Some key factors typically within our control when we plan research include: the days of the week, hours of a day, and location in which we conduct the research, as well as how we incentivize the end users to participate. Depending on the sample size required for research, offering a few time slots earlier or later in the day during the normal work week may be sufficient to mitigate selection bias and improve the external validity of your data. Particularly if you plan to conduct the research at a site nearby and convenient to a clinical site with ample Rheumatologists (for example). Further, consider the incentive you offer participants, and whether it is enough to encourage participation from those typically reluctant to take time off from work.
A note on bias and validity
In collecting these thoughts on alternative recruiting methods and limiting research to business hours, the concepts of bias and validity are recurrent throughout. Particularly selection bias and external validity. It is worthwhile to consider the potential limitations of human factors research for medical devices as it is typically conducted. When one considers all the possible biases and threats to validity inherent in research with human subjects, these things can start to add up (Image below).
I make this point to discourage cutting too many corners when stakes are high – for example, in a human factors validation study. There are many variables at play when a validation study is planned, and sadly (but understandably in some cases) robust research design is not always deemed the most important. When, in the name of budgets and timelines, we sacrifice adequate representation of distinct user groups, a logistical plan that minimizes selection bias, or sufficient investment in realistic use scenarios – we introduce bias and detract from the validity of any conclusions we draw. Those who understand where the “minimum of 15 representative users from each distinct user group” requirement comes from know that from a statistical reliability perspective that really is the bare minimum to be confident you have provided ample opportunity for major use-related issues to present themselves for each use case, assuming robust research design. If, on top of going with the bare minimum from a sample size perspective for validation, you also “economize” your research design in one or more of the above ways, you are increasing the chances that your conclusions will be based on invalid data.
Bottom line – be as intentional about the logistical planning for your research as you are about the development of research materials and planned analyses because these logistics can have just as much impact on the validity of your conclusions!