Vanessa Karkoutly | Design.Research.Strategy.

Considering the Ethical Implications of Using AI in User Experience Research

December 2025

Introduction

Artificial intelligence is rapidly transforming how user experience research (UXR) is practiced, and while this integration offers real efficiency gains, it introduces ethical complexities that the field is only beginning to consider.

In December 2025, I wrote an academic paper that identified the ethical principles that are most at risk when using AI in UXR, considered how practitioners of UXR might mitigate those risks and/or reduce possible instances of harm resulting from AI use, and contributed to defining what it means to be an ethical research practitioner in an AI-driven environment. This page is a summary of this paper.

TL;DR: UXR is inherently human-centered, and

being an ethical UXR practitioner requires preserving the humanity in the process.

Principles and Guidelines for Ethical UXR

UXR occupies a grey area of ethical oversight. Because most UXR objectives fall outside the regulatory definition of "research," studies are generally exempt from formal institutional review. As a result, the responsibility for defining and upholding ethical standards rests largely with individual practitioners and organizations.

Despite variation in definition, ethical UXR practice generally organizes around two themes:

Protecting research participants
Ensuring quality research

Participant protections include establishing safe and inclusive research environments, collecting informed and ongoing consent, and maintaining data privacy. Quality research requires objectivity, such as impartial data collection free from bias, and fair representation, meaning findings are reported accurately and without selective omission.

Common Use Cases of AI in UXR

Currently, UXR practitioners are leveraging tools like Microsoft Copilot, ChatGPT, Maze, Dovetail, and Looppanel to streamline tasks across the research lifecycle. Tools with supervised machine learning capabilities are being used to support thematic coding, sentiment tracking, and quantifying known issues. Tools with unsupervised machine learning capabilities can enable exploratory analysis like affinity mapping and persona discovery, and tools with transformer models are being used to conduct literature reviews, summarize transcripts, and draft research documentation. Each capability carries distinct ethical implications.

Ethical Challenges in Using AI in UXR

Implications to "protecting research participants"

Safety: Many AI tools used in UXR are not fully accessible, potentially excluding participants who rely on assistive technologies. AI-moderated synthetic interview tools can also cause participant distress through the absence of genuine human interaction and rigid conversational constraints.
Consent: The "black box" nature of many AI systems obstructs the transparency required for informed consent. Participants may not fully understand how their data is collected, stored, or used, and advanced algorithms can infer personal attributes beyond what participants knowingly disclose.
Data Privacy: AI models can retain sensitive participant data for training purposes without clear disclosure, complicating both privacy protections and legal accountability.

Implications to "ensuring quality research"

Objectivity: Supervised models trained on biased datasets can produce discriminatory outcomes, known as algorithmic bias. Unsupervised models can similarly perpetuate discriminatory patterns in raw data. Additional threats include selective amplification, which can skew literature reviews and user opinion data, inconsistent outputs driven by differences in training and prompt interpretation, and hallucinations, in which AI generates inaccurate or fabricated information.
Fair Representation: These objectivity challenges compound into issues of fair representation. AI also lacks the cultural and situational context critical to interpreting human behavior, resulting in surface-level analysis and incomplete findings.

Corporate, Organizational, and Legislative Responses

Organizations are responding to these ethical challenges by forming cross-disciplinary governance teams to establish responsible AI use guidelines. One emerging framework, the H.E.A.R.T. model developed by Kaleb Loosbrock, frames AI use as a partnered "dance" in which the UXR practitioner leads, ensuring AI remains human-centered, amplifying rather than replacing researcher expertise, and transparent in practice.

Legislatively, California, Colorado, Texas, and Utah have enacted AI governance laws aimed at protecting consumer rights to data privacy, fairness, and transparency. While these laws address some UXR-specific ethical challenges, significant regulatory gaps remain, and the everyday responsibility of ethical practice continues to fall on individual practitioners and organizations.

Ethical Considerations and Recommendations

Considerations for "protecting research participants"

Safety: Before conducting research, practitioners should complete a risk assessment, ideally involving cross-disciplinary experts, and evaluate the accessibility of any AI tools or interfaces participants will interact with directly.
Consent: Consent forms, policies, and outreach communications should be updated to reflect AI use, drafted in plain language, and reviewed by legal counsel. Practitioners should also fully educate themselves about their AI tools in order to meaningfully inform participants.
Data Privacy: Tools should be evaluated against established procurement frameworks before adoption. Practitioners should practice data minimization and anonymize or redact personally identifiable information before inputting data into AI systems.

Considerations for "ensuring quality research"

Objectivity: AI tools should be regularly tested for bias and anomalies, and outputs should always be individually reviewed for accuracy. Requesting citations or timestamps within prompts can help surface inaccuracies. Peer review at critical stages of the research process is strongly recommended as well.
Fair Representation: Human oversight must be preserved throughout use, and practitioners should pair AI outputs with other empirical findings for triangulation and bring their own contextual knowledge to the interpretation of results. The researcher, not the AI, should always be the final arbitrator of truth.

Conclusion

UXR practitioners widely acknowledge that AI's greatest value lies in automating manual process tasks, freeing time for higher-order strategic work. However, AI is not a replacement for human expertise, and successful integration demands continuous oversight, methodological rigor, and ethical governance.

The deeper risk is overreliance. Research confirms that cognitive offloading to AI can diminish critical thinking over time, and when practitioners defer to AI outputs uncritically, what one expert calls "magic-8-ball thinking," they risk surrendering the agency necessary to maintain ethical oversight. When AI handles the full arc of a research process, participants risk being reduced to data points, and the human-centeredness that defines UXR disappears.

Being an ethical practitioner of UXR requires more than managing tools responsibly. It requires actively preserving humanity in the research process, continuously evaluating whether AI is enhancing the work, or quietly replacing what makes that work meaningful.

References

Belenguer, L. (2022). AI Bias: Exploring discriminatory algorithmic decision-making models and the application of possible machine-centric solutions adapted from the pharmaceutical industry. AI and Ethics, 2(2). https://doi.org/10.1007/s43681-022-00138-8

Botero, D., & Zweifel-Keegan, C. (2024). US State AI Governance Legislation Tracker. Iapp.org. https://iapp.org/resources/article/us-state-ai-governance-legislation-tracker/

Foley, A., & Melese, F. (2025). Disabling AI: power, exclusion, and disability. British Journal of Sociology of Education, 1–22. https://doi.org/10.1080/01425692.2025.2519482

Mankoff, J., Kasnitz, D., & Camp, L. J. (2024). AI Must Be Anti-Ableist and Accessible. Acm.org, 67(12). https://doi.org/10.1145//3662731

Mortensen, D. (2019, March 19). Conducting Ethical User Research. The Interaction Design Foundation. https://www.interaction-design.org/literature/article/conducting-ethical-user-research

User Interviews. (2025). The State of User Research 2025. https://www.userinterviews.com/state-of-user-research-report

Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. A., & Krishna, R. (2022). Explanations Can Reduce Overreliance on AI Systems During  Decision-Making. Cornell University. https://doi.org/10.48550/arxiv.2212.06823