Building trust in medical AI: A sociotechnical approach

Authors: Nayara Güércio (Research Impact Officer, Trilateral Research), Amelia Williams (Senior Research Impact Officer, Trilateral Research), Sarah Murray (Senior Research Analyst, Trilateral Research), Fotis Basamakis (Data Scientist, Trilateral Research)

Artificial intelligence (AI) and data-driven innovation are widely touted as the next leap forward in medical technology, with the potential to improve care and relieve overburdened healthcare systems. However, even as AI innovation accelerates in research centres and tech labs all over the world, trust remains a major hurdle. Trust in medical AI has a dual nature, driven in part by technical issues such as data quality, bias, reliability, and explainability, as well as broader social factors such as professional cultures, institutional practices, patient experience, and public understanding. Evidence suggests that even highly accurate systems may struggle to gain acceptance if clinicians or patients are uncertain about how they work or what their adoption might mean in practice. Thus, technical solutions alone rarely resolve the trust deficit.

Addressing this issue requires treating responsible and trustworthy AI as a sociotechnical challenge that cannot be resolved through engineering alone. Interdisciplinary teams that integrate technical expertise with insights from ethics, law, and other social sciences are better positioned to identify risks early, design for fairness, and build systems that patients, clinicians, and other healthcare professionals can rely on. In medical contexts, where AI is increasingly used to support prediction, risk stratification, and treatment decisions, getting this right can support the delivery of more effective and efficient care. The following sections outline how this sociotechnical approach can be operationalised when creating medical AI systems, using the EU-funded iToBoS (Intelligent Total Body Scanner for Early Detection of Melanoma) project, which developed an AI-powered tool for melanoma detection, as a case study.

Identifying bias during technical development

While developing the AI model for melanoma detection, early bias assessments revealed that female patients experienced a disproportionately high false positive rate in melanoma identification, suggesting gender-related disparities in model performance. In particular, female patients were 26% more likely to be incorrectly classified as at risk of developing melanoma, highlighting an imbalance in prediction errors across patient groups.

Rather than treating this as a technical anomaly, the development team recognised it as a signal that broader sociotechnical considerations needed to inform development. A collaboration between data scientists, clinicians, and ethics and legal experts was initiated to reassess the model and improve its fairness without sacrificing clinical performance. The results showed that optimisation techniques alone do not always guarantee fairness, and that trustworthy and responsible AI development requires sustained interdisciplinary collaboration.

Putting sociotechnical collaboration into practice

Throughout the development of the AI-powered melanoma diagnostic tool, sociotechnical collaboration was operationalised through four interconnected workstreams.

Dataset pre-processing

The sociotechnical team applied pre-processing techniques to the training data, designed to mitigate bias while preserving clinically meaningful differences. Rather than simply removing sensitive attributes such as gender, age, or socio-economic indicators, which can be clinically significant and are often necessary for fairness assessment, the team iteratively adjusted the underlying training data to reduce correlations that could lead to unequal outcomes. Following the identification of gender-related disparities in false positive rates, gender representation in the dataset was adjusted, iterative re-training was conducted, and structured fairness assessment procedures were applied. This process improved the consistency of model performance across demographic groups and demonstrated how ethical considerations can inform technical decision-making.

Continuous impact assessment

A continuous Privacy Impact Assessment+ (PIA+) process structured collaboration between technical partners and social science and humanities researchers. The sociotechnical team developed a PIA+ risk table cataloguing risks and mitigation measures across five categories: Privacy and Data Protection, Autonomy, Transparency, Trust, and Clinical Effectiveness. This tool functioned as a shared reference point for joint assessment of the social, ethical, and regulatory consequences of design choices, considering the GDPR and EU AI Act alongside broader societal and ethical impacts. Iterative workshops ensured that newly identified risks fed into technical modifications and governance planning. In this way, impact assessment functioned as a sociotechnical mechanism, strengthening system design to maximise clinical benefits and minimise harm in real-world healthcare contexts.

Explainability

Explainability was treated as a core sociotechnical mechanism for building trust. Through the implementation of Concept Relevance Propagation (CRP), the team connected AI predictions to clinically meaningful concepts such as asymmetry or colour variation (dermatological criteria used in melanoma assessment), making AI outputs more interpretable and actionable for clinicians. The sociotechnical team assessed these explanations for fidelity, robustness, usability, and regulatory alignment. Explainability also supported bias detection by revealing which features drove predictions across demographic groups. Explainability became a joint output of technical and social science expertise, reinforcing a central lesson of the project: that trustworthy and responsible AI emerges when transparency is co-developed across disciplines.

Co-creation with stakeholders

Co-creation was embedded throughout the project as a structured sociotechnical process, rather than a one-off consultation. The team organised a series of ethics and emerging technology group meetings with patient advocates, stakeholder workshops and interviews on explainable AI in healthcare, and targeted discussions with clinicians on training and workflow integration. These engagements surfaced concerns about data protection, transparency, sustainability, clinician training, non-expert understandability, and accountability that would not typically emerge from technical development alone. This feedback directly influenced system design, documentation practices, and explainability features, ensuring that the tool aligned with both clinician and patient needs.

Ethics as an enabler of innovation

Ethical and social requirements are sometimes framed as obstacles that slow technological progress. The development process behind the AI-powered melanoma detection tool suggests the opposite. Integrating sociotechnical perspectives strengthened development by helping the team identify risks earlier and adapt design decisions before they became costly issues at deployment or barriers to uptake by future users. Embedding ethics into the process from the outset, rather than outsourcing it to external review, meant that responsible AI was developed by design. This is a critical distinction: across domains, responsible-by-design AI systems are more robust, trusted, and better positioned for real-world adoption than those subjected to ethics review after the fact.

From development to deployment: key lessons

Real-world medical AI development demonstrates that technical performance alone does not guarantee the adoption of AI tools in real-world healthcare settings. Trustworthy med tech emerges from sociotechnical collaboration that connects technical design with professional practice, governance requirements, and patient expectations. This experience offers three broader lessons for the field.

First, sociotechnical collaboration can be integrated into development without disrupting existing workflows. When interdisciplinary expertise is present from the beginning, it guides design choices rather than appearing as a corrective measure later, strengthening the overall results.

Second, collaboration across technical and social domains allows teams to recognise emerging risks sooner. Questions related to bias, governance, and usability become visible during development rather than after deployment.

Third, sociotechnical approaches support the transition of medical technology from research settings into clinical practice. AI tools that clinicians and patients perceive as understandable, fair, and transparent are more likely to be adopted and maintained over time.

From research to real-world use

This sociotechnical approach is not limited to the medical domain. It reflects a broader way of developing technology, where interdisciplinary collaboration between technical experts, social scientists, legal scholars, practitioners, and users is integrated from the earliest stages of innovation. As medical AI continues to evolve, adoption will increasingly depend on the ability of development teams to bridge technical and social science expertise, ensuring that tools support clinical decision-making while respecting patient expectations and institutional responsibilities.

The practices described in this article draw on Trilateral Research’s experience in the iToBoS project, where our sociotechnical team worked alongside clinicians, engineers, and patient representatives to integrate ethical, legal, and social perspectives throughout the development of an AI-powered melanoma diagnostic tool. Experiences from this collaboration reinforce a broader lesson: sociotechnical collaborations that incorporate ethical, legal, and clinical perspectives alongside technical design produce AI systems that are not only effective, but also interpretable, governable, and deployable in real-world healthcare settings. Responsible, trustworthy AI is not an afterthought, it is the result of an interdisciplinary, collaborative process.

—

We would like to acknowledge the other Trilateral Research team members who contributed to the iToBoS project, including Niamh Aspell, Zita McCrea, Robin Renwick, Padraic Fleming, Ilaria Bonavita, Elahe Naserianhanzaei, and Juan Fernando Castro Orozco, as well as iToBoS partners from the National Technical University of Athens (NTUA) who collaborated with us on technical work. Thanks also to Padraic Fleming and Diana Ramirez for reviewing.