If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
The Covid pandemic brought the need for more realistic remote consultations into focus. 2D telemedicine solutions fail to replicate the fluency or authenticity of in-person consultations. This research reports on an international collaboration on the participatory development and first validated clinical use of a novel, real-time 360-degree 3D Telemedicine system worldwide. Development of the system - leveraging Microsoft's HoloportationTM communication technology – commenced at Canniesburn Plastic Surgery Unit, Glasgow in March 2020.
Methods
Research followed VR CORE guidelines on development of Digital Health trials, placing patients at the heart of the development process. This consisted of three separate studies - a clinician feedback study (23 clinicians, Nov-Dec 2020), a patient feedback study (26 patients, Jul-Oct 2021), and a cohort study focusing on safety and reliability (40 patients, Oct 2021 - Mar 2022). “Lose, Keep and Change” feedback prompts were used to engage patients in the development process and guide incremental improvements.
Results
Participatory testing demonstrated improved patient metrics with 3D in comparison to 2D Telemedicine, including validated measures of satisfaction (p<0.0001), realism or ‘presence’ (Single Item Presence scale, p<0.0001), and quality (Telehealth Usability Questionnaire, p=0.0002). Safety and clinical concordance (95%) of 3D Telemedicine with a face-to-face consultation were equivalent or exceeded estimates for 2D Telemedicine.
Conclusions
One of the ultimate goals of telemedicine is for the quality of remote consultations to get closer to the experience of face-to-face consultations. These data provide the first evidence that HoloportationTM communication technology brings 3D telemedicine closer to this goal than a 2D equivalent.
Real-time 3D Telemedicine has previously been proposed within a research setting only, with constraints on cost, complexity, bandwidth and technology [
]. Although there appears to be significant theoretical value in assessing surgical patients, as yet no validated clinical data exists on the benefits of such a system in comparison to standard 2D Telemedicine. Here we detail the participatory development of a real-time 3D Telemedicine system, in conjunction with an international group of healthcare and industrial stakeholders. We also discuss the first real-world use of a 3D Telemedicine system with Plastic Surgery patients.
Increasing access to care in Lower to Middle Income Countries
Initial discussions regarding the development of a 3D Telemedicine system, leveraging Microsoft's Holoportation communication technology, commenced in December 2019 - between Canniesburn Plastic Surgery Unit, Glasgow, UK; Korle Bu Teaching Hospital, Accra, Ghana; and Microsoft Corporation, Redmond, USA. This centered around the potential for increasing access to specialised reconstructive surgical care in Lower to Middle Income countries (LMIC). The research team visited the Ministry of Health, Ghana, in February 2020 to initiate an international collaboration on this project. Geospatial mapping set out the early vision, using census data and overland travel times to estimate increased access to reconstructive care [
]. With the global Covid pandemic enforcing the first UK lockdown in March 2020, the focus of the project rapidly pivoted to potential delivery of remote care during Covid.
VR CORE Guidelines on developing Digital Health technologies
The 3D Telemedicine system was co-developed with patients using international guidelines for digital or Virtual Reality (VR) clinical trials, as proposed by the Virtual Reality Clinical Outcomes Research Experts (VR CORE). These guidelines aim to improve methodological quality in digital health technology trials, dividing development into three phases (VR 1 to 3), akin to Clinical Trial Phases 1 to 3 [
]. Participatory development is one of the key elements of these guidelines, focusing on the principles of human-centered design. The importance of this in digital trials has been previously underestimated, as “lack of patient involvement, poor requirement definitions, and non-adaptation to user feedback are some of the common factors that explain failures of digital interventions” [
]. The present study placed patients at the heart of the development process and reports on the Participatory development (VR 1) and early Clinical Trial (VR 2) phases. The objectives of this research were to co-develop a patient-centered 3D Telemedicine system, compare validated outcome measures with a 2D system, assess alignment with an in-person consultation, and to ensure safety, reliability and clinical concordance.
METHODS AND RESULTS
Ethics
NHS Greater Glasgow and Clyde (GGC) R and I approvals GN20HS181/ GN20HS300. NHS GGC governance meetings monitored the project bi-annually. Participants consented in writing. Patient data controlled by NHS GGC. STROBE guidelines followed for reporting of cohort trials.
] and consisted of preliminary work including focus groups, stakeholder collaborations, equality assessments, and initial prototyping. Weekly collaboration meetings were held over a 2-year period in UK, Ghana and USA, commencing March 2020. To inform prototype development - a focus group was held with 23 clinicians from Canniesburn Plastic Surgery Unit, Glasgow, UK in June 2020 - exploring desired functions, identification of needs, potential benefits, risks, use during Covid, and implementation (Supplementary Table 1). An Equality Impact Assessment (EQIA) [
], and collected data on factors that may influence access or use of novel technology, such as deprivation and educational level. This was followed by three separate studies: a clinician feedback study – providing feedback to incrementally improve and shape the 3D Telemedicine sytem prior to patient testing; a patient feedback study – to compare 3D and 2D Telemedicine systems; and a cohort study focusing on safety, reliability and clinical concordance of 3D with face-to-face examination (Flowchart 1).
Flowchart 1
Prototyping and Initial Set up
Initial system set-up took place in September 2020, when a research team from Microsoft Corporation, Redmond, USA travelled to Glasgow, UK. The system – inspired by Microsoft's HoloportationTM research [
] - consisted of an array of 10 Azure Kinect cameras connected to a Fusion server that fuses each camera's depth output to create a 3D 360-degree model, and a Render server that covers the model in RGB video output. This was linked to a “viewer” room where the patient could be viewed in 360-degrees on a computer screen over the existing hospital network (Figures 1-5, Supplementary Videos 1-3). The “viewer” room was set up in both the test site hospital (West Glasgow Ambulatory Care Hospital, WGACH) and remotely at Canniesburn Plastic Surgery Unit. The WGACH viewer site was used in the present studies (full system features - Supplementary Table 1).
Figure 13D Rig Set up. Multiple Kinect cameras surround the patient within the clinic room. In the centre is a chequerboard used to calibrate the system.
Figure 2Sensate Anterolateral Thigh (ALT) Flap to Above knee amputation (AKA). This patient required resurfacing of an area of skin grafted post-traumatic residual limb, that provided a poor interface with the prosthetic limb. A sensate ALT flap was used to resurface the residual limb.
Figure 3Patient in 3D rig. The same patient with resurfaced AKA sitting in the 3D Telemedicine system. The screen allows him to view the same images as the clinician.
Figure 4Clinician Viewer room. The clinician can see the patient in 3D on the left screen. On the right is a standard Telemedicine video call. Note the difference in field of view and ability to position patient to see the right AKA.
Patient and clinician feedback was used to shape the development of the 3D Telemedicine system, assess usability of the outcome instruments, and provide data for follow-on trials. Sample sizes were not required for the participatory development component of this study. Data analysis carried out by a blinded, independent statistical service using R version 3.4.1 [
Study 1: Clinician Feedback Study : Iterative improvements in 3D system
Clinician feedback testing focused on optimisation of the prototype system prior to patient testing. Full details are provided in Supplementary Appendix 1. In brief, 23 clinicians were assessed in two batches in November and December 2020, to provide incremental improvements in usability, and assess responsiveness to change between batches. Key findings included a strong correlation between quality of consultation (satisfaction) and the realism or “presence” of the 3D system (Pearson r = 0.8451, p<0.0001, Graph 1). The 3D system was found to have a significantly higher realism or “presence” than the 2D system in batch 2. Improvements were instituted in response to clinician feedback, prior to patient testing (Supplementary Table 1).
Graph 1Correlation of Presence with Satisfaction. Clinician Satisfaction correlation with Presence Questionnaire Score
Study 2: Patient Feedback Study: 3D versus 2D Telemedicine
Method
Patient feedback testing provided the first real-world data on patients’ perceptions of the 3D Telemedicine system in comparison to 2D Telemedicine. A preliminary cohort of six patients was used to test acceptability of the outcome instruments. Longer form scales such as the Presence Questionnaire with 29 items [
] were found to be too long and confusing for patients, and therefore short forms of “presence” testing were considered, with final selection of the validated Single Item Presence scale [
26 patients from Canniesburn Plastic Surgery Unit clinic participated (July - Oct 2021) (Supplementary Table 2). Patients were seen in 3D and 2D Telemedicine by a single clinician, without randomisation (11 patients 2D first, 15 3D first). Each consultation lasted 10 minutes, followed by completion of questionnaires.
Outcome Instruments
Single Item Presence scale asked the key question “To which extent did you feel present in the virtual clinic, as if you were really there?” to provide insights into how closely the 3D clinic aligns with an in-person consultation [
How to ask about patient satisfaction? The visual analogue scale is less vulnerable to confounding factors and ceiling effect than a symmetric Likert scale.
3D versus 2D Telemedicine validated outcome measures
3D was rated higher on satisfaction (p<0.0001), presence (p<0.0001), and quality (TUQ, p=0.0002) (Boxplot 1). Mental effort rating scale was equivalent (P=0.2965), scoring highly in both systems suggesting that the 3D system is not more complex to use than normal telemedicine systems. Patients’ subjective views on accuracy of diagnosis and ease of positioning their body part for examination were also significantly higher (Table 1). 100% patients preferred 3D (for patient comments see Supplementary Table 3) and usability was rated highly (SUS 87.02).
Table 1Patient Feedback Study – 3D versus 2D Outcome Measures. Validated outcome measures included satisfaction, Mental Effort Rating Scale, Single Item Presence Scale, System Usability Scale and Telehealth Usability Questionnaire. Non-validated outcome measures included patients’ subjective views on ability of clinician to make an accurate diagnosis, and ease of positioining their body part for examination.
“Lose, Change and Keep” prompts noted limitations related predominantly to quality of 3D spatial resolution in real-time (Supplementary Table 1). Subjective interview was overwhelmingly positive, with patients in particular finding that the 3D system allowed positioning their body part for examination much easier than with 2D (data not shown) (Supplementary Video 4).
Carry-over Effects
Between-group comparisons of satisfaction, presence and TUQ were done using a general linear model which included subject, group and order factors (3D or 2D system first, to determine any effect of carry-over). No effect of carry-over was noted for these outcome measures (data not shown).
VR PHASE 2: EARLY CLINICAL EFFICACY AND SAFETY
Study 3: Cohort Study: 3D Telemedicine versus face-to-face
Method
The cohort study aimed to assess safety and reliability. A paired design was used with patients seen in 3D Telemedicine first, followed by face-to-face, by the same clinician to allow assessment of clinical concordance. Ten minutes were given per examination. Clinicians and patients both completed questionnaires for each consultation.
Sample size
Pilot study with recruitment of 40 patients. Primary and secondary outcomes not designated for pilot studies. Post-study power calculations indicated that with a paired design, 16 patients would have been sufficient (using primary outcome with University of North Norway UNN score as primary outcome; delta 0.4; SD 0.52; power 0.9)
Participants
40 patients and 10 clinicians (5 residents, 2 nurse specialists, 1 physiotherapist, 2 consultants; mean age 37.5; range 28-43; 7F, 3M) from Canniesburn Plastic Surgery Unit participated (Oct 2021 – Mar 2022, Flowchart 2). For inclusion criteria and patient demographics – see Supplementary Tables 4 & 5.
Flowchart 2
Outcome Instruments
UNN scale, validated for orthopaedic patients, is one of the only scales available designed to assess quality of both telemedicine and face-to-face consultations [
]. NASA TLX is a measure of the workload or “task load” of an activity - assessing subdomains of mental demand, physical demand, temporal demand, performance, effort and frustration [
Hart, S.G. and L.E. Staveland, Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, in Advances in Psychology, P.A. Hancock and N. Meshkati, Editors. 1988, North-Holland. p. 139-183.
Operational safety issues occurred in 0%, technical issues 20%, reliability issues 7.5%, and image artefacts 12.5%. All issues were temporary and did not curtail the clinical consultation. Clinical concordance of 3D consultations with face-to-face management plan was 95%. 2 cases of clinical discrepancy did not result in clinical safety concerns (Supplementary Table 6).
3D versus Face-to-Face Validated Outcome measures
Clinicians’ UNN Sum score was significantly higher for a face-to-face than for 3D consultation (Table 2). Nonetheless, the difference in sum score between 3D and face-to-face was only 0.4, with both consultation types approximating to scores between “good” and “very good”. The minimal clinically important difference (MCID) has not yet been established for UNN score. The UNN authors suggested a difference of 0.3 as a non-inferiority limit in their study, but this appears to be arbitrary and not based on clinical data. Patients rated the face-to-face higher on satisfaction than 3D, mean 90.08 versus 83.58 respectively (mean difference 6.50 [CI 2.40-10.60]; p=0.0027; paired t test).
Table 2Cohort Study – 3D Telemedicine versus Face-to-Face UNN Score. The UNN score is subdivided into subdomains and co-operation, examination, treatment, informing patient and overall.
Differences in Clinician versus Patient outcome measures for 3D system
Patients rated the 3D system higher on satisfaction than clinicians (p<0.0001). This may be related to differences in mental effort (p=0.0042) and task load (NASA TLX, p=0.020). Usability (SUS) and quality scores (TUQ) were equivalent (Table 3).
Table 3Cohort study – Clinicians versus Patient outcome measures of the 3D Telemedicine system only
Clinician satisfaction, presence (PQ) and usability (SUS) significantly improved over the 2 year development process of these three studies (Graph 2).
Graph 2Improvements in outcomes during development process. Outcome measurements improved significantly during incremental feedback and development, for ratings of satisfaction, presence (PQ) and usability (SUS) over the development process. 2020 refers to scores from the Clinician Feedback Study Batch 1, 2021 refers to Clinician Feedback Study Batch 2, and 2022 to the Cohort study clinician scores. PQ is converted to a 100 point scale for this graph. 95% CI bars shown. Comparison of 2020 with 2022 scores with unpaired t-test: Satisfaction (p=0.026), PQ (p=0.021), SUS (p=0.017).
Boxplot 13D versus 2D patient outcome measures. Satisfaction, Presence (Single Item Presence Scale) and Quality (TUQ) all significantly higher for the 3D system. All scores converted to 0-100 scale.
] - and education level, did not significantly correlate with satisfaction, usability (SUS), quality (TUQ) or task load (NASA TLX) of the 3D system (Supplementary Table 7).
Harms
No harms occurred during 3D Telemedicine consultations.
Discussion
3D Telemedicine: The future of remote consultations
This is the first study to examine and demonstrate the potential benefits of 3D Telemedicine in comparison to a 2D equivalent. Previous research has either existed in a research setting only, or has not proven any validated benefits of 3D Telemedicine over existing telemedicine technology [
]. Therefore, until now, the advantages of 3D Telemedicine have been purely speculative. This is also the first clinical system to employ Microsoft's HoloportationTM communication technology - a true 360 degree 3D system that fuses multiple depth cameras - with previous research only employing single depth cameras that do not permit a true 360 degree coverage [
Placing Patients at the Heart of the Development Process
Development of the 3D system has focused on medical communication from the ground-up, and has not been adapted from business or other user groups. The VR CORE Guidelines, akin to the Model for Assessment of Telemedicine guidelines, place participatory design at the heart of the development process [
]. Clinician and patient feedback testing allowed co-development of a system that was fit to context and user group. This incremental, methodical approach directly translated to improved outcome measures, as can be seen by improvements in Clinician satisfaction from 56.6% to 70.1%, and usability (SUS) from 61 (grade D) to 80.5 (grade A) over the development cycle (Graph 2). It is hoped that in using a patient-centered development approach, that this system will be both relevant and embraced by patients in future clinical practice.
3D Telemedicine increases the realism and satisfaction of the remote consultation
Patient feedback testing suggested a strong user preference (100% preferred 3D), satisfaction (p<0.0001), and closer alignment with an “in-person” consultation for the 3D Telemedicine system in comparison to 2D. The Single Item Presence scale asking the question “To which extent did you feel present in the virtual clinic, as if you were really there?” strongly favoured the 3D system (p=0.001). The Telehealth Usability Questionnaire, an overall measure of the quality of telemedicine, favoured the 3D system (p=0.0002). Subjective interview was very positive, with patients in particular finding positioning their bodypart for examination much easier with the 3D system.
The importance of “presence” or realism in remote Clinical Consultations
Critically, this study points towards real-time 3D telemedicine bringing the remote consultation experience closer than ever towards a face-to-face consultation. This is one of the fundamental goals of telemedicine. Although patients still prefer a face-to-face consultation (higher UNN score/ satisfaction), 3D Telemedicine was rated significantly higher in terms of satisfaction and realism than 2D Telemedicine (Table 1). These study data also highlight the importance of ‘presence’ or realism of a Telemedicine system. Firstly, ‘presence’ correlates strongly with satisfaction of the clinical consultation (Graph 1). Secondly, data from other groups suggests that ‘presence’ correlates with improved human performance. The more immersive experience provided by 3D Telemedicine may therefore increase the fluency and execution of the consultation [
]. Together these provide impetus and rationale to continue research into more immersive forms of Telemedicine.
Clinical Relevance of real-time 3D consultations in Plastic Surgery
Although testing indicated improved key metrics in patient satisfaction and presence or realism, these are relatively abstract terms that are difficult to translate into direct clinical benefits for patients. These potential benefits are numerous - for example, examining patients with flap reconstructions on the side or back of the body, conducting remote physiotherapy without requiring the patient to re-position multiple times, the ability to examine multiple joints from different angles, ease of positioning the patient with limited mobility who is unable to move for the camera, and avoiding having an intermediary to move a camera around the patient. Furthermore, the ability to draw on the 3D patient model allows the clinician to more accurately explain an operation on the actual patient's body, and in doing so provides a more ‘personalised medicine’ approach. Together, these advantages allow the clinician to conduct a higher fidelity, more fluent consultation than with a 2D system.
Safety and Clinical Concordance
Clinical concordance of the 3D system with in-person consultations (95% in this study) is equivalent or exceeds previous estimates for other surgical Telemedicine consultations. These include 92% surgeons estimating that their telemedicine decisions are as good as face-to-face [
]. These data come from non-paired studies which are considerably less accurate in measuring concordance than the paired design used in the present study, due to the heterogenous nature of medical conditions.
Leave nobody behind in a digital world
The 3D system aimed to not “leave anybody behind in a digital world” and therefore the interface was made as simple as possible for patients. Technological equality and inclusion of the system was demonstrated by high usability of the system irrespective of patient deprivation or education levels (Supplementary Table 7).
Outcome measurement instruments
Given that the key goal of any Telemedicine system is to bring the remote consultation as close as possible to the experience of a face-to-face consultation, “presence” was deemed to be one of the primary research outcomes. Many scales measuring “presence” in digital environments were found to be too lengthy and confusing by patients [
]. Questionnaires that work with clinicians or university students will not necessarily translate adequately to patients, with 27% of the UK population having the lowest level of literacy [
]. The Single Item Presence question “To which extent did you feel present in the virtual clinic, as if you were really there?” was clear, simple and resonated strongly with patients [
]. This will therefore form the primary outcome for the follow-on RCT. The Telehealth Usability Questionnaire (TUQ) is a newer, validated questionnaire measuring subdomains of usefulness, ease of use, effectiveness, reliability and satisfaction, giving a metric of the overall quality of the telemedicine system [
]. Although it contains questions on generic topics that do not differentiate between technical systems, such as “Telehealth saves me travelling”, it is clear to understand and provides a number of important metrics, and will therefore be used as a secondary outcome in the follow-on trial.
Limitations, Bias and Generalisation
Inherent limitations of these data include lack of randomisation and blinding. Use of validated, generic outcome scales minimizes information bias and allows generalisation of these data to other trials employing the same instruments.
Future Directions
A follow-on VR Phase 3 Randomised trial will provide definitive evidence of clinical benefits. Concurrently, a test bed has been set up in Ghana to explore increasing access to reconstructive surgical care in LMIC. Health economics, health frameworks, and organisational change will be incorporated in future analyses. Further technical changes will focus on improving Clinician satisfaction by reducing Mental Effort and Task Load (supplementary Appendix 2). This may be aided by improving spatial processing of a 3D object through increased resolution, higher frame rate, reduced lag time, and use of more intuitive, natural interfaces such as the HoloLens [
This research details the participatory development of a 3D Telemedicine system, with the first validated comparative clinical use in patients worldwide. One of the ultimate goals of telemedicine is for the quality of remote consultations to get closer to the experience of face-to-face consultations. Together the data presented here points towards significant potential in bringing the remote consultation closer to this goal, which is of particular relevance to specialities with a strong 3D focus such as Plastic Surgery. In summary:
1)
3D Telemedicine increases the realism of the remote Clinical consultation, and is closer in experience to a face-to-face consultation than a 2D consultation.
2)
“Presence” or realism correlates with satisfaction with the consultation. This is a strong driver to invest in technologies that increase the fidelity of telemedicine consultations.
3)
3D Telemedicine appears to significantly increase patient satisfaction in comparison to a standard 2D consultation.
4)
Safety and clinical concordance are acceptable for 3D Telemedicine - equivalent or exceeding estimates for 2D Telemedicine
Funding
Medical Research Scotland CVG-1742-2020, Global Challenges Research Fund SFC1236-105, Jean Brown Bequest, NHS GGC endowments Covid Fund.
Ethical Approval
NHS Greater Glasgow and Clyde Research and Innovation board approvals GN20HS181/GN20HS300. ClinicalTrials.gov identifiers NCT05267197/NCT04444323. Designated a Covid priority study by the Health Research Authority, UK. All patient data controlled by NHS GGC.
Terminology
Presence, immersion and realism. “Presence” is an experiential, subjective and psychological quality in virtual environments, associated with feelings of “being there”. “Immersion” is associated with objective, technical aspects of virtual systems that help the user feel a sense of presence. “Realism” can be considered a dimension of “presence”, with stronger realism leading to increased presence. For the purposes of this paper, which is aimed at a clinical audience, these terms are used interchangeably.
Patient Consents
Patient consents were obtained for publication of identifiable photographs and videos
Conflict of Interest
The authors have no disclosures.
Acknowledgements
An extended research team included many others at the West of Scotland NHS Innovations Hub, UK; Korle Bu Hospital, Ghana; and Microsoft Corporation, Redmond, USA.
Presented at: BBC News 13th July 2022 and BBC Good Morning Scotland Radio 13th July 2022; American Society of Plastic Surgeons meeting, Atlanta, USA October 2021; British Association Plastic and Reconstructive Surgeons, Celtic Meeting, Dunblane September 2021; Future Surgery Show, London, November 2021.
How to ask about patient satisfaction? The visual analogue scale is less vulnerable to confounding factors and ceiling effect than a symmetric Likert scale.
Hart, S.G. and L.E. Staveland, Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, in Advances in Psychology, P.A. Hancock and N. Meshkati, Editors. 1988, North-Holland. p. 139-183.
dependent t test
00555-1&id=gr1.jpg){kind=link}
00555-1&id=gr2.jpg){kind=link}
00555-1&id=gr3.jpg){kind=link}
00555-1&id=gr4.jpg){kind=link}
00555-1&id=gr5.jpg){kind=link}
00555-1&id=gr6.jpg){kind=link}
00555-1&id=gr7.jpg){kind=link}
00555-1&id=gr8.jpg){kind=link}