A Journey Around the Globe with Dr. Cara Liebert
By Mohammed Al Kadhim
Dr. Dana Lin and Dr. Cara Liebert are Co-Principal Investigators awarded the Department of Surgery’s first Center for Innovation in Global Health Global Surgery Seed Grant.
Congratulations for receiving the first Global Surgery Seed Grant from the Department of Surgery. Your project (Implementation of the ENTRUST Platform for Global Surgical Education and High-Stakes Assessment in Low- and Middle-Income Countries) was selected from more than 40 proposals that were submitted through the CIGH. What does that mean to you?
Dr. Liebert: “Thank you very much. We are so grateful for all the support from the department. The award means a lot to us because funding is crucial to allow our work to expand and will make it possible for us to continue remarkable collaborations with global partners.”
Please tell us about the stages of developing the ENTRUST platform.
Dr. Cara Liebert and Dr. Dana Lin began developing the ENTRUST Assessment Platform approximately four years ago. The two Stanford Surgery faculty members had previously collaborated on a variety of other projects in surgical education. Their goal was developing a virtual patient simulation platform that would allow the assessment of surgical decision-making.
“We found that there were already existing tools that aim to assess surgical skills and evaluate basic knowledge, but other than clinical observations and Oral Board Exams, there were limited tools to evaluate complex surgical decision-making. We wanted to develop a platform to decrease faculty evaluation and assessment’s burden and allow nuanced evaluation of clinical decision-making. Our goal in developing ENTRUST was to complement existing assessment tools for evaluating intra-operative performance, such as SIMPL, OSATS, and GEARS,” Dr. Liebert said.
She added: “We found that ENTRUST looked very promising in terms of its ability to evaluate not just the knowledge but also the ability to make complex surgical decisions and how it develops over the trajectory from a medical student to an intern, a graduating resident, and an attending surgeon.”
Who built the ENTRUST platform?
“We identified a collaborator, Dr. Edward Melcer, at the University of California at Santa Cruz. Dr. Melcer is an Assistant Professor of Computational Media, and his expertise and interest is in designing online platforms and serious games for learning and assessment, so our interests aligned.
“With seed grant funding from the Division of General Surgery and Department of Surgery, Dr. Lin and I assembled a research team, developed ENTRUST, and served as content experts. Together with Dr. Melcer, we assembled and hired a team of programmers, user interface designers, and artists comprised of master’s students in Serious Games and Playable Media and PhD students in Computational Media at UCSC. One of those master’s students was Jason Tsai, a talented full-stack programmer who initially worked part-time with us. We later hired him on a full-time basis to become our lead programmer who will take the platform to the next level. It is great working with Jason and see him progress from a master’s student to a full-time Stanford lead programmer for ENTRUST.”
Dr. Cara Liebert
Dr. Dana Lin
What were the challenges associated with improving the ENTRUST platform?
“We had financial concerns because we needed more support to increase the security infrastructure as a high stakes exam, accessibility across the globe, expand ENTRUST from an assessment platform to also include a learning platform.”
What about engaging globally, and your collaboration with the College of Surgeons of East, Central, and Southern Africa (COSECSA)?
Once the initial ENTRUST platform was built, Beta testing began. One of those testers was Dr. Sherry Wren, an internationally renowned Global Surgery leader with a long history in humanitarian efforts. Dr. Wren found in ENTRUST a great potential and utility in using it with the College of Surgeons of East, Central, and South Africa (COSECSA) in their examination process.
Circumstances play a critical role sometimes; for as the COVID-19 pandemic had just hit the world and the whole examination paradigm had turned upside-down. What previously had been the need for trainees in 14 different countries in sub-Saharan Africa to travel for in-person oral examinations to complete their MCS (Membership of the College of Surgeons) exam, would now be held as a virtual online exam in 2021.
Dr. Cara Liebert said: “At this point, we began a collaboration with Chair of the Examinations and Credentials Committee for COSECSA, Professor Abebe Bekele. He found in ENTRUST a great utility for their MCS Examinations.”
Dr. Liebert and Dr. Lin successfully piloted three clinical cases on the ENTRUST Assessment Platform during the MCS Exam in 2021. ENTRUST was well-received by the examinees, who expressed a desire to continue to use ENTRUST for assessment and learning. Together with Professor Bekele, they planned to implement ENTRUST as a formal portion of the MCS Exam in 2022 and develop an ENTRUST Learning Platform for COSECSA trainees.
“It is this collaboration that inspired us to apply for the global surgery seed grant through the CIGH. We believe that this grant will enable us to expand ENTRUST as a global surgical education learning and assessment platform for low- and middle-income countries,” Dr. Liebert said.
“It’s been a wonderful collaboration”
On this collaboration with COSECSA, Dr. Liebert confirmed: “The collaboration has been very successful; we meet on Zoom from different locations and author clinical cases together with Professor Bekele and his team. All cases are reviewed by COSECSA faculty to confirm that the treatment algorithms and the scoring are consistent with standard of care and systems-based practices in COSECSA. Accordingly, we beta test the cases together and deploy them online. We were also part of the orientation sessions for trainees for the MCS Exam, and during the examination day, we were present to deploy the ENTRUST pilot cases.”
What’s the impact of Stanford Surgery’s global programs?
“Our goal is to make an impact in advancing global surgical education; we believe that improving access to surgical education resources will help prepare trainees across the world to meet the needs of their communities and help address the global surgical disease burden.
“The visible impact that we can sense motivates us to do more, and what better place would be to achieve that goal than Stanford Surgery? Our department has and will always be keen in sharing surgical advancement and cutting-edge innovation with partner institutions all over the globe. We aim to build bridges of solidarity with surgeons everywhere.”
How is patient care and health education different in other countries than how it is in the US?
“I spent a month in Zimbabwe during my surgical residency, at the University of Zimbabwe in Harare. It was striking to see how limited the resources were, including very limited stock of sutures and supplies in the operating theatre and on the patient wards.
“While here in the US every surgery residency program has a simulation center, resources for surgical education are much more limited in low- and middle-income countries across the globe.
“Our hope is that the ENTRUST Learning and Assessment platform will have a significant impact on access to global surgical education tools. It is free and digitally accessible online. We hope that for institutions with limited educational resources or formal curriculum, ENTRUST can be used to augment learning and assessment for students and residents.”
How exciting is it to see the impact of your global work? And how does it build into the next stages?
“It was incredibly exciting to see the COSECSA examinees utilize ENTRUST in November 2021 and to hear their feedback. We asked them to complete a questionnaire afterwards, and it was striking that 92% of the examinees from COSECSA wanted to use ENTRUST as a learning platform. We received feedback that the opportunity for junior residents to walk through the evaluation and clinical decision-making for surgical patients in a safe and virtual environment was very valuable.
“That helped us see that the assessment impact was of ENTRUST was high, but also as a learning platform, ENTRUST would provide a safe and accessible learning environment for junior trainees to prepare for the transition to a senior resident. We hope ENTRUST will increase their knowledge, clinical decision-making skills, and confidence.”
COSECSA’s next MCS examinations, and the COSECSA conference at Namibia/ Winter 2022
“This year the MCS Exam will be in November 2022. It will be completely virtual and ENTRUST will be included as a formal portion of the exam. We have expanded the platform for enhanced functionality, added an interactive tutorial, and increased the ability to support a broader array of surgery cases.
“Dr. Lin and I are very excited that we will be travelling to Namibia to attend the COSECSA Conference in early December. In the first part of the conference, we will be able to meet in-person with Professor Bekele and members of the Examinations and Credentials Committee in COSECSA for the first time after collaborating with them virtually. We will also have the opportunity to present our ongoing work with ENTRUST to the committee.
“Dr. Lin and I have also been invited to be examiners for the Fellowship of the College of Surgeons (FCS) Exam in Namibia in December. This is the examination residents in COSECSA training programs take after completing their training, akin to the American Board of Surgery Certifying Oral Board Exam. We are honored and excited for this opportunity to serve as FCS examiners.
“Finally, Dr. Lin and I will attend the COSECSA scientific conference where COSECSA faculty and trainees will present their scientific research.”
Do you have other projects in collaboration with Global Engagement at Stanford Surgery?
“Recently, we started working with Global Engagement, other faculty members at Stanford Surgery, and The University of Global Health Equity in Rwanda/ UGHE to develop learning modules at the medical student level on the ENTRUST platform in collaboration with Professor Bekele, and Fogarty Fellow Dr. Barnabas Alayande. This collaboration will allow us to develop and deploy the ENTRUST Learning Platform for medical students at UGHE and determine how the learning platform could be most effective.
“Our goals after we create the ENTRUST learning modules at UGHE in Rwanda for the medical students are to create ENTRUST Learning Platform modules for COSECSA and expand the ENTRUST Learning Platform to other universities globally.
“If the ENTRUST medical student modules at UGHE/ Rwanda are well-received, we can offer these or similar modules to medical students in other universities that are part of COSECSA. And similarly, if we create a set of residents’ learning modules for COSECSA and they are well-received, then we could consider approaching other global surgical educational organizations to share ENTRUST with a larger audience."
Thank you, Dr. Liebert, the work you and Dr. Lin are doing is ABSOLUTELY GREAT.