Spotlight: Dr. Carla Pugh

Talking with Dr. Carla Pugh, Professor of Surgery at Stanford University School of Medicine and the Director of the Technology Enabled Clinical Improvement (T.E.C.I.) Center, is inspiring. She is the first surgeon in the United States to obtain a PhD in Education, and her work focuses on using technology to change how medical and surgical education is conducted. Her research involves the use of simulation and advanced engineering technologies to develop new approaches for assessing and defining competency in clinical procedural skills.

Dr. Pugh holds three patents on the use of sensor and data acquisition technology to measure and characterize hands-on clinical skills. Currently, over two hundred medical and nursing schools are using one of her sensor-enabled training tools for their students and trainees. She is an international expert on the use of sensors and motion tracking technology for performance measurement.

In 2011 Dr. Pugh received the Presidential Early Career Award for Scientists and Engineers from President Barak Obama at the White House. In 2014 she was invited to give a TEDMed talk on the potential uses of technology to transform how we measure clinical skills in medicine. In April 2018, Dr. Pugh was inducted into the American Institute for Medical and Biological Engineering. She was a member of the judging panel of The Intuitive Foundation which recently granted a $ 1 Million Prize to two winners of the Global Surgical Training Challenge.

Q: You argue that technology had huge potential to facilitate medical education and training. How much of a difference can utilizing innovation and modern technology make to improve the quality of patient care and surgical outcome on a wide scale? What are the main obstacles that you confront? How does technology help you accomplish educational and training goals?

I was so excited to get into medical school.  I knew that it was going to be hard work and that it required a lot of reading of textbooks and fast-paced learning, but I was up for the task. In my mind, I expected a different workflow going from my medical school training to my surgical training. Medical school is like “adult learner” meets “massive context and content.” Then when you go to surgery, it’s a hands-on field.

In training, most of surgery and access to learning surgery - about 70% - was still through textbooks and lectures. Then I have my time in the OR and the only other opportunity I had was to check out videos from the library. At that time there was no YouTube, there was no Google, none of that.

But I was eager for constant information, I was someone who wanted all the information that I needed at my fingertips. I wanted 24 hours access to information. But it wasn’t there. Even when the library closed at midnight, I would try to find libraries at other schools that stay open 24 hours a day. I was a frustrated learner and textbooks didn’t meet my needs. For example, I would be looking at a 2D diagram of a 4-dimensional process that had nearly limitless inputs that effect it. From a scientific perspective, that was a huge hurdle.

Luckily, I had an opportunity as a student to see the potential of computer-based learning. My school, Howard, was participating in a pilot with the National Board of Medical Examiners who were piloting a computer-based exam where you can have pictures or real scenarios of clinical cases. I knew that this will be needed in surgery.

I really needed videos. I needed access to pictures of instruments to search for. You get a textbook and they talk about 20 instruments then they show a picture that has just two.

I had a lot of desire as a resident to learn everything but that was only possible on your own time - and that was the motivation. Technology and computer-based learning was a big dream of mine at that time.

Now today, we have all this computer power - all the databases, Google, YouTube videos. We have access to all that information but it’s not that accurate and it’s not properly labeled. It is difficult to find what you need. So, you go from a library system with a librarian who can help you find anything but its in 2D, to now where you have information overload but its completely disorganized and unrated for its accuracy and utility. So, we just switched out one problem for another.

Fast forward to the work. I had the opportunity and the grand pleasure of working with a number of really smart people in this phase when I was introduced to advanced technology and data acquisition and human computer interface workflows. That was when the lightbulb turned on for me. I knew that I could tap into the unconscious!

When someone is an expert, there’s a level of unconscious effort to their workflow. When they were learning, it was just like learning to play a piano - you had the black and white and you are counting. But when you become a master pianist, that counting and black and white goes away. You are in the unconscious mode. You are an expert now, and fluent.

So, I realized if I could digitize master surgeons and quantify their workflow and their fluidity in their execution in what they do, then I can bring that to light. None of the things that they do are in a textbook, Half of it doesn’t even have a verbal language. So, when we had a chance to transform expertise into visible and reproducible data, that was huge and super exciting. These are all the moves that you make; you go from left to right here; this is the velocity in which you did it.

Now I can do what I do in surgery, and in technology-enabled evaluation, immediately from pattern recognition. Human beings are good at this, we’re trained to see patterns since we were kids: it’s either a square or a circle; it’s red or it’s yellow. I can now see someone’s raw motion data and I can see the difference between two different persons. And that’s when we knew that we have the opportunity to transform access to performance feedback that matched the load of complexity of what we do in the operation room. Turning the information into something that is simple and visible in a form that we never had before is an exciting opportunity. The technology is advancing in this area which could continue our ability to do different data visualization approaches, data displays, and data capture.

We are in a point where master surgeons are excited to line up and give us their data because they know that it’s useful for training. Now, we have a benchmark. Before this, even when simulation became a technology in the surgical profession, we were still mainly using simulations with students and residents to teach them outside of the operating room. But, the biggest problem with that is that we are still using human observation to say whether they did well or not. Then, we further translate that human observation into a verbal language which is an additional filter. There’s someone doing something with their hands and there’s another human who’s observing it. But there are things that humans cannot observe and track, like force or velocity. So, the first filter is human observation and the second filter is the verbal description of what you observed. These two filters cut the information down by 70% in terms of its truth, accuracy, and usefulness.

Today, technology can capture information in detail that human beings can’t capture with observation. Instead of having experienced surgeons be the only assessment for resident performance, we now have the ability to combine that expert observation with technology-based observations. When you have the combination of human and computer you are getting closer to something that is more accurate that could translate into learning objectives.

Trying to translate a verbal description of what I saw you do into a learning objective that someone can train on is extremely difficult. But when you have data and you have a visible benchmark, you can say, “This is what it looks like when someone is doing it well.”

Now you can practice. Now you can videotape yourself and see which parts of the operation you can improve. Now you have individualized training and ability.

That’s the biggest thing, we were training residents using human observation alone and giving verbal feedback but now we have an opportunity to combine the best of the best.

Q: Can you share with us the progress of your “tracking movements” research and the steps you have taken in developing technologies that improve surgeons’ performance and skills? How can you use this technology, and the data it generates, to enhance the confidence and skills of surgeons? 

There are multiple ways to skin a cat, but a few of them are not effective. That’s a common phrase that people use in mostly hands-on professions and it’s true in surgery as well.

So as an example, two experts could do things differently and both could get good outcomes, and that’s ok. But if you take a hundred experienced surgeons you may find that there are 6 different ways to do the procedure; three of them give great outcomes, two are ok, and 1 is less than perfect in terms of efficiency and safety.

We can identify the approaches that we should not be handing down by tradition. These different approaches have not been empirically evaluated and tested with respect to procedure outcomes, but we are now systematically learning that we are capable of picking out those approaches. Whether it was a physical exam or a surgical operation, the same thing exists.

That’s the big picture of what we are learning in our research.

Q: You were recently a judge for the “The Intuitive Foundation’s Global Surgical Training Challenge” in December 2022 – congratulations! How do would you describe this experience, and how does this approach help train health care providers in the places where teaching and training are needed most?

It was a huge honor and very exciting to work with that team and to be part of that mission. Anyone in healthcare who has a system level of thinking realizes that there’s different levels of resources, not only financially but also in equipment and in terms of community support for the physician, or even in terms of access to care and even health expectation and the sense of entitlement to be well.

Just to think about that at an individual level is variable by country. In some countries, people think that healthcare or even feeling well is their right. In other places, people settle to the idea that they may die by the age of 45 because they don’t have access to healthcare. To them, it is normal and they are used to it.

So, to have a foundation that has a strategic plan to make a difference is huge. I commend the Intuitive Foundation and Catherine Mohr for focusing in this area and pulling together an amazing team of evaluators and judges.  It was a great pleasure working with them and learning from their variety of expertise and perspectives. There was global experience on the table, and it was an amazing process.

When I think about the goal of building inexpensive simulations for training surgical procedures, I see that in other programs prior to this would bring their simulation to certain areas, do the training, then leave. That’s one kind of help. But when you help people build their own simulations and you enable them to develop a protocol on how to distribute that locally, that becomes a different level of interaction. You’re building a community in that space. You’re enabling a community to own subject matter expertise and build it themselves and then be a collaborator. Even some of the materials may be unapplicable: I can bring my own simulator to some other country but even if they wanted to build the simulator a have it reverse engineered, they may have different materials available to them locally that might be different from what I have. So it’s not only the knowledge, because many of them are knowledgeable. But when we partner together in terms of fabrication and clinical knowledge, we can then provide a framework of what we all agreed would be the minimum requirements of a training tool that could be utilized for teaching and procedure.

In this challenge you asked me about, they brought their own knowledge and their own materials, and we gave them a platform to exercise it, we gave the financial support and collaboration.

We asked them difficult questions: Are these the right materials? Is this reproducible? Where are you getting those materials from? A few of them were based on 3D printing and we didn’t know if that was available or not. It turned out it was, but the materials weren’t.

No one in the US would know what it takes a person who is local to deliver this. They are on the ground, making those phone calls and connections, they know if they can build a community of people that support the fabrication, and what the easiest route might be. For example, you get three vendors and one of them will take forever because its source materials are second hand from some other place.

Thinking together and brainstorming together on options was the nuts and bolts of this partnership. Also, giving them the finances and advisors to query the fabrication of things is crucial.  

Q: What were the major factors that you based the judging criteria upon to select the group(s) who received the reward? Tell us more about All SAFE, the project of designing a dual-approach model to teach surgeons how to perform laparoscopic treatment of ectopic pregnancies that was the top winner (winning first prize of $700K)?

There were different stages for proposals that came in, and there were initially over 15 proposals. There was a series of assessments then there were a few who made it to the final round, then there was another level of assessment in terms of their ability to deliver. I think that for us, the main evaluation consideration was related to the team. Who were the people on the team? What was their level of experience with developing a curriculum assessing skills and performance and their access to the intended user?

If you already built something before, you have access to it, and you already have a curriculum, those are the basic elements that will enable you to succeed more than someone pulling together the team for the very first time where there might not be clear team dynamics and workflow.

Some proposals didn’t make it far because their teams were new and pulled together. They didn’t have a great workflow between the team members. Some were across different countries and made connections, which sounds great in terms of having interdisciplinary teams, but if their workflows don’t allow them to work together very well then the chances of them codeveloping a product are slim. Some people struggled with partnering with locals or to come together and be able to fabricate.

The other main thing that we looked at was reproducibility of the simulation trainer and usability of the fabrication manual. It is one thing to build something and its something else to teach someone else how to build it. Are all your instructions clear? Some were amazing. They made training videos just like a cooking show: these are the ingredients, you set them all out, this one you purchase here, we looked at these three materials and chose this one for this reason in terms of longevity and shelf life, things like that. It was really great to see the variety of fabrication training. Some of them were slides and some were videos. We asked ourselves: Is this something easy for other people to build? Is it scalable? Does it take three months to build one, or are the materials so accessible that you could build twenty of them in a month? Those were two different things. Does it take six months to build a stimulator for a ten-minute exercise? Then that is imbalanced in terms of utilization of resources and feasibility.

The other main variable was building an assessment. We didn’t just want you to build a simulator that other people could use and play with. Does the simulator really assess skill? Not only as a training tool but also as an assessment tool. Assessment was huge. Now that you trained on this and you feel like you know how to do this, we can put you in a different scenario and can you show that you learned something and that you know how to do the critical aspects of this procedure. So that was the final puzzle piece that completed the challenge.

Q: We learned that you were recently awarded a grant from Wellcome Leap Organization through its SAVE program. Can you tell us a bit about your proposal, the goals, and partners you intend to engage with?

The Wellcome Leap program was exciting to learn about and in many ways its another example of an organization that sees a need and puts money in that space to enable a difference in global healthcare. This really gives people hope that there is more than one group who is paying attention and willing to put money in that space. Hope is worth a lot, to see people who are paying attention and want to put in formal programs and strategic planning to address a global need.

Our proposal is about digitizing the expertise of master surgeons. It iis related to our use of sensors and our understanding of a performance assessment that we’ve built with multiple grants from the government and the multiple development opportunities that we had with our funding. We are very well versed in a variety of sensors and biomarkers that can help digitize the surgeon. If you can digitize expertise of a master surgeon, then you have criteria and performance to track the learning curve of someone who is on the learning curve to competency as well as someone who is on the learning curve for mastery, because you will know what mastery is and what it looks like.

For our proposal we took our expertise in wearable technology and data acquisition and quantification of mastery and combined that with two groups that help with the deliverable. One of those groups is Surgical Safety Technologies. They are an AI based company that looks at OR data and can use computer vision to annotate models and give data on OR workflows and performance. So just the power of their artificial intelligence in computing added to our wearable technology data output to metrics, that takes things to another level in terms of being able to capture and quantify an efficient assess performance.

The main goal was to take those two approaches and combine the computing power to then generate a detailed report card that can provide individualized information on where a trainee is efficient, where a trainee met the mark, and where there were opportunities for improvement based on a mastery performance database in that section of the operation. We then detail it for each phase of the operation; the beginning, the middle, the end, the stapling, the removal of a specimen, and the reconstruction. There are these key phases in every operation so being able to get feedback and the comparison with mastery for each of those segments of the procedure is a major opportunity.

The other major collaborator we have is The Academy for Surgical Coaching and the idea of an existing group that has a number of surgeons who use video alone for coaching other surgeons. The great benefit is that we will be able to enact, for the first time, data driven coaching.

So, combining those three capabilities together., we’re using the computing power of the T.E.C.I. center with Surgical Safety Technologies to then deliver a report card that enables coaches to then show the surgeons to look at the card and agree on what they need to address based on the data. So being able to do data driven coaching, we can think that this will end up with a major contribution in terms of shortening the learning curve and shortening training because right now all of our training is based on human observation and verbal feedback.

Now to combine that is a big dream. With data driven feedback plus coaching - that’s a huge opportunity! We are super excited to partner with these collaborators that we have.

It takes a village to do this type of work and I’ve just been very lucky to have very dedicated people who are excited about the work. We’ve been able to craft the proposal so that the National Institute of Health, the DoD, and now Wellcome Leap believes in it. I feel like I’m living a dream. It’s super exciting to have people believe in it and want to partner with and help us because it’s important work. I’m honored to be part of the story and part of the team.