The extensive use of emerging technologies like generative AI and consultations in the metaverse may be in the cards for hospitals in Shanghai, China.
The 2,000-bed Zhongshan Hospital has been tasked by the city, which is also the country's major financial hub, to become its "model hospital of the future."
In her keynote presentation at HIMSS25 APAC, Aurel Tsien, a professor at Fudan University and a director at Zhongshan Hospital, noted that their affiliate hospital based its smart hospital strategy on three pillars: patient-friendly healthcare, functional resource management, and smart business operations.
Doctor-led innovation in algorithm technology
Zhongshan has been working with doctors to develop "super smart" algorithms based on multimodal large language models (LLM) to support their decision-making. One algorithm, Prof Tsien shared, can be used to screen elderly patients in the physical examination room and provide personalised lifestyle recommendations.
Another algorithm was developed for its nursing department's Enhanced Recovery After Surgery programme. "We collect and use different types of data from each patient to create personalised protocols before and after surgery."
Additionally, the hospital incorporated a multimodal algorithm in its virtual reality-based surgery training. It also developed a pharmaceutical platform for analysing and combining multi-omics data with clinical data for research projects, particularly clinical trials.
Meanwhile, Zhongshan developed a digital imaging platform that digitalises pathology workflows to meet the imaging needs of different hospital departments.
"Before, the entire process took two to five days. But with AI, we are able to reduce the diagnosis process to five to 10 minutes, which is a big jump in efficiency for pathology."
China, Prof Tsien mentioned, is facing a shortage of pathologists. "We really need the help of algorithm technology to fill in the gap where we cannot serve that many pathology needs from various departments."
Zhongshan in the metaverse
Zhongshan has recently piloted its digital hospital in the metaverse as part of initiatives to meet its high patient volume. With 20,000 patient visits per day and only 2,000 doctors around to provide care, the hospital realised it has to "build a precise, accurate digital twin of patients."
For this state-backed project, the hospital follows a "general plus dedicated model schema." Firstly, it collects data from patients through various workflows and uses LLM to process it and identify their intention. After that, they deploy a specific algorithm and respond to them through the digital avatar of an attending doctor or nurse; according to Prof Tsien, the hospital developed a pipeline of dedicated algorithms.
"The algorithm from our backend could be a particular AI-based diagnosis tool, or a chatbot that can help relieve a patient's anxiety, or provide drug prescription recommendations or advice on taking their medications."
In this project, the most challenging part, Prof Tsien admitted, was importing doctors' medical expertise into the algorithms. "We have to collaborate with our physicians to reconstruct the whole reasoning loop for the algorithm to understand the steps to follow in clinical decision making."
The project team also had to improve the algorithms' understanding accuracy so they would not miss any important details from their patients. "Through extensive modifications, we're able to increase the understanding accuracy up to 90%, for example, in the algorithm for our stomach cancer ward."
At present, Zhongshan's digital model has more than 80% of patients actively interacting.
"In the future, we look forward to using this kind of metaverse technology to alleviate some burden in the hospital workflows. We hope that in this process, we can collaborate with more users to truly capture their intentions and improve the model's accuracy," Prof Tsien said.
