Dev and Doc: AI For Healthcare Podcast

Dr Keith Grimes is a HealthTech consultant and General Practitioner working with companies to transform clinical ideas into something impactful. He worked as the digital health director in Babylon Health prior to its demise, and currently runs his own consulting firm, Curistica. This is one not to miss! References HealthTech consulting at Curistica www.curistica.com Prof Amanda Goodall on leadership theory https://amandagoodall.com/ For those interested in Leadership opportunities: -Faculty of medical leadership and management https://www.fmlm.ac.uk/ -Bite labs https://www.bitelabs.io/ <p>Dev&Doc is a podcast where doctors and developers deep dive into the potential of AI in healthcare.<br> 👨🏻‍⚕️Doc - <a href="https://www.linkedin.com/in/dr-joshua-auyeung/">Dr. Joshua Au Yeung</a><br> 🤖Dev - <a href="https://twitter.com/zeljkokr">Zeljko Kraljevic</a><br> <a href="https://www.linkedin.com/build-relation/newsletter-follow?entityUrn=7216474068085026817">LinkedIn Newsletter</a><br> <a href="https://youtube.com/@DevAndDoc">YouTube</a><br> <a href="https://podcasters.spotify.com/pod/show/devanddoc">Spotify</a><br> <a href="https://podcasts.apple.com/gb/podcast/dev-and-doc-ai-for-healthcare-podcast/id1751495120">Apple</a><br> <a href="https://aiforhealthcare.substack.com/">Substack</a><br> For enquiries - 📧 <a href="mailto:Devanddoc@gmail.com">Devanddoc@gmail.com</a> </p> 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kraljevic/ 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovici027d Timestamps 00:00 start 1:10 Career career career - GP, babylon health, digital consultancy 6:40 working as a rural GP in Scotland 9:21 time is the biggest factor of clinical impact 12:11 finding impact through data  21:29 leading by example  23:52 Should doctors be leading healthtech businesses?  30:10 why do healthtech start-ups not have clinicians earlier?  36:30 Babylon failure - importance of having clinical influence at the top  43:55 experience being grilled on BBC newsnight  49:45 lessons learnt from the downfall of Babylon  52:25 6 values of consulting firm Curistica  55:51 common problems in start ups  59:36 how AI will change the healthcare landscape

How do we reach the holy grail of a clinically safe LLM for healthcare? Dev and Doc are back to discuss news with Meta's LlaMA model and potential of healthcare LLMs finetuned on top like BioLlaMa. We discuss the key steps in building a clinically safe LLM for healthcare for healthcare and how this was pursued by Hippocratic AI's latest model - Polaris. 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua-auyeung/ 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr The podcast 🎙️ 🔊Spotify: https://podcasters.spotify.com/pod/show/devanddoc 📙Substack: https://aiforhealthcare.substack.com/ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kraljevic/ 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovici027d References Hippocratic AI LLM- https://arxiv.org/pdf/2403.13313 BioLLM tweet - https://twitter.com/aadityaura/status/1783662626901528803 Foresight lancet paper -https://www.thelancet.com/journals/landig/article/PIIS2589-7500(24)00025-6/fulltext Linear processing units- https://wow.groq.com/lpu-inference-engine/ Timestamps 00:00 Start 01:10 Intro- llama3 , a chatGPT level model in our hands 06:53 Linear processing units to run LLMs 09:42 BioLLM for medical question and answering 11:13 quality and size of dataset, using youtube transcripts 12:41 Question and answering pairs do not reflect the real world - holy grail of healthcare llm 18:43 Dev has Beef with hippocratic AI 20:25 Step1 Training a clinical foundational model from scratch 22:43 Step 2 Instruction tuning with multi-turn simulated conversation 24:15 Step 3 training the model to guide model in tangential conversations 27:42 Focusing on the hospital back office and specialist nurse phone calls 33:02 Evaluating Polaris - clinical safety LLM , bedside manner, medical safety advice

In this special episode we share a live recording of our live podcast episode at the Rewired UK conference, where NHS, industry and policy markers unite. We discuss current LLMs from a technical and practical perspective. Dive into how to build Foundational models for the National health service and our experiences. We were also privileged to be joined by head of digital at Cambridge University Hospital NHS trust, Dr. Wai Keong Wong on how to evaluate AI products and discussions on automating administrative tasks for clinicians with ambient clinical documentation. 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua-auyeung/ 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr The podcast 🎙️ 🔊Spotify: https://podcasters.spotify.com/pod/show/devanddoc 📙Substack: https://aiforhealthcare.substack.com/ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kraljevic/ 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovici027d 00:00 intro 02:05 AI vs doctors - are language models ready to replace doctors? 05:22 the tranformer models and attention 08:51 human labour for reinforcement learning 11:00 building the NHS LLM, key concepts 13:55 foresight GPT - predicting the next clinical event in a patient timeline. 16:29 is text enough? 17:19 £3.8B investment into NHS digitisation and admin automation - ambient clinical documentation 20:14 how do you evaluate AI products for the NHS? 26:24 how do you vet the tech companies and future proof your purchase? 27:23 do clinicians need more digital health education? 28:41 transparency of AI models and benchmarks 31:30 question - EHR data created by AI leads to homogenisation and errors 34:03 question - training on structured vs unstructured EHR data 38:06 question - LLMs as a brain. How do we give it a body? 41:05 framework for ai deployment

What do Prompt engineers have in common with telephone operators in the 1870s? Spoiler - they're both dying professions 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua-auyeung/ 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr The podcast 🎙️ 🔊Spotify: https://podcasters.spotify.com/pod/show/devanddoc 📙Substack: https://aiforhealthcare.substack.com/ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kraljevic/ 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovici027d 00:00 Highlights 01:10 Intro - where did prompt engineering go wrong? 4:10 what is prompt engineering fundamentally? 10:54 LLMs training data reflects prompt engineering 12:32 prompts are model dependent 14:02 prompts that make you think 18:26 combining expert and generalist medical models for doctors 19:49 Diagnostic reasoning prompts, is it interpretable? 26:55 can we find prompts more elegantly/ systematically ? 28:42 Will prompts become obsolete? Models that self discover prompts 31:09 Telephone operators and Prompt engineers - death of a profession Refs Prompt "hacks" (oh man) - https://learnprompting.org/docs/intermediate/chain_of_thought Diagnostic prompt interpretability paper - https://www.nature.com/articles/s41746-024-01010-1 self - discover https://arxiv.org/abs/2402.03620 telephone operators - https://www.history.com/news/rise-fall-telephone-switchboard-operators

How do we align AI models for healthcare? 👨‍⚕️ And importantly, the moral codes and ethics that we practice everyday, how does the LLM deal with ethical scenarios like the trolley problem for example? This is a fascinating topic and one we spend a lot of time thinking about. In this episode Dev and Doc, Zeljko Kraljevic and I cover all the up to date topics around reinforcement learning, the benefits and where it can go wrong. We also discuss different RL methods including the algorithms used to train ChatGPT (RLHF). Dev and Doc is a Podcast where developers and doctors join forces to deep dive into AI in healthcare. Together, we can build models that matter. 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua... 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr The podcast 🎙️ 🔊Spotify: https://open.spotify.com/show/3QO5Lr3... 📙Substack: https://aiforhealthcare.substack.com/ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kral... 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovic...00:00 Highlights 01:27 start 4:38 aligning ethics of ai models 7:04 doctors ethical choices daily 8:00 RLHF and AI training methods 16:29 reinforcement learning 19:35 Preference model -rewarding models correctly can make or break the success 27:05 exploiting reward function, model degradation (and how to fix it) Ref AI intro paper - https://pn.bmj.com/content/23/6/476 Open AI RLHF paper - https://arxiv.org/abs/1909.08593 War and peace of LLMs! - https://arxiv.org/abs/2311.17227

In this episode Doc goes on an adventure to chair an LLM/ generative AI conference session and reflects on his experience. Dev and Doc also discuss big news on meta's Llama3 and Code LlaMa. Dev and Doc is a Podcast where developers and doctors join forces to deep dive into AI in healthcare. Together, we can build models that matter. 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua-auyeung/ 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr The podcast 🎙️ 🔊Spotify: https://open.spotify.com/show/3QO5Lr3w4Rd6lqwlfKDaB7?si=e7915d844994403e 📙Substack: https://aiforhealthcare.substack.com/ Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kraljevic/ 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovici027d 00:00 Highlight 00:36 Start 1:57 Are researchers just using Generative AI to get presentations /publications? 6:18 Hype cycles , lack of real world clinical studies using LLMs 8:08 LlaMa3 , Code LlaMa announcement and insights 13:30 Google bard / Gemini ultra second on leaderboard 17:30 wrap up and end

Dr. Joshua Au Yeung, a medical expert in AI advancements, and Zeljko Kraljevic, a dedicated developer in healthcare tech, dive into thrilling predictions for AI in 2024. They discuss the transformative potential of GPT-5 and the implications of ambient note-taking in medical practices. The duo highlights the tension between open-source AI models and profit-driven approaches like the GPT store. Insights into AI's role in clinical tasks and emerging tools for scientific writing enrich their conversation, painting a vibrant picture of the future of healthcare technology.

Josh, a training Neurologist and AI researcher, teams up with Zeljko, an AI researcher and CTO, to explore the transformative impact of AI in clinical coding. They discuss the hurdles of inaccurate coding and how AI can streamline this process, enhancing both efficiency and accuracy. The conversation highlights AI's role in the UK national stroke audit, where natural language processing tools are making a difference. They also stress the importance of collaboration between AI and human coders while addressing privacy concerns and the support needed for successful implementation.

In this episode, Dev and Doc sit down to discuss artificial general intelligence from the perspective of a neurologist and computer scientist. We dive into the current developments around AGI , the 2 controversial schools of thought, LLMs and neuroscience, and give hot takes about whether we will ever reach AGI. Dev and Doc is a Podcast where developers and doctors join forces to deep dive into AI in healthcare. Together, we can build models that matter. 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua-auyeung/ 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 00:00 start 01:05 intro 03:46 two camps of AGI - Yann Lecun vs Geoffrey Hinton, Architecture vs Data 07:47 Do emergent capabilities of LLMs pose a threat to humanity? 08:45 Intelligence and AGI - neuroscience and computer science approach 16:59 LLMs vs the human brain 24:16 Do AIs need a human touch? - Intrinsic personalities, temperaments, motivations, joy and reward The podcast 🎙️ 🔊Spotify: https://open.spotify.com/show/3QO5Lr3w4Rd6lqwlfKDaB7?si=e7915d844994403e 📙Substack: https://aiforhealthcare.substack.com/ 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kraljevic/ 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovici027d

De-identifying and anonymising PHI (protected/personal health information) in health records is one of the central pillars of AI success in healthcare. Without de-identified data we cannot share data between hospitals , train models confidentially, or safely create large language models. Live from new orleans, Dev and Doc are here to dive into this fascinating topic, as well as describe our experiences of building and deploying an AI model with over 99% recall for redaction of PHI. Dev and Doc is a Podcast where developers and doctors join forces to deep dive into AI in healthcare. Together, we can build models that matter. 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung - https://www.linkedin.com/in/dr-joshua... 🤖Dev - Zeljko Kraljevic https://twitter.com/zeljkokr Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) 00:00 start 00:52 intro 2:10 what is PHI? Personal /private health information 7:00 approaches on de-identifying hospital records 9:55 the problem with over-redaction /anonymisation 11:33 using deep learning for anonymisation 14:13 our experiences building a over 99% recall model VS manual annotation 18:03 how to make a high performing model - the art of annotations 24:49 Dev and Docs annotation method (Zeljko et al.) 30:42 how do you prevent overfitting? 31:54 ensuring model performs in new hospital / environments 33:23 future 34:48 synthetic data The podcast 🎙️ 🔊Spotify: https://open.spotify.com/show/3QO5Lr3... 📙Substack: https://aiforhealthcare.substack.com/ 🎞️ Editor- Dragan Kraljević https://www.instagram.com/dragan_kral... 🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovic...