David J. Hand

David Hand, a professor of statistics at Imperial College London, dives into the fascinating world of dark data and its implications on analysis. He explains how unseen data can mislead conclusions, especially in critical areas like public health and AI. Hand contrasts data-driven and theory-driven models, emphasizing the risks of relying solely on the former. He also addresses the limitations of large language models, cautioning against their perceived intelligence. The discussion reveals the intricate balance between data transparency and public trust in research.

Author David J. Hand explores 'dark data' - uncollected, misrecorded or concealed data, and its impact on decision-making. Topics include flawed data in historical predictions, Campbell's Law and Goodhart's Law in social decision making, John Law's financial schemes, and the role of dark data in fraud detection and machine learning.

David J. Hand, author of "Dark Data: Why What You Don't Know Matters," dives into the fascinating world of unseen data. He discusses how traditional statistics often overlook vital uncollected information, which can skew understanding. With intriguing case studies, he illustrates the real-world implications of dark data, from polling inaccuracies to fraud detection challenges. Hand emphasizes the need for refined data collection strategies and how recognizing dark data can transform decision-making and analytical methods, particularly in machine learning.