Connor Tan is a physicist and senior data scientist working for a multinational energy company where he co-founded and leads a data science team. He holds a first-class degree in experimental and theoretical physics from Cambridge university. With a master's in particle astrophysics. He specializes in the application of machine learning models and Bayesian methods. Today we explore the history, pratical utility, and unique capabilities of Bayesian methods. We also discuss the computational difficulties inherent in Bayesian methods along with modern methods for approximate solutions such as Markov Chain Monte Carlo. Finally, we discuss how Bayesian optimization in the context of automl may one day put Data Scientists like Connor out of work. Panel: Dr. Keith Duggar, Alex Stenlake, Dr. Tim Scarfe 00:00:00 Duggars philisophical ramblings on Bayesianism 00:05:10 Introduction 00:07:30 small datasets and prior scientific knowledge 00:10:37 Bayesian methods are probability theory 00:14:00 Bayesian methods demand hard computations 00:15:46 uncertainty can matter more than estimators 00:19:29 updating or combining knowledge is a key feature 00:25:39 Frequency or Reasonable Expectation as the Primary Concept  00:30:02 Gambling and coin flips 00:37:32 Rev. Thomas Bayes's pool table 00:40:37 ignorance priors are beautiful yet hard 00:43:49 connections between common distributions 00:49:13 A curious Universe, Benford's Law 00:55:17 choosing priors, a tale of two factories 01:02:19 integration, the computational Achilles heel 01:35:25 Bayesian social context in the ML community 01:10:24 frequentist methods as a first approximation 01:13:13 driven to Bayesian methods by small sample size 01:18:46 Bayesian optimization with automl, a job killer? 01:25:28 different approaches to hyper-parameter optimization 01:30:18 advice for aspiring Bayesians 01:33:59 who would connor interview next? Connor Tann: https://www.linkedin.com/in/connor-tann-a92906a1/ https://twitter.com/connossor

Today we had a fantastic conversation with Professor Max Welling, VP of Technology, Qualcomm Technologies Netherlands B.V.  Max is a strong believer in the power of data and computation and its relevance to artificial intelligence. There is a fundamental blank slate paradgm in machine learning, experience and data alone currently rule the roost. Max wants to build a house of domain knowledge on top of that blank slate. Max thinks there are no predictions without assumptions, no generalization without inductive bias. The bias-variance tradeoff tells us that we need to use additional human knowledge when data is insufficient. Max Welling has pioneered many of the most sophistocated inductive priors in DL models developed in recent years, allowing us to use Deep Learning with non-euclidean data i.e. on graphs/topology (a field we now called "geometric deep learning") or allowing network architectures to recognise new symmetries in the data for example gauge or SE(3) equivariance. Max has also brought many other concepts from his physics playbook into ML, for example quantum and even Bayesian approaches.  This is not an episode to miss, it might be our best yet!  Panel: Dr. Tim Scarfe, Yannic Kilcher, Alex Stenlake 00:00:00 Show introduction  00:04:37 Protein Fold from DeepMind -- did it use SE(3) transformer?  00:09:58 How has machine learning progressed  00:19:57 Quantum Deformed Neural Networks paper  00:22:54 Probabilistic Numeric Convolutional Neural Networks paper 00:27:04 Ilia Karmanov from Qualcomm interview mini segment 00:32:04 Main Show Intro  00:35:21 How is Max known in the community?  00:36:35 How Max nurtures talent, freedom and relationship is key  00:40:30 Selecting research directions and guidance  00:43:42 Priors vs experience (bias/variance trade-off)  00:48:47 Generative models and GPT-3  00:51:57 Bias/variance trade off -- when do priors hurt us  00:54:48 Capsule networks  01:03:09 Which old ideas whould we revive  01:04:36 Hardware lottery paper  01:07:50 Greatness can't be planned (Kenneth Stanley reference)  01:09:10 A new sort of peer review and originality  01:11:57 Quantum Computing  01:14:25 Quantum deformed neural networks paper  01:21:57 Probabalistic numeric convolutional neural networks  01:26:35 Matrix exponential  01:28:44 Other ideas from physics i.e. chaos, holography, renormalisation  01:34:25 Reddit  01:37:19 Open review system in ML  01:41:43 Outro 

Welcome to the Christmas special community edition of MLST! We discuss some recent and interesting papers from Pedro Domingos (are NNs kernel machines?), Deepmind (can NNs out-reason symbolic machines?), Anna Rodgers - When BERT Plays The Lottery, All Tickets Are Winning, Prof. Mark Bishop (even causal methods won't deliver understanding), We also cover our favourite bits from the recent Montreal AI event run by Prof. Gary Marcus (including Rich Sutton, Danny Kahneman and Christof Koch). We respond to a reader mail on Capsule networks. Then we do a deep dive into Type Theory and Lambda Calculus with community member Alex Mattick. In the final hour we discuss inductive priors and label information density with another one of our discord community members.   Panel: Dr. Tim Scarfe, Yannic Kilcher, Alex Stenlake, Dr. Keith Duggar Enjoy the show and don't forget to subscribe! 00:00:00 Welcome to Christmas Special!  00:00:44 SoTa meme  00:01:30 Happy Christmas!  00:03:11 Paper -- DeepMind - Outperforming neuro-symbolic models with NNs (Ding et al) 00:08:57 What does it mean to understand?  00:17:37 Paper - Prof. Mark Bishop Artificial Intelligence is stupid and causal reasoning wont fix it 00:25:39 Paper -- Pedro Domingos -  Every Model Learned by Gradient Descent Is Approximately a Kernel Machine 00:31:07 Paper - Bengio - Inductive Biases for Deep Learning of Higher-Level Cognition 00:32:54 Anna Rodgers - When BERT Plays The Lottery, All Tickets Are Winning 00:37:16 Montreal AI event - Gary Marcus on reasoning  00:40:37 Montreal AI event -- Rich Sutton on universal theory of AI 00:49:45 Montreal AI event -- Danny Kahneman, System 1 vs 2 and Generative Models ala free energy principle 01:02:57 Montreal AI event -- Christof Koch - Neuroscience is hard 01:10:55 Markus Carr -- reader letter on capsule networks 01:13:21 Alex response to Marcus Carr  01:22:06 Type theory segment --  with Alex Mattick from Discord 01:24:45 Type theory segment -- What is Type Theory  01:28:12 Type theory segment -- Difference between functional and OOP languages  01:29:03 Type theory segment -- Lambda calculus  01:30:46 Type theory segment -- Closures  01:35:05 Type theory segment -- Term rewriting (confluency and termination)  01:42:02 MType theory segment -- eta term rewritig system - Lambda Calculus   01:54:44 Type theory segment -- Types / semantics  02:06:26 Type theory segment -- Calculus of constructions  02:09:27 Type theory segment -- Homotopy type theory  02:11:02 Type theory segment -- Deep learning link  02:17:27 Jan from Discord segment -- Chrome MRU skit  02:18:56 Jan from Discord segment -- Inductive priors (with XMaster96/Jan from Discord)  02:37:59 Jan from Discord segment -- Label information density (with XMaster96/Jan from Discord)  02:55:13 Outro

Dr. Eray Ozkural is an AGI researcher from Turkey, he is the founder of Celestial Intellect Cybernetics. Eray is extremely critical of Max Tegmark, Nick Bostrom and MIRI founder Elizier Yodokovsky and their views on AI safety. Eray thinks that these views represent a form of neoludditism and they are capturing valuable research budgets with doomsday fear-mongering and effectively want to prevent AI from being developed by those they don't agree with. Eray is also sceptical of the intelligence explosion hypothesis and the argument from simulation. Panel -- Dr. Keith Duggar, Dr. Tim Scarfe, Yannic Kilcher 00:00:00 Show teaser intro with added nuggets and commentary 00:48:39 Main Show Introduction  00:53:14 Doomsaying to Control   00:56:39 Fear the Basilisk!   01:08:00 Intelligence Explosion Ethics   01:09:45 Fear the Automous Drone! ... or spam   01:11:25 Infinity Point Hypothesis   01:15:26 Meat Level Intelligence  01:21:25 Defining Intelligence ... Yet Again   01:27:34 We'll make brains and then shoot them  01:31:00 The Universe likes deep learning  01:33:16 NNs are glorified hash tables  01:38:44 Radical behaviorists   01:41:29 Omega Architecture, possible AGI?   01:53:33 Simulation hypothesis  02:09:44 No one cometh unto Simulation, but by Jesus Christ   02:16:47 Agendas, Motivations, and Mind Projections   02:23:38 A computable Universe of Bulk Automata  02:30:31 Self-Organized Post-Show Coda  02:31:29 Investigating Intelligent Agency is Science  02:36:56 Goodbye and cheers!   https://www.youtube.com/watch?v=pZsHZDA9TJU

This week Dr. Tim Scarfe, Dr. Keith Duggar and Connor Leahy chat with Prof. Karl Friston. Professor Friston is a British neuroscientist at University College London and an authority on brain imaging. In 2016 he was ranked the most influential neuroscientist on Semantic Scholar.  His main contribution to theoretical neurobiology is the variational Free energy principle, also known as active inference in the Bayesian brain. The FEP is a formal statement that the existential imperative for any system which survives in the changing world can be cast as an inference problem. Bayesian Brain Hypothesis states that the brain is confronted with ambiguous sensory evidence, which it interprets by making inferences about the hidden states which caused the sensory data. So is the brain an inference engine? The key concept separating Friston's idea from traditional stochastic reinforcement learning methods and even Bayesian reinforcement learning is moving away from goal-directed optimisation. Remember to subscribe! Enjoy the show! 00:00:00 Show teaser intro  00:16:24 Main formalism for FEP  00:28:29 Path Integral  00:30:52 How did we feel talking to friston?  00:34:06 Skit - on cultures (checked, but maybe make shorter)  00:36:02 Friston joins  00:36:33 Main show introduction  00:40:51 Is prediction all it takes for intelligence?  00:48:21 balancing accuracy with flexibility  00:57:36 belief-free vs belief-based; beliefs are crucial   01:04:53 Fuzzy Markov Blankets and Wandering Sets   01:12:37 The Free Energy Principle conforms to itself   01:14:50 useful false beliefs  01:19:14 complexity minimization is the heart of free energy [01:19:14 ]Keith:   01:23:25 An Alpha to tip the scales? Absoute not! Absolutely yes!   01:28:47 FEP applied to brain anatomy   01:36:28 Are there multiple non-FEP forms in the brain?  01:43:11 a positive conneciton to backpropagation   01:47:12 The FEP does not explain the origin of FEP systems   01:49:32 Post-show banter  https://www.fil.ion.ucl.ac.uk/~karl/ #machinelearning

This week Dr. Tim Scarfe, Sayak Paul and Yannic Kilcher speak with Dr. Simon Kornblith from Google Brain (Ph.D from MIT). Simon is trying to understand how neural nets do what they do. Simon was the second author on the seminal Google AI SimCLR paper. We also cover "Do Wide and Deep Networks learn the same things?", "Whats in a Loss function for Image Classification?",  and "Big Self-supervised models are strong semi-supervised learners". Simon used to be a neuroscientist and also gives us the story of his unique journey into ML. 00:00:00 Show Teaser / or "short version" 00:18:34 Show intro 00:22:11 Relationship between neuroscience and machine learning 00:29:28 Similarity analysis and evolution of representations in Neural Networks 00:39:55 Expressability of NNs 00:42:33 Whats in a loss function for image classification 00:46:52 Loss function implications for transfer learning 00:50:44 SimCLR paper  01:00:19 Contrast SimCLR to BYOL 01:01:43 Data augmentation 01:06:35 Universality of image representations 01:09:25 Universality of augmentations 01:23:04 GPT-3 01:25:09 GANs for data augmentation?? 01:26:50 Julia language @skornblith https://www.linkedin.com/in/simon-kornblith-54b2033a/ https://arxiv.org/abs/2010.15327 Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth https://arxiv.org/abs/2010.16402 What's in a Loss Function for Image Classification? https://arxiv.org/abs/2002.05709 A Simple Framework for Contrastive Learning of Visual Representations https://arxiv.org/abs/2006.10029 Big Self-Supervised Models are Strong Semi-Supervised Learners

In this special edition, Dr. Tim Scarfe, Yannic Kilcher and Keith Duggar speak with Gary Marcus and Connor Leahy about GPT-3. We have all had a significant amount of time to experiment with GPT-3 and show you demos of it in use and the considerations. Note that this podcast version is significantly truncated, watch the youtube version for the TOC and experiments with GPT-3 https://www.youtube.com/watch?v=iccd86vOz3w

This week Dr. Tim Scarfe, Dr. Keith Duggar and Yannic Kilcher discuss multi-arm bandits and pure exploration with Dr. Wouter M. Koolen, Senior Researcher, Machine Learning group, Centrum Wiskunde & Informatica. Wouter specialises in machine learning theory, game theory, information theory, statistics and optimisation. Wouter is currently interested in pure exploration in multi-armed bandit models, game tree search, and accelerated learning in sequential decision problems. His research has been cited 1000 times, and he has been published in NeurIPS, the number 1 ML conference 14 times as well as lots of other exciting publications. Today we are going to talk about two of the most studied settings in control, decision theory, and learning in unknown environment which are the multi-armed bandit (MAB) and reinforcement learning (RL) approaches - when can an agent stop learning and start exploiting using the knowledge it obtained - which strategy leads to minimal learning time 00:00:00 What are multi-arm bandits/show trailer 00:12:55 Show introduction 00:15:50 Bandits  00:18:58 Taxonomy of decision framework approaches  00:25:46 Exploration vs Exploitation  00:31:43 the sharp divide between modes  00:34:12 bandit measures of success  00:36:44 connections to reinforcement learning  00:44:00 when to apply pure exploration in games  00:45:54 bandit lower bounds, a pure exploration renaissance  00:50:21 pure exploration compiler dreams  00:51:56 what would the PX-compiler DSL look like  00:57:13 the long arms of the bandit  01:00:21 causal models behind the curtain of arms  01:02:43 adversarial bandits, arms trying to beat you  01:05:12 bandits as an optimization problem  01:11:39 asymptotic optimality vs practical performance  01:15:38 pitfalls hiding under asymptotic cover  01:18:50 adding features to bandits  01:27:24 moderate confidence regimes   01:30:33 algorithms choice is highly sensitive to bounds  01:46:09 Post script: Keith interesting piece on n quantum  http://wouterkoolen.info https://www.cwi.nl/research-groups/ma... #machinelearning

This week Dr. Tim Scarfe, Dr. Keith Duggar, Yannic Kilcher and Connor Leahy cover a broad range of topics, ranging from academia, GPT-3 and whether prompt engineering could be the next in-demand skill, markets and economics including trading and whether you can predict the stock market, AI alignment, utilitarian philosophy, randomness and intelligence and even whether the universe is infinite!  00:00:00 Show Introduction  00:12:49 Academia and doing a Ph.D  00:15:49 From academia to wall street  00:17:08 Quants -- smoke and mirrors? Tail Risk  00:19:46 Previous results dont indicate future success in markets  00:23:23 Making money from social media signals?  00:24:41 Predicting the stock market  00:27:20 Things which are and are not predictable  00:31:40 Tim postscript comment on predicting markets  00:32:37 Connor take on markets  00:35:16 As market become more efficient..  00:36:38 Snake oil in ML  00:39:20 GPT-3, we have changed our minds  00:52:34 Prompt engineering a new form of software development?  01:06:07 GPT-3 and prompt engineering  01:12:33 Emergent intelligence with increasingly weird abstractions  01:27:29 Wireheading and the economy  01:28:54 Free markets, dragon story and price vs value  01:33:59 Utilitarian philosophy and what does good look like?  01:41:39 Randomness and intelligence  01:44:55 Different schools of thought in ML  01:46:09 Is the universe infinite?  Thanks a lot for Connor Leahy for being a guest on today's show. https://twitter.com/NPCollapse -- you can join his EleutherAI community discord here: https://discord.com/invite/vtRgjbM

#machinelearning This week Dr. Tim Scarfe, Dr. Keith Duggar and Yannic Kilcher speak with veteran NLU expert Dr. Walid Saba.  Walid is an old-school AI expert. He is a polymath, a neuroscientist, psychologist, linguist,  philosopher, statistician, and logician. He thinks the missing information problem and lack of a typed ontology is the key issue with NLU, not sample efficiency or generalisation. He is a big critic of the deep learning movement and BERTology. We also cover GPT-3 in some detail in today's session, covering Luciano Floridi's recent article "GPT‑3: Its Nature, Scope, Limits, and Consequences" and a commentary on the incredible power of GPT-3 to perform tasks with just a few examples including the Yann LeCun commentary on Facebook and Hackernews.  Time stamps on the YouTube version 0:00:00 Walid intro  00:05:03 Knowledge acquisition bottleneck  00:06:11 Language is ambiguous  00:07:41 Language is not learned  00:08:32 Language is a formal language  00:08:55 Learning from data doesn’t work   00:14:01 Intelligence  00:15:07 Lack of domain knowledge these days  00:16:37 Yannic Kilcher thuglife comment  00:17:57 Deep learning assault  00:20:07 The way we evaluate language models is flawed  00:20:47 Humans do type checking  00:23:02 Ontologic  00:25:48 Comments On GPT3  00:30:54 Yann lecun and reddit  00:33:57 Minds and machines - Luciano  00:35:55 Main show introduction  00:39:02 Walid introduces himself  00:40:20 science advances one funeral at a time  00:44:58 Deep learning obsession syndrome and inception  00:46:14 BERTology / empirical methods are not NLU  00:49:55 Pattern recognition vs domain reasoning, is the knowledge in the data  00:56:04 Natural language understanding is about decoding and not compression, it's not learnable.  01:01:46 Intelligence is about not needing infinite amounts of time  01:04:23 We need an explicit ontological structure to understand anything  01:06:40 Ontological concepts  01:09:38 Word embeddings  01:12:20 There is power in structure  01:15:16 Language models are not trained on pronoun disambiguation and resolving scopes  01:17:33 The information is not in the data  01:19:03 Can we generate these rules on the fly? Rules or data?  01:20:39 The missing data problem is key  01:21:19 Problem with empirical methods and lecunn reference  01:22:45 Comparison with meatspace (brains)  01:28:16 The knowledge graph game, is knowledge constructed or discovered  01:29:41 How small can this ontology of the world be?  01:33:08 Walids taxonomy of understanding  01:38:49 The trend seems to be, less rules is better not the othe way around?  01:40:30 Testing the latest NLP models with entailment  01:42:25 Problems with the way we evaluate NLP  01:44:10 Winograd Schema challenge  01:45:56 All you need to know now is how to build neural networks, lack of rigour in ML research  01:50:47 Is everything learnable  01:53:02  How should we elevate language systems?  01:54:04 10 big problems in language (missing information)  01:55:59 Multiple inheritance is wrong  01:58:19 Language is ambiguous  02:01:14 How big would our world ontology need to be?  02:05:49 How to learn more about NLU  02:09:10 AlphaGo  Walid's blog: https://medium.com/@ontologik LinkedIn: https://www.linkedin.com/in/walidsaba/