In episode 64 of The Gradient Podcast, Daniel Bashir speaks to Richard Socher.Richard is founder and CEO of you.com, a new search engine that lets you personalize your search workflow and eschews tracking and invasive ads. Richard was previously Chief Scientist at Salesforce where he led work on fundamental and applied research, product incubation, CRM search, customer service automation and a cross-product AI platform. He was an adjunct professor at Stanford’s CS department as well as founder and CEO/CTO of MetaMind, which was acquired by Salesforce in 2016. He received his PhD from Stanford’s CS Department in 2014.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:20) Richard Socher origin story + time at Metamind, Salesforce (AI Economist, CTRL, ProGen)* (22:00) Why Richard advocated for deep learning in NLP* (27:00) Richard’s perspective on language* (32:20) Is physical grounding and language necessary for intelligence?* (40:10) Frankfurtian b******t and language model utterances as truth* (47:00) Lessons from Salesforce Research* (53:00) Balancing fundamental research with product focus* (57:30) The AI Economist + how should policymakers account for limitations?* (1:04:50) you.com, the chatbot wars, and taking on search giants* (1:13:50) Re-imagining the vision for and components of a search engine* (1:18:00) The future of generative models in search and the internet* (1:28:30) Richard’s advice for early-career technologists* (1:37:00) OutroLinks:* Richard’s Twitter * YouChat by you.com* Careers at you.com* Papers mentioned* Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions* Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank* Grounded Compositional Semantics for Finding and Describing Images with Sentences* The AI Economist* ProGen* CTRL Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 63 of The Gradient Podcast, Daniel Bashir speaks to Joe Edelman.Joe developed the meaning-based organizational metrics at Couchsurfing.com, then co-founded the Center for Humane Technology with Tristan Harris, and coined the term “Time Well Spent” for a family of metrics adopted by teams at Facebook, Google, and Apple. Since then, he's worked on the philosophical underpinnings for new business metrics, design methods, and political movements. The central idea is to make people's sources of meaning explicit, so that how meaningful or meaningless things are can be rigorously accounted for. His previous career was in HCI and programming language design.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast: Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro (yes Daniel is trying a new intro format)* (01:30) Joe’s origin story* (07:15) Revealed preferences and personal meaning, recommender systems* (12:30) Is using revealed preferences necessary? * (17:00) What are values and how do you detect them? * (24:00) Figuring out what’s meaningful to us* (28:45) The decline of spaces and togetherness* (35:00) Individualism and economic/political theory, tensions between collectivism/individualism* (41:00) What it looks like to build spaces, Habitat* (47:15) Cognitive effects of social platforms* (51:45) Atomized communication, re-imagining chat apps* (55:50) Systems for social groups and medium independence* (1:02:45) Spaces being built today* (1:05:15) Joe is building research groups! Get in touch :)* (1:05:40) OutroLinks:* Joe's 80m lecture on techniques for rebuilding society on meaning (youtube, transcript)* The discord for Rebuilding Meaning—join if you'd like to help build ML models or metrics using the methods discussed* Writing/papers mentioned:* Tech products (that don’t cause depression and war)* Values, Preferences, Meaningful Choice* Social Programming Considered as a Habitat for Groups* Is Anything Worth Maximizing* Joe’s homepage, Twitter, and YouTube page Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 62 of The Gradient Podcast, Daniel Bashir speaks to Ed Grefenstette.Ed is Head of Machine Learning at Cohere and an Honorary Professor at University College London. He previously held research scientist positions at Facebook AI Research and DeepMind, following a stint as co-founder and CTO of Dark Blue Labs. Before his time in industry, Ed worked at Oxford’s Department of Computer Science as a lecturer and Fulford Junior Research Fellow at Somerville College. Ed also received his MSc and DPhil from Oxford’s Computer Science Department.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:18) The Ed Grefenstette Origin Story* (08:15) Distributional semantics and Ed’s PhD research* (14:30) Extending the distributional hypothesis, later Wittgenstein* (18:00) Recovering parse trees in LMs, can LLMs understand communication and not just bare language?* (23:15) LMs capture something about pragmatics, proxies for grounding and pragmatics* (25:00) Human-in-the-loop training and RLHF—what is the essential differentiator? * (28:15) A convolutional neural network for modeling sentences, relationship to attention* (34:20) Difficulty of constructing supervised learning datasets, benchmark-driven development* (40:00) Learning to Transduce with Unbounded Memory, Neural Turing Machines* (47:40) If RNNs are like finite state machines, where are transformers? * (51:40) Cohere and why Ed joined* (56:30) Commercial applications of LLMs and Cohere’s product* (59:00) Ed’s reply to stochastic parrots and thoughts on consciousness* (1:03:30) Lessons learned about doing effective science* (1:05:00) Where does scaling end? * (1:07:00) Why Cohere is an exciting place to do science* (1:08:00) Ed’s advice for aspiring ML {researchers, engineers, etc} and the role of communities in science* (1:11:45) Cohere for AI plug!* (1:13:30) OutroLinks:* Ed’s homepage and Twitter* (some of) Ed’s Papers* Experimental support for a categorical compositional distributional model of meaning* Multi-step regression learning* “Not not bad” is not “bad”* Towards a formal distributional semantics* A CNN for modeling sentences* Teaching machines to read and comprehend* Reasoning about entailment with neural attention* Learning to Transduce with Unbounded Memory* Teaching Artificial Agents to Understand Language by Modelling Reward* Other things mentioned* Large language models are not zero-shot communicators (Laura Ruis + others and Ed)* Looped Transformers as Programmable Computers and our Update 43 covering this paper* Cohere and Cohere for AI (+ earlier episode w/ Sara Hooker on C4AI)* David Chalmers interview on AI + consciousness Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 61 of The Gradient Podcast, Daniel Bashir speaks to Ken Liu.Ken is an author of speculative fiction. A winner of the Nebula, Hugo, and World Fantasy awards, he is the author of silkpunk epic fantasy series Dandelion Dynasty and short story collections The Paper Menagerie and Other Stories and The Hidden Girl and Other Stories. Prior to writing full-time, Ken worked as a software engineer, corporate lawyer, and litigation consultant.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:00) How Ken Liu became Ken Liu: A Saga* (03:10) Time in the tech industry, interest in symbolic machines* (04:40) Determining what stories to write, (07:00) art as failed communication* (07:55) Law as creating abstract machines, importance of successful communication, stories in law* (13:45) Misconceptions about science fiction* (18:30) How we’ve been misinformed about literature and stories in school, stories as expressing multivalent truths, Dickens on narration (29:00)* (31:20) Stories as imposing structure on the world* (35:25) Silkpunk as aesthetic and writing approach* (39:30) If modernity is a translated experience, what is it translated from? Alternative sources for the American pageant* (47:30) The value of silkpunk for technologists and building the future* (52:40) The engineer as poet* (59:00) Technology language as constructing societies, what it is to be a technologist* (1:04:00) The technology of language* (1:06:10) The Google Wordcraft Workshop and co-writing with LaMDA* (1:14:10) Possibilities and limitations of LMs in creative writing* (1:18:45) Ken’s short fiction* (1:19:30) Short fiction as a medium* (1:24:45) “The Perfect Match” (from The Paper Menagerie and other stories)* (1:34:00) Possibilities for better recommender systems* (1:39:35) “Real Artists” (from The Hidden Girl and other stories)* (1:47:00) The scaling hypothesis and creativity* (1:50:25) “The Gods have not died in vain” & Moore’s Proof epigraph (The Hidden Girl)* (1:53:10) More of The Singularity Trilogy (The Hidden Girl)* (1:58:00) The role of science fiction today and how technologists should engage with stories* (2:01:53) OutroLinks:* Ken’s homepage* The Dandelion Dynasty Series: Speaking Bones is out in paperback* Books/Stories/Projects Mentioned* “Evaluative Soliloquies” in Google Wordcraft* The Paper Menagerie and Other Stories* The Hidden Girl and Other Stories Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 60 of The Gradient Podcast, Daniel Bashir speaks to Hattie Zhou.Hattie is a PhD student at the Université de Montréal and Mila. Her research focuses on understanding how and why neural networks work, based on the belief that the performance of modern neural networks exceeds our understanding and that building more capable and trustworthy models requires bridging this gap. Prior to Mila, she spent time as a data scientist at Uber and did research with Uber AI Labs.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (01:55) Hattie’s Origin Story, Uber AI Labs, empirical theory and other sorts of research* (10:00) Intro to the Lottery Ticket Hypothesis & Deconstructing Lottery Tickets* (14:30) Lottery tickets as lucky initialization* (17:00) Types of masking and the “masking is training” claim* (24:00) Type-0 masks and weight evolution over long training trajectories* (27:00) Can you identify good masks or training trajectories a priori?* (29:00) The role of signs in neural net initialization* (35:27) The Supermask* (41:00) Masks to probe pretrained models and model steerability* (47:40) Fortuitous Forgetting in Connectionist Networks* (54:00) Relationships to other work (double descent, grokking, etc.)* (1:01:00) The iterative training process in fortuitous forgetting, scale and value of exploring alternatives* (1:03:35) In-Context Learning and Teaching Algorithmic Reasoning* (1:09:00) Learning + algorithmic reasoning, prompting strategy* (1:13:50) What’s happening with in-context learning?* (1:14:00) Induction heads* (1:17:00) ICL and gradient descent* (1:22:00) Algorithmic prompting vs discovery* (1:24:45) Future directions for algorithmic prompting* (1:26:30) Interesting work from NeurIPS 2022* (1:28:20) Hattie’s perspective on scientific questions people pay attention to, underrated problems* (1:34:30) Hattie’s perspective on ML publishing culture* (1:42:12) OutroLinks:* Hattie’s homepage and Twitter* Papers* Deconstructing Lottery Tickets: Zeros, signs, and the Supermask* Fortuitous Forgetting in Connectionist Networks* Teaching Algorithmic Reasoning via In-context Learning Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 59 of The Gradient Podcast, Daniel Bashir speaks to Professor Kyunghyun Cho.Professor Cho is an associate professor of computer science and data science at New York University and CIFAR Fellow of Learning in Machines & Brains. He is also a senior director of frontier research at the Prescient Design team within Genentech Research & Early Development. He was a research scientist at Facebook AI Research from 2017-2020 and a postdoctoral fellow at University of Montreal under the supervision of Prof. Yoshua Bengio after receiving his MSc and PhD degrees from Aalto University. He received the Samsung Ho-Am Prize in Engineering in 2021.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:15) How Professor Cho got into AI, going to Finland for a PhD* (06:30) Accidental and non-accidental parts of Prof Cho’s journey, the role of timing in career trajectories* (09:30) Prof Cho’s M.Sc. thesis on Restricted Boltzmann Machines* (17:00) The state of autodiff at the time* (20:00) Finding non-mainstream problems and examining limitations of mainstream approaches, anti-dogmatism, Yoshua Bengio appreciation* (24:30) Detaching identity from work, scientific training* (26:30) The rest of Prof Cho’s PhD, the first ICLR conference, working in Yoshua Bengio’s lab* (34:00) Prof Cho’s isolation during his PhD and its impact on his work—transcending insecurity and working on unsexy problems* (41:30) The importance of identifying important problems and developing an independent research program, ceiling on the number of important research problems* (46:00) Working on Neural Machine Translation, Jointly Learning to Align and Translate* (1:01:45) What RNNs and earlier NN architectures can still teach us, why transformers were successful* (1:08:00) Science progresses gradually* (1:09:00) Learning distributed representations of sentences, extending the distributional hypothesis* (1:21:00) Difficulty and limitations in evaluation—directions of dynamic benchmarks, trainable evaluation metrics* (1:29:30) Mixout and AdapterFusion: fine-tuning and intervening on pre-trained models, pre-training as initialization, destructive interference* (1:39:00) Analyzing neural networks as reading tea leaves* (1:44:45) Importance of healthy skepticism for scientists* (1:45:30) Language-guided policies and grounding, vision-language navigation* (1:55:30) Prof Cho’s reflections on 2022* (2:00:00) Obligatory ChatGPT content* (2:04:50) Finding balance* (2:07:15) OutroLinks:* Professor Cho’s homepage and Twitter* Papers* M.Sc. thesis and PhD thesis* NMT and attention* Properties of NMT, * Learning Phrase Representations* Neural machine translation by jointly learning to align and translate * More recent work* Learning Distributed Representations of Sentences from Unlabelled Data* Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models* Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes’ Rule* AdapterFusion: Non-Destructive Task Composition for Transfer Learning Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 58 of The Gradient Podcast, Daniel Bashir speaks to Professor Steve Miller.Steve is a Professor Emeritus of Information Systems at Singapore Management University. Steve served as Founding Dean for the SMU School of Information Systems, and established and developed the technology core of SIS research and project capabilities in Cybersecurity, Data Management & Analytics, Intelligent Systems & Decision Analytics, and Software & Cyber-Physical Systems, as well as the management science oriented capability in Information Systems & Management. Steve works closely with a number of Singapore government ministries and agencies via steering committees, advisory boards, and advisory appointments. Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:40) Steve’s evolution of interests in AI, time in academia and industry* (05:15) How different is this “industrial revolution”?* (10:00) What new technologies enable, the human role in technology’s impact on jobs* (11:35) Automation and augmentation and the realities of integrating new technologies in the workplace* (21:50) Difficulties of applying AI systems in real-world contexts* (32:45) Re-calibrating human work with intelligent machines* (39:00) Steve’s thinking on the nature of human/machine intelligence, implications for human/machine hybrid work* (47:00) Tradeoffs in using ML systems for automation/augmentation* (52:40) Organizational adoption of AI and speed* (1:01:55) Technology adoption is more than just a technology problem* (1:04:50) Progress narratives, “safe to speed”* (1:10:27) OutroLinks:* Steve’s SMU Faculty Profile and Google Scholar* Working with AI by Steve Miller and Tom Davenport Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 57 of The Gradient Podcast, Andrey Kurenkov speaks to Blair Attard-Frost.Note: this interview was recorded 8 months ago, and some aspects of Canada’s AI strategy have changed since then. It is still a good overview of AI governance and other topics, however.Blair is a PhD Candidate at the University of Toronto’s Faculty of Information who researches the governance and management of artificial intelligence. More specifically, they are interested in the social construction of intelligence, unintelligence, and artificial intelligence, the relationship between organizational values and AI use, and the political economy, governance, and ethics of AI value chains. They integrate perspectives from service sciences, cognitive sciences, public policy, information management, and queer studies for their research.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter or MastodonOutline:* Intro* Getting into AI research* What is AI governance* Canada’s AI strategy* Other interestsLinks:* Once a promising leader, Canada’s artificial-intelligence strategy is now a fragmented laggard* The Ethics of AI Business Practices: A Review of 47 Guidelines Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 56 of The Gradient Podcast, Daniel Bashir speaks to Linus Lee. Linus is an independent researcher interested in the future of knowledge representation and creative work aided by machine understanding of language. He builds interfaces and knowledge tools that expand the domain of thoughts we can think and qualia we can feel. Linus has been writing online since 2014–his blog boasts half a million words–and has built well over 100 side projects. He has also spent time as a software engineer at Replit, Hack Club, and Spensa, and was most recently a Researcher in Residence at Betaworks in New York. Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:00) Linus’s background and interests, vision-language models* (07:45) Embodiment and limits for text-image* (11:35) Ways of experiencing the world* (16:55) Origins of the handle “thesephist”, languages* (25:00) Math notation, reading papers* (29:20) Operations on ideas* (32:45) Overview of Linus’s research and current work* (41:30) The Oak and Ink languages, programming languages* (49:30) Personal search engines: Monocle and Reverie, what you can learn from personal data* (55:55) Web browsers as mediums for thought* (1:01:30) This AI Does Not Exist* (1:03:05) Knowledge representation and notational intelligence* Notation vs language* (1:07:00) What notation can/should be* (1:16:00) Inventing better notations and expanding human intelligence* (1:23:30) Better interfaces between humans and LMs to provide precise control, inefficiency prompt engineering* (1:33:00) Inexpressible experiences* (1:35:42) Linus’s current work using latent space models* (1:40:00) Ideas as things you can hold* (1:44:55) Neural nets and cognitive computing* (1:49:30) Relation to Hardware Lottery and AI accelerators* (1:53:00) Taylor Swift Appreciation Session, mastery and virtuosity* (1:59:30) Mastery/virtuosity and interfaces / learning curves* (2:03:30) Linus’s stories, the work of fiction* (2:09:00) Linus’s thoughts on writing* (2:14:20) A piece of writing should be focused* (2:16:15) On proving yourself* (2:28:00) OutroLinks:* Linus’s Twitter and website Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 55 of The Gradient Podcast, Daniel Bashir speaks to Professor Suresh Venkatasubramanian. Professor Venkatasubramanian is a Professor of Computer Science and Data Science at Brown University, where his research focuses on algorithmic fairness and the impact of automated decision-making systems in society. He recently served as Assistant Director for Science and Justice in the White House Office of Science and Technology Policy, where he co-authored the Blueprint for an AI Bill of Rights.Have suggestions for future podcast guests (or other feedback)? Let us know here!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:25) Suresh’s journey into AI and policymaking* (08:00) The complex graph of designing and deploying “fair” AI systems* (09:50) The Algorithmic Lens* (14:55) “Getting people into a room” isn’t enough* (16:30) Failures of incorporation* (21:10) Trans-disciplinary vs interdisciplinary, the limiting nature of “my lane” / “your lane” thinking, going beyond existing scientific and philosophical ideas* (24:50) The trolley problem is annoying, its usefulness and limitations* (25:30) Breaking the frame of a discussion, self-driving doesn’t fit into the parameters of the trolley problem* (28:00) Acknowledging frames and their limitations* (29:30) Social science’s inclination to critique, flaws and benefits of solutionism* (30:30) Computer security as a model for thinking about algorithmic protections, the risk of failure in policy* (33:20) Suresh’s work on recourse* (38:00) Kantian autonomy and the value of recourse, non-Western takes and issues with individual benefit/harm as the most morally salient question* (41:00) Community as a valuable entity and its implications for algorithmic governance, surveillance systems* (43:50) How Suresh got involved in policymaking / the OSTP* (46:50) Gathering insights for the AI Bill of Rights Blueprint* (51:00) One thing the Bill did miss… Struggles with balancing specificity and vagueness in the Bill* (54:20) Should “automated system” be defined in legislation? Suresh’s approach and issues with the EU AI Act* (57:45) The danger of definitions, overlap with chess world controversies* (59:10) Constructive vagueness in law, partially theorized agreements* (1:02:15) Digital privacy and privacy fundamentalism, focus on breach of individual autonomy as the only harm vector* (1:07:40) GDPR traps, the “legacy problem” with large companies and post-hoc regulation* (1:09:30) Considerations for legislating explainability* (1:12:10) Criticisms of the Blueprint and Suresh’s responses* (1:25:55) The global picture, AI legislation outside the US, legislation as experiment* (1:32:00) Tensions in entering policy as an academic and technologist* (1:35:00) Technologists need to learn additional skills to impact policy* (1:38:15) Suresh’s advice for technologists interested in public policy* (1:41:20) OutroLinks:* Suresh is on Mastodon @geomblog@mastodon.social (and also Twitter)* Suresh’s blog* Blueprint for an AI Bill of Rights* Papers* Fairness and abstraction in sociotechnical systems* A comparative study of fairness-enhancing interventions in machine learning* The Philosophical Basis of Algorithmic Recourse* Runaway Feedback Loops in Predictive Policing Get full access to The Gradient at thegradientpub.substack.com/subscribe