Optimizing AI Behavior: Customization and Fine-Tuning Explained

Nathan presents a comprehensive AI automation framework developed over three years, applicable to process automation and generative AI integration. This episode of The Cognitive Revolution, offers critical insights for businesses looking to leverage AI effectively. Learn about choosing AI tasks, understanding work deeply, and optimizing AI performance in the wake of GPT-4 fine-tuning launch.

Slide Deck

Waymark's case study on GPT-3 fine-tuning

Apply to join over 400 founders and execs in the Turpentine Network

RECOMMENDED PODCAST:

1 to 100 | Hypergrowth Companies Worth Joining

Every week we sit down with the founder of a hyper-growth company you should consider joining. Our goal is to give you the inside story behind breakout, early stage companies potentially worth betting your career on. This season, discover how the founders of Modal Labs, Clay, Mercor, and more built their products, cultures, and companies.

Apple: https://podcasts.apple.com/podcast/id1762756034

Spotify:https://open.spotify.com/show/70NOWtWDY995C8qDqojxGw

History 102

Every week, creator of WhatifAltHist Rudyard Lynch and Erik Torenberg cover a major topic in history in depth -- in under an hour.

Spotify: https://open.spotify.com/show/36Kqo3BMMUBGTDo1IEYihm

Apple: https://podcasts.apple.com/us/podcast/history-102-with-whatifalthists-rudyard-lynch-and/id1730633913

YouTube: https://www.youtube.com/@History102-qg5oj

SPONSORS:

Building an enterprise-ready SaaS app? WorkOS has got you covered with easy-to-integrate APIs for SAML, SCIM, and more. Join top startups like Vercel, Perplexity, Jasper & Webflow in powering your app with WorkOS. Enjoy a free tier for up to 1M users! Start now at https://bit.ly/WorkOS-Turpentine-Network

Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive

The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR

Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/

80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/speak to accelerate your career and contribute to solving pressing AI-related issues.

CHAPTERS:

(00:00:00) About the Show

(00:00:22) Sponsor: WorkOS

(00:01:22) About the Episode

(00:04:20) Introduction to AI Automation

(00:13:43) Current AI Capabilities

(00:17:24) Sponsors: Oracle | Brave

(00:19:28) 3 Ways to work with AI

(00:25:10) Choosing Work for AI to Do

(00:31:18) Sponsors: Omneky | 80000 hours

(00:33:20) What AI Can Do vs. Can't Do

(00:47:11) Understanding the Work

(00:52:58) Documenting the Work

(01:00:50) Optimising AI Performance

(01:02:14) Prompt Engineering Gold Standards

(01:07:05) Optimising Information: Retrieval Augmented Generation

(01:09:29) Optimising AI Behaviour: Fine-tuning AI Models

(01:50:38) Outro