Running Large AI Models: Hardware Insights

In this Marketing Over Coffee:

Learn about Snoop at the Olympics, SearchGPT, Dyson Headphones and more!

Direct Link to File

Brought to you by our sponsors: Wix Studio and NetSuite

Olympic watch: Snoop Dogg carrying the torch, Flava Flav supporting Water Polo

Llama 3.1 released to the public – what it takes for you to run it yourself

7:07 – 7:54 Wix Studio is the web platform that gives agencies and enterprises the end-to-end efficiency to design, develop and deliver exactly the way they want to!

SearchGPT is coming for search traffic

Google pays to access reddit data

13:52 – 15:19 NetSuite is the number one cloud financial system, bringing accounting, financial management, inventory, HR, into ONE platform, and ONE source of truth.

Google gives up on ditching 3rd party cookies, AdTech execs say it doesn’t matter anyway

Hidden Google tool to find discrepancies between GA4 and Google Ads conversion data (conversions vs. key events)

Dyson back with more headphones – 40db noise reduction, 55 hour battery

Deadpool vs. Wolverine

Gen AI Course Updates done: Special Discount on the newest Generative AI for Marketing Course! Hands on excercises to put AI to work for you! USE CODE MOC now!

Join John, Chris and Katie on threads, or on LinkedIn: Chris, John, and Katie

Sign up for the Marketing Over Coffee Newsletter to get early access!

Our theme song is Mellow G by Fonkmasters.

Machine-Generated Transcript

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.

John Wall – 00:00
Today’s episode is brought to you by Netsuite and Wix Studio.

Speaker 2 – 00:10
This is marketing over coffee with Christopher Penn and John Wall.

John Wall – 00:17
Good morning. Welcome to marketing over coffee. I’m John Wall. Christopher Penn and I have to lead with Snoop Dogg at the Olympics. I think this has just been an amazing thing that he’s done. I mean, my kids were watching Snoop and Kevin Hart do Olympic commentary from four years ago on YouTube, months, two or three months ago. And so they were even, they were all excited. They’re like, “Snoop’s going to the Olympics.” Yeah. I don’t know. It’s just kind of amazing how he’s become this american icon or treasure. I mean, do you have any thoughts on that? And what does it mean? Or is it. Does it matter?

Christopher Penn – 00:54
It’s amazing marketing, if you think about it. It is amazing marketing. And talk about a rebrand. He has somehow managed to rebrand himself basically as a non sex offender. Bill Cosby.

John Wall – 01:08
Yeah, right. I’ve seen that joke a number of times where they have ice cube and Bill Cosby, and it says something like, “Who would have believed 30 years ago that one of these people would be a convicted felon and another would be a leading provider of children’s entertainment?”

Christopher Penn – 01:22
Exactly. Someone also did point out, I saw a meme on Instagram pointing out the uncanny resembles of the Olympic torch. That’s new for carrying to a certain thing you smoke.

John Wall – 01:33
Yes, I know. It’s like he was waving the torch, as it were. A similar one, too, which didn’t get as much press, but I thought was also great is flavor Flav, the hype man from public enemy sponsoring the men’s and women’s water polo team for five years. He has gone all in. And just an amazing story heard about how, members of the team who have already won in the Olympics are having to work side jobs and do all kinds of stuff to make this possible. He has thrown in some money to make it easier for them. So, yeah, I wanted to lead off with that because I think that’s fun. But also, huge news, though, on the AI front. I think the big one that you had talked about earlier was meta releasing a model. That is, the whole entire model is out there. Tell us more about that.

John Wall – 02:12
That is, the whole entire model is out there. Tell us more about that.

Christopher Penn – 02:16
Yeah, so meta creates this model family called Lama llama two, three, and now 3.1. These are open weights models. And what that means is if you use a service like chat, GPT, or Gemini, you use the services in a web browser. You can’t get at the underlying engine that makes it work. You can’t tune it beyond a certain point, and so on and so forth. With Lama, this is a file, big file that you download, and if you have the right hardware and software, you can run it yourself. Which means that on your laptop, if you have like a high end MacBook, you can run two of the three llama models. And you need a server for the really big one.

You can run these things and have your own generative AI. Like you could unplug the Internet and just turn it off and it would still work because it’s a self contained engine. The big llama model, the llama 3.14 or five b, you can clearly tell these are not named by marketers, is so capable that it is a peer to chat GPT’s model to Google Gemini to anthropic Claude. Which is insane when you think about it, because it means that your organization, if you have the budget to buy the hardware, you could run this in your company, and then generative AI is yours forever.

No one can take it away if OpenAI goes out of business tomorrow because they’re burning cash like crazy. If anthropic goes out of business tomorrow, you still have access to state of the art gender AI through these models. And there was just an interview yesterday morning with Meta’s head of model development saying llama four just started training last month. They expect to take a full year of training. It’s going to have even more data going into it, and it will have tool usage natively built into the model in an agent way, so you won’t have to have third party add ons. It will be able to natively go out and search the web, it will natively be able to go and do a bunch of different things. So what these things are doing are like meta is releasing just state of the art stuff that anyone can download for free.

John Wall – 04:20
Yeah. And what are the cause? I noticed that is tiered. I mean, if you were to think about wanting to do this yourself, what do you need to be thinking about as far as hardware and which levels can you take advantage of? Like how does that all work?

Christopher Penn – 04:32
It goes by memory. How much video memory, your gpu, your graphics card has the best platform to run these models on, believe it or not, is actually a MACD, because Macs have shared memory between regular computer memory and video memory. Whereas with Windows machines you have a gpu, a graphics card, and then that has its own dedicated memory. The rule of thumb is roughly 1.5gb of ram per 2 billion parameters. So the llama model comes three flavors, 8 billion, 70 billion, or 405 billion parameters. The 8 billion parameter model needs four and a half 5gb of videograph. Any graphics card can use that. If you have the built in one that comes with like a mid range laptop, you can run that, no problem.

The 70 billion parameter model requires about 40gb of video ram. So you’re talking like a really nice graphic card. Nvidia RTX 4080 4070. Obviously Macs and things like that, and it’s going to heat up the room, but mid range, like the M one Max MacBook from a few years ago that can run the 70 billion paramount. You won’t be able to do anything else on the laptop, but it will run that. The 405 billion parameter needs about 250 or 300gb of video ram. The current Mac studio, I think if you max it out on ram, it can run that. Otherwise you have to buy and or build a rack for that. So I have seen on Reddit people have a server rack with like eight graphics cards slotted into a main board, stuff like that. It’s like 6000 watts of power that whatever room they put it in needs like Industrial AC. That’s the level of hardware you need to run the big one. That model is really intended for companies.

The ideal use case, believe it or not, for the kind of company that would want, that would be like a three letter government agency where you want the power generator. But under no circumstances can you let that data ever leave your network and you want on a computer that isn’t connected to the Internet. And this is what these things allow is, right. You could put on a, I’m sure in Langley somewhere there’s a GPU cluster and they’ve downloaded Lambda 405. And now they’re like, “Oh yeah. Here’s all of our agents field notes for the, this particular investigation. Let’s put it into the secure air gapped computer and have the model work on it and then come back with answers.” And then, someone, takes off a flat, secure flash drive and brings it up to, upstairs to the rest of the office. But that’s the kind of scenario where you would want that really big model for data that you just cannot, under any circumstances let a third party have, no matter what the service level agreement says.

John Wall – 07:06
Right. Okay, well, we just have to take a second. We want to thank Wix Studio for their support of marketing over coffee. I’ve only got one minute to tell you about Wix Studio, the web platform for agencies and enterprises. Whether you manage ten sites or 1000, here are a few things you can do from start to finish in a minute or less on studio: set up native marketing integrations in a click. Reuse templates, widgets, and sections across sites. Create a seamless handover by adding tutorials, guides, and more to client dashboards. Work on the same canvas at the same time with all your team members, and leverage best in class SEO defaults like server side rendering and automated structured data markup across all your Wix sites. Times up, but the list keeps going. Step into Wix Studio and see for yourself. Check it out over at wix studio.com and we thank them for their support of the show.

So the different flavors, like you said, a government agency would want to go at the top end. Is there really any easy way to choose between those three besides just the horsepower? I mean, basically, are you saying that you just have to live with the current level that you’re able to, drive and support, or is there some other reason?

Christopher Penn – 08:13
If you want to use the models on your own hardware, yes. So you have to decide. So the smaller the model is, the less capable it is. The dumber it is, the less knowledge it has. For most users. For most average tasks, like “Write a blog post”, whatever. The 70 billion parameter model is a pretty good model. It’s very versatile, it’s very capable. If you want state of the art, if you want your own version of chat GPT, the 405 billion parameter model is the one to go for, but you need a lot of horsepower to drive it. But that knowledge, super, that model is super knowledgeable. It’s very smart, it’s very capable. It is up here today’s top models. And when Lambda four comes out, presumably sometime in 2025, that is likely to be appear or exceeding today’s models in terms of capabilities and things. So I’m sure it will also have substantial requirements for hardware, but that the four or five model is the one that meta uses on all of its SaaS services that they run with.

John Wall – 09:11
Okay, we had another thing, big news. Interesting OpenAI crossing over to search, announcement about search GPT. I mean, I don’t know. This does seem to be the search engine’s greatest fear, this is what we’ve seen everybody worrying about. But what do you think? And is it actually publicly available? Have you been able to play with it?

Christopher Penn – 09:29
It’s not publicly available yet. It’s in private beta testing. Clearly it’s gunning for perplexity, which is the AI based search engine that has gotten a decent user base and has a reasonably good reputation, although wired did do a very critical piece on it recently. But it definitely has search engines on the crosshairs. And OpenAI is best positioned because they have the user base from chat GPT. They have 175 million users or 200 million users. And that certainly should be making places like Mountain View and Redmond go, “Huh, I guess we better up our game,” particularly since Google’s AI answers when it rolled out were not well received. So having search GPT basically trying to be a Google substitute is a legitimate potential challenger to Google.

John Wall – 10:15
And we’ve talked in the past about the fact that you need to step up your publishing and publishing stuff that can now be read and taken by these models. Anything else new on that front though, as far as we can be pretty certain that this is going to happen and is going to come along, are there any other ways to prepare for it?

Christopher Penn – 10:33
The biggest thing that we know recently that’s been a controversy all its own is that a lot of these AI companies are all heavily scraping YouTube. Because they need the multimodal capabilities, they need the image data, they obviously need the text data from all the closed captions and things. So the single best thing that your brand could do today is put a crap ton of video content on YouTube that is, that gets decent views. If it gets, one or two views, as long as it’s showing up, that is keyword rich for the topics you care about. That is brand rich. So you want to mention, like in all of my videos, I mention trust insights all the time. I want it.

Anytime I talk about AI, I want the brand name and I want my own name in there. So that when a model, when a scraper comes by and extracts the closed captions, it’s pulling out the brand names and stuff like that. And there it is. You will notice, for example, at the bottom of the Trust Insights blog, if you go to our blog, there’s now new boilerplate. At the bottom it says, “If you’re a human, just skip this, if you’re a machine.” And then it’s, there’s a dense paragraph that I just loaded up that says, “Trust Insights is the world’s preeminent AI consulting, management consulting services,” basically so that every time someone’s scraping the RSS feed, they’re going to get that big whopping chunk of text and that’s going to go into people’s training data as well.

You know, we can certainly do it on marketing over coffee. Marketing coffee is the world’s leading marketing podcast with all, with guests, these guests. And you should always listen to marketing over coffee. Marketing over coffee is available on these devices and so on, so forth. It’s just a wall of text and that wall of text then, because the way to do it, you have to integrate it into the post itself so it shows up in the RSS feed. Then you set your rss feed to show full post, and then every scraper that comes by and vacuums it up is going to vacuum up your stuff. And then when they repost it on the web, spamming the web with this stuff, that gets duplicated and it amplifies the power of essentially your programming for these models.

Those are two. So those are the two things I would do. Number one, YouTube, YouTube. You need to be cranking out content on YouTube. If your blog, if your podcast, if your whatever is not on YouTube, you are missing out. Fix that today. Number two, change your boilerplates on all the texts that you have control over to make sure that it is well seeded with your brand and your key topics so that models can vacuum that up.

John Wall – 12:55
All right, so keyword stuffing back in the mix for everybody that remembered that.

Christopher Penn – 13:00
Back in business, 2006 SEO.

John Wall – 13:02
Here we go, back in the game. So that’s fantastic. We can keep cranking that up. That reminded me of another story. I’ve got a link in the show notes about. I had not realized this, that Reddit, well, I did know that Reddit was turning down all scrapers and Reddit basically said, “Look, you show up with the money, otherwise you don’t get to scrape.” And Google has a deal that people are saying is worth $60 million a year to Reddit for them to actually get access to that, because it’s been interesting.

People have been noticing Google results have been throwing more Reddit product reviews and things like that because with the upvoting, they’re considering that data to better. And I guess that’s even caused some trouble over on the Reddit side. They’ve had to close comments on things to stop people from trying to game reviews of products and things like that because they know that if it’s good over on Reddit, it can give them some extra juice and kind of bump them up another notch.

We also have to take a second. We want to thank Netsuite for their support of marketing over coffee. For all of our clients, there comes a point where they get large enough and they’re managing so many systems that you’re just caught up in the bureaucracy of it all, you’re actually spending more maintaining all this complexity. Smart businesses reduce costs and headaches when they get large enough by graduating to Netsuite by Oracle.

Netsuite is the number one cloud financial system, bringing accounting, financial management, inventory, hr into one platform and one source of truth. With Netsuite, you reduce it costs because Netsuite lives in the cloud with no hardware required access to from anywhere. You cut the cost of maintaining multiple systems because you’ve got one unified business management suite. You improve efficiency by bringing all your major business processes into one platform, slashing manual tasks and errors. Over 37,000 companies have already made the move. So do the math. See how you’ll profit with Netsuite.

Again, we’ve seen it firsthand for our clients. Instead of building all these integrations or running batch reports so that you can get inventory and the financials in order along with the marketing and sales stuff, just get it all one platform. And of course, having it in the cloud makes a whole slew of headaches go away. By popular demand, Netsuite has extended its one of a kind flexible financing program for a few more weeks. Head to Netsuite.com/coffee. That’s Netsuite.com/coffee. Again, netsuite.com/coffee. And we thank Netsuite by Oracle for their support of the show.

Google giving up on ditching third party cookies. This has been, the war cry for years and has been pushed back and pushed back. And now they basically said, “Forget it, we just can’t even do it.” There were a number of marketing and ad tech people that chimed in this article that I read that I’ve got the link to and they’re basically all saying, well, it doesn’t really matter. Like, you still need to be getting away from this. And even though they can’t kill it, that doesn’t change anything that you should be doing. Do you agree with that or is there anything else there?

Christopher Penn – 15:49
Certainly the laws that have sprung up since GDPR, definitely, like you said, to be compliant with the laws, you still have to be phasing out third party cookies. Am I surprised that Google said they’re not getting rid of it when in the report they said it reduces ad performance? Like, no, I mean, it’s like, oh, look, we’re going to make less money if we turn these off, so we’re not going to do that.

John Wall – 16:11
Yeah, they push back. We’ve kind of seen more and more of that as they continue to go. Another hidden tool, I don’t know, over on search engine land, a hidden Google report to find discrepancies between Ga four and Google Ads. So I’ll, if that’s your thing, I’ll throw that in there. You can check that out. Just thought that was interesting. There was something kind of buried in there.

At the heart of that they’re saying was that it’s really about the fact that conversions versus key events are two different things. And they’re saying that’s why that report is hidden, because they don’t want to be calling that out. Is there anything else we should be considering or looking into when we’re talking about conversions versus key events?

Christopher Penn – 16:49
So here’s the thing about Google Analytics. This is not as well known as it should be. Up to 30% of the data that Google Analytics shows isn’t real. With things like consent banners and privacy things and ad blocks and stuff. Google has been at everyone, but Google in particular has been losing data more and more every year.

So what they’ve done is they’ve started using AI to infer the missing data, to guess at the missing data, at what should be there but isn’t. And that means that anytime you’re doing inference, anytime you’re doing any kind of imputation, it’s probabilistic, it’s guesswork. Not only will the numbers be wrong, but they will be unpredictably wrong. You don’t know how the AI is solving, has. Is filling in the blanks. It could be doing mad libs for all we know.

And that, in turn, means that you’re going to have more and more discrepancies in your analytics data as as more devices and more technologies block tracking. And here’s what’s important about that. It’s uneven. Like if all data was missing at random, you can inference that and fill in the gaps. And there’s great ways to do that statistically. When data is missing, not at random, when there’s certain populations that behave differently, suddenly you can’t do that anymore.

So, for example, iPhones are a substantial chunk of missing data because of the way Apple’s privacy policies work. Which means that if you just try to get fill in the blanks naively, you’re going to be guessing performance about a sector that probably behaves differently than Android users, Apple users, android users are, we know, are economically different, they spend differently, they tend to be demographically different.

And so you can’t just, you just can’t just naively guess what the missing iPhone date is. That audience may behave statistically differently. And so Google doesn’t tell us how they’re filling in the blanks. They just say that we are and that the data is what it is. And so you don’t know how reliable it is. I recently did.

My friend Andy Crescendina reached out recently, asked about comparing GA four data to other data sources. And we looked at it. If you go from Cloudflare at the very beginning of the website, the edge, to all the GA four, there’s a 300% difference. Cloudflare shows we have 100,000 users. Google says we have 3000. Granted, a lot of the stuff on the edge is junk like bots and spam bots and stuff like that. But then you go to WP engine, which is our web host, it says we have 10,000 users. And WP engine says, “This is what we’re billing you for.” Like, great.

John Wall – 19:31
Yeah, I noticed that.

Christopher Penn – 19:32
Yeah, exactly. And then you go to Google, it says 3000 users. Okay? So between what gets through Cloudflare is legitimate traffic and what WP engine bills you for, and what GA four says you got for people is a 7000 person difference, 10,000 versus 3000. So you’re like, “What’s going on? Is 70% of our traffic really missing?” And if so, then what’s in Google Analytics has got to be just pure guesswork.

John Wall – 19:58
It’s completely made up all the way. Yeah. It’s like trying to sort through that stuff is an ongoing battle, but not going to in nobody’s interest to make it easier for you for free. That’s for damn sure. We know that. That’s how that runs. All right, we’ve got a gear watch. I had a couple of things. Dyson back with more headphones we had talked about. They had those headphones that were noise reduction and had the bane mask on it. So they’ve got a new thing that does have -40 decibels noise reduction they’re talking about, and a 55 hours battery, which that’s the upside of making those things so big, is that you can have the giant battery strapped on the side and make that work. I was interested. I actually got a pair of broken down Bose QC 35s. They’re noise reduction headphones. QC 35, two from a few years back. And I was amazed.

I went through all the hassle of replacing the pads, cleaning them up, replacing the battery, getting it all to where it needs to be. And then I’m like, “Wow, my beats fit pro really have way better noise reduction then these.” The last three generations have been huge changes in the quality of what’s coming through. Yeah, constantly moving forward. I think we’re overdue for another round of headphones. I mean, both the Apple Air, Apple Max, and the beats fit Pro and the AirPods are all getting along in the tooth as far as product cycle. Normally they drop those for Christmas, but I don’t know what the chip shortages and things like that. Will we actually see them or not?

Christopher Penn – 21:26
The big thing I saw recently was Bose’s new ultra open that supposedly are the open circuit open air headphones. And I have no idea if they’re any good or not, but I. My Instagram will not stop with the advertising on them.

John Wall – 21:38
They know you’re in the bullseye for those. Yeah, I saw those, too. They’re around the $300 price point, and I’m a little bit interested in them, but not enough to throw down 300 when I’ve already got two sets of beat fit Pro. But, yeah, maybe we should dig a little bit deeper. I could find somebody at Bose who might be willing to let us take some for a test spin or something, because I do like that idea of an open earbud. The way you can exercise or be walking in traffic and not have to worry about dying. That’s a bonus.

Christopher Penn – 22:07
Yeah, I used the bone conductance for those.

John Wall – 22:09
Yeah. Do you still like those? I’ve talked to people that are like, after an hour, they get to be a little bit tough on your head, but how do you like those?

Christopher Penn – 22:17
Yeah, I mean, I will wear them for hours at a time, but I find them very comfortable. The audio quality is not good. It is good enough to work out to. It is not good. Like, if you want to sit down and enjoy the, like, the Berlin Symphonic Orchestra, these are not the headphones doing. If you’re just running and you need the environmental awareness and you can hear the traffic around you anyway, it doesn’t matter, right? You’re not going to be like, “Oh, I think that cello’s out at you”, and you’re like, “There’s a car.” So they’re good enough for what they they’re intended for. What is nice, though, is that they’re great for airline travel. If you use, like, the super heavy duty foam noise reduction earplugs, you can wear the bone conductions and actually hear pretty decently. But the foam earplugs really do a good job sealing out all the ambient noise. I find that better and less tiring on my ears than active noise cancellation.

John Wall – 23:07
That’s interesting. Yeah. And that’s a great way to go. And, yeah, for anybody out there who hasn’t already gotten that memo, you’ve got to be using noise reduction of some kind on flights. It just like, I literally just feel better when I get to my destination, you’re like, your head is not taking a beating. You don’t realize how much abuse you’re taking. All right, that sounds good. I’ve got a bunch of books in the queue. We have.

Seth Godin actually just sent over a galley of his latest, so I’ll be talking to him. And both Tom Webster and Tamson Webster have books coming up too. I saw a pre order for Tamsen. Tom’s preorder is there too. Tamsen emailed me and so I was able to hit that right away. But I need to get a copy of Tom’s too, if we can get that going. So all interviews that’ll be coming up, hopefully, if they’re willing to come on the show, I’m assuming we’ll take an invite. But most authors are looking for every opportunity they can to promote. So we’ve got that. Deadpool versus Wolverine is on the list. I have not seen it, so I’m still trying to avoid spoilers and I’ve already been undone. I saw a few things online that I didn’t wish I hadn’t heard, but that’s just the way that goes.

Christopher Penn – 24:10
That movie had an incredible amount of p and a spent. So PNA is print and advertisement. And generally speaking, with movies, it’s it’s two to one or one to one, whatever the production cost is. You spend a dollar production, go spend a dollar on promotion. I think they might. They for sure, at least at one to one, if not more, because I’ve seen Hugh Jackman and Ryan Reynolds on literally every possible place you can be. Their hot ones. Interview, by the way, is well worth watching because it’s funny just to see Hugh Jackman cry. But they must have spent at least $200 million on promotion on this film.

John Wall – 24:45
Yeah, well, and it seems like it’s going to go. Yeah, we could go down that rat hole of Marvel too. There’s all kinds of things going on there. Robert Downey Junior announcing his return as Doctor doom and a whole bunch of other things on that front. But yeah, we’ll leave it at that. Oh, I can throw in Star Trek prodigy, too. There’s a new season of that if you’re into that. That’s actually been surprisingly filled with cameos and a lot of other stuff on that front, so that’s worth watching. But otherwise, that’ll do it for us for this week. So until next week, enjoy the coffee.

Christopher Penn – 25:13
Enjoy the coffee.

Speaker 2 – 25:15
You’ve been listening to marketing over coffee. Christopher Penn blogs at christopherspen. Read more from John J. Wall at JW 5150 dot com. The marketing over coffee theme song is called Melo G by funk Masters, and you can find it at musicalley from Mevio. Or follow the link in our show notes.

The post Olympics, Running Your Own AI, and Planning for AI Search appeared first on Marketing Over Coffee Marketing Podcast.