The One Where We Geek Out on Mobile App Observability with Austin Emmons of Embrace Mobile

Jan 28, 2025

Ask episode

Chapters

Transcript

Episode notes

About our guest:

Austin Emmons is an iOS Developer at Embrace Mobile, a company that works on Observability for mobile applications and beyond. Austin has been developing for Apple platforms since the early iOS days. Outside of tech, he enjoys mountain biking, rock climbing, and taking his dog, Nacho, on new adventures.

Find our guest on:

Find us on:

All of our social channels are on bento.me/geekingout
All of Adriana's social channels are on bento.me/adrianamvillela

Show notes:

Additional notes:

Transcript:
ADRIANA: Hey, fellow geeks! Welcome to Geeking out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today, I have Austin Emmons. Welcome, Austin.

AUSTIN: How's it going?

ADRIANA: Not bad. Super happy to have you here.

AUSTIN: Happy to be here. Thanks for having me.

ADRIANA: And where are you calling from?

AUSTIN: I'm based in Pittsburgh, Pennsylvania.

ADRIANA: Awesome. Well, are you ready for the lightning round questions?

AUSTIN: Yeah, let's do it.

ADRIANA: Okay, first question. Are you a lefty or a righty?

AUSTIN: Righty.

ADRIANA: Okay. Do you prefer iPhone or Android?

AUSTIN: I'm an iOS developer, so iPhone. I get tempted every, every time Google comes out with a new Pixel. I'm definitely tempted, but I have to say iPhone.

ADRIANA: Cool. Do you prefer Mac, Linux, or Windows?

AUSTIN: Mac my entire life. And I got made fun of a lot in college when I showed up to computer science with a MacBook and I was just like, well, I'm, I'm, I'm dual booting Windows when I need to, but I would get out of that as soon as possible because I just, I would. Yeah, I had to hack together a lot of stuff just to get Java compiling and everything. And that was, that was fun. But yeah, definitely Mac.

ADRIANA: Oh, damn, that's so cool. Yeah, I don't know, like, when I was in university, if there was like any. Anyone I ever saw with, actually. So when I was in school, there were very few of us with laptops and certainly not, I don't know of anybody who had a Mac at the time because I think they were like, also so expensive.

AUSTIN: Yeah, yeah, no, I, I cut a lot of lawns the summers prior to save up for the first Mac. And when I say hack together some stuff, I just had to, you know, look on the other side of the Internet, I guess, to figure out the. The instructions did not come in the course syllabus like it did for everybody else.

ADRIANA: Yeah, yeah, yeah, that's so cool. Okay, next question. What's your favorite programming language?

AUSTIN: Swift. I really like. It's, it's, it's strict, very strict, but it's also very expressive. And if I need to write something quick or for some personal projects, Ruby would be my, my go-to. I've had a prior life as a Rails developer where I learned a lot of the server side stuff and so Ruby really, you know, opened my eyes to that. Yeah. And yeah, I find like throughout my career it's either like you're a Python shop or a Ruby shop. Somehow I've thread the needle to lean on the Ruby side of things, and now I'm at a Python shop. But I'm an iOS developer, so I don't have to focus on it too much.

ADRIANA: Oh, that's so interesting. It's funny, I've had quite a few Ruby folks on this podcast and the thing that I always find with the Ruby folks is that they really, really love Ruby, which I think it's so cool. I think it speaks to the community.

AUSTIN: It's a. It's just like, it's just simple to me. I don't know, like it just clicks of, okay, that makes sense. And it's maybe not as powerful, but the community for sure is there. Like, it's amazing. And then when you want to tear open somebody's gem or somebody's work, you. You can. And so, yes, it's open source to the fullest, I think, which is awesome.

ADRIANA: That's very cool. It's funny, I was talking to someone because I. You. You were talking about like, you know, as Ruby being your go to if you want to like, throw something together quickly. And I actually had a very similar conversation with someone last week about this. Also interviewing for the podcast and she was saying like, you know, she knows a bunch of languages, but like the one that she always comes home to is, is Ruby. So I thought that was interesting and it's kind of cool hearing it from two people now.

AUSTIN: Yeah, I mean, recommend it to anybody that's trying either getting into programming or even if you've been a seasoned programmer, just try it out. It'll change how you think about programming. But that's any language. If you try out a new. That's what I love about kind of taste testing new languages. It's just like, okay, how do you do a for loop? Even that could be so different. And it's still for loop, but it's just good enough. And Ruby, for me, it was actually the innumerable like package or, you know, they have so many tiny little algorithms or methods that you can use just to map and all those. It's like that was a whole new introduction to me of like, oh, one, I get these for free. That's awesome. One. And then again another, I can chain them together and really do what I need to. And now I go to other languages and I. That's like the first thing I ask for. Look at, it's like, okay, now I need to get back into that more functional type of programming.

ADRIANA: Right.

AUSTIN: Even though Ruby is. Is very object oriented. So, yeah, it's, it's a good language.

ADRIANA: Cool. Yeah. And to what you were saying earlier too, I think it's interesting. Like, I think one of my favorite things about like tackling new languages is that compare and contrast. Right. Because you already, you have the experience like of a base language and it's always so interesting to see how different languages approach different things and how they have their nuances and, you know, if they're more verbose than other languages you've worked with and whatnot.

AUSTIN: Yeah, I, I mean for most of my career I actually had to straddle iOS and Android and that was very similar. Where it wasn't just I'm working in two different languages and I have to compare and contrast. Java at that point and Objective-C on the iOS side. It was actually the platforms themselves. It's like, okay, how do I show full screen content? You know, iOS you call it a view controller. Android, you call it an activity at that point. You still can, but they've kind of shifted the thinking to fragments and, and compose now. And so it's like you had to stay along and be up to date with every change that the platform developers were making on top of the changes of the language. And I would always implement the same feature two or three times where it's like, implement it for iOS first. Okay, that works. Have to implement it for Android. It's the same feature but I have to do it slightly differently. But I did it in a way that I have this experience that I like a little more. Okay, what can I take from that Android side and actually come back to my iOS implementation and improve that a little bit. And if you have the time, that's really beneficial because it just stretches your brain out and yeah, I couldn't recommend it enough.

ADRIANA: That's awesome. That's so cool. So I'm, I think I know the answer, but what, what do you prefer developing on more iOS or Android?

AUSTIN: You know, I recently had, it's iOS but I recently had some work to do. It's probably a year, year and a half ago in Kotlin on Android and I had stepped away from Android for probably three or four years and then come back to it for just a really quick two or three month project, and I loved it. Kotlin and, what they've done, it's just, I don't know, so much more intuitive than Java. It really feels like it is a first party product. In the early days of Android, when I was in my early days, I guess you could feel that it was an open source project and you could feel that the design patterns that they were using were different depending on what part of the platform you're working in. Whereas iOS was, everything is very cohesive. You know the Apple platforms and the frameworks, they provide very common design patterns. And so you knew like it, you felt used to it even though you had never seen this before. So you know, transitioning from requesting some the device location to the device motion, you know, it's almost identical code. On Android, there might have been separate patterns that used and so I think nowadays Android has leveled up and those design patterns are more similar or at least maybe it's just the entire community developing these packages have everybody's leveled up and come together on how they like Android code to look right.

ADRIANA: Oh cool, that's awesome. Definitely seeing an evolution in the right direction on that one.

AUSTIN: Yeah, just established patterns I think would be the, the best way to put it is like those have started to actually like solidify and take shape. And I mean, nowadays it's 10 years, almost 15 years for these platforms. So they're, they're getting up there in terms of the maturity which is interesting. And now we have new stuff to go work on and we'll see what the next frontier is, I guess.

ADRIANA: That's very cool. And by the way like I, I, I want to go back to like your mention of Kotlin. So my dad is, he's like a software architect and he's retired now but he has been like a huge proponent of Kotlin forever. So he always like goes on and on about how much he liked and he was like an early adopter of Java. And then his, his thought around Java was like oh it's, it's like it's an anti pattern to programming just because you know, Java is like so, so verbose and so like heavy. And then when he, he, he did some, some Kotlin for, for some work that he was doing he would just go on and on and on and on about, about Kotlin and how elegant it is. So anyway, it made me maybe think of, of that comment that he made once you mentioned your, your Kotlin work as well.

AUSTIN: Yeah, it's very similar to Swift too. There's, there's just going back to that, just that tiny comparison to the subtle nuance. It's, it's amazing like, and I can't think of like a good example. It's, but it's like the day to day stuff that you run into of just declaring a variable or you know, describing something as being lazy. It's like this is a pattern that has been well established in programming for years and now they've just made it a concise little keyword. And that's fantastic that it shows the evolution and it's again very expressive language, very type safe and just, you know, has null safety as well. So just a very safe language for. It's just, it's just helpful. Just the language itself is helpful, which is great when you're a developer and that's the tool you're using.

ADRIANA: Yeah, absolutely. Very cool. Okay, next question. Question. Do you prefer dev or ops?

AUSTIN: Probably dev. And yeah, there's. It's tough because I want to build a new thing, but that thing also has to exist once I, you know, we put it out and, and maintaining and managing that is, is a whole different beast. But yeah, I would say building new features, working on new tools, trying to take the technology that we are given by these platform developers or these bigger corporations and put them together in new and different ways or just even just playing around with somebody's open source project that's new and different. It's like, okay, does this inspire me? Is this interesting? Is this useful? Can I use this on my day to day? That is a lot of fun and the hope is to be able to contribute back and put something else out there that somebody else finds interesting and useful and they can use on their day to day. And so yeah, for me, definitely just the development side of it.

ADRIANA: Cool. And on a similar vein, similar ish, I guess. Do you prefer JSON or YAML?

AUSTIN: For. Well, it depends what we're doing for config files. YAML 100% of the time. That's the Ruby and me, I think that was part of the Rails development is all. It's all YAML config for APIs and sending data to a server to a back end, it's all JSON and I don't know if anybody's used YAML on that side of it yet. That would be. I'd be curious to come across that.

ADRIANA: Awesome. Also, do you prefer spaces or tabs?

AUSTIN: Spaces. Yeah, I know they take more space.

ADRIANA: I'm part, I'm part of Team Spaces, recent converts. So I'm. I'm down.

AUSTIN: Yeah.

ADRIANA: There's something. They're a lot more consistent too. Like you, no space in one OS is going to be the same as the other OS.

AUSTIN: And well, I don't know if this makes me weird or not, but I need my tabs to be converted to spaces. I hate the space score six times. Like, if I'm in a text editor where I have to space out my indentation, I would go nuts. And so I'm sure no one is doing that, but I don't know when I have to, like, format something in Slack to send a code snippet off or something, and I find myself counting spaces, making sure they all line up. I'm like, all right, maybe. Maybe I should just move to tabs.

ADRIANA: Yeah, I know what you mean. I have my VSCode configured to convert the tab key to spaces, so I'm totally down. Okay, two more questions. Do you prefer to consume content through video or text?

AUSTIN: Video, I would say. I even. I'm not a reader. I. But I. I recently got a library card and signed up for Libby, which is a great app if. If people haven't checked it out. But I stick in the audiobook section. And so I am listening to stuff as I'm walking the dog or, you know, prepping to get. Get to bed or something. And video is. Is kind of analogous to that where it's like, I need to be told and shown. Diagrams are fantastic, and straight text just. I find myself catching every, like, sixth, seventh, tenth word. It's kind of spaces out the longer the document. And then I'm just like, wait, I need to go back and. And reread all of this. So I don't know. I just have zero attention span and for. For words. And that's just practice. But, yeah, I don't know. I think my mom would say.

ADRIANA: I was gonna say it's interesting because, so my daughter, she's in the same boat. She'd rather do, like, video over text. And she's also, like, not a. A big reader. And she got a lot of flack from people like, oh, you need to, like, you need to love books. Why don't you love books? And it's like. But she consumes all her stuff through video, so who cares how she gets her information as long as she gets her information? So...Yeah, exactly. And I think, like, people get too hung up on. On, like, how you're getting your information. It doesn't matter, like, because we all learn in different ways. So. Yeah, I. I just want, like. I think it's important to remind folks of that because people can get so judgy over stuff like that, you know?

AUSTIN: I do. There is, like, There is a really big sense of accomplishment when you close a book and it's the final page. It's just. I don't know what else does that. I mean, finishing a video game. You know, I've been watching a lot of Elden Ring because that's a new big game and, and it's just amazing that like, oh, I got to the end of that. It's just this massive epic. And so books. Yeah, there's. There's nothing replaceable about a physical book.

ADRIANA: Yeah, true. I, I definitely agree. Although, like, I don't think I've. I hardly have any physical books in my possession anymore. I might have like five. Everything now is, is like on my, on my Kindle just from the space perspective. Like, I just don't have a spot in my house to keep so much stuff. I had to get rid of a lot of my physical books a long time ago.

AUSTIN: I, I think once I have a place that I know I'm gonna be in for a long time, then I might start accruing some of it. But just moving, I've had to move two or three times in the last six years and, and it's just awful to have a box of books that weighs 50 pounds down the stairs. Let's go.

ADRIANA: So, yeah, moving in itself is like a very awful experience. Like even from a small place where you're like, nah, I don't have that much stuff. And then you're like, where. How have I been keeping all this crap for so long? Where has it been hiding there?

AUSTIN: In college, I did have a roommate that I lived with. It was just him and I and we lived on this fourth floor of the walk up. And there was a really heavy box that I help him move up the stairs. And I dropped it in our kitchen when we got to the top. What the heck is in this? Is it just weight? Like, this is just dead weight? What the heck is it? And he just looks at me. Oh no, that's my weight set. And it was just of little like dumbbells and all this stuff. The only time it got carried in the two or three years we were there is when I moved it up and then moved it out. He didn't use it at all. I didn't use it at all. And I was just like, you just gotta get rid of this. This is not going in the truck.

ADRIANA: Oh my God. Yeah, that's the, that's the worst. I, Yeah, I have. I, I bought a. One of those weight sets. You know the ones that you like turn a knob and it'll like, it's.

AUSTIN: Oh, very cool.

ADRIANA: And oh my God, when it came in the mail and my I, I work out in, in, in this room, which is like on the second floor of my house. When it came in the mail, I'm like, how the hell am I gonna carry this thing? Because I, I think like, each dumbbell is like 50 pounds or some ridiculous weight like that. So I, I, I think I either asked my husband. Yeah, I think I asked my husband. I'm like, you do it. Use your manly strength, please. I can't do it.

AUSTIN: Get two people. It's like moving a couch at that point.

ADRIANA: Yeah, yeah, that stuff is heavy. Okay. All right. Final question of our lightning, not so lightning round questions is, what is your superpower?

AUSTIN: Oh, man, I don't know what direction I want to go. I, I have a, like, probably like seeing ahead, maybe seeing the future. If I had a superpower. I'm, when I played sports or something, I always had a knack for just knowing what was going to happen before it happened. And just, just like the vision, I guess, is, you know, in sports as well, you have to kind of skate to where the puck is going, as Michael Scott says, and just being able to play. You know, I played soccer mostly, so playing the ball into space to let people run onto it was really important. And that's carried through into technology of just knowing, oh, okay, this, I, I can see the development happening here. This is kind of the direction it's going. So maybe I should meet up with it up ahead. And that can be really, really useful. You know, sometimes you say, I'm going to meet up with it here, and it's, it's taking a 90 degree turn in a direction away from you, and you're like, I'm in the middle of nowhere. So it's not 100 of the time, but I think just having an understanding of, okay, this is kind of shaping up. Yeah. Let me, let me, you know, get ready for the next, the next act. And so, yeah, that's, I mean, I would love, you know, freeze rays or flames or something physical as well, but something cerebral would be, would be very cool as well.

ADRIANA: Yeah, yeah, that, that's a great superpower because. Well, especially in tech, like, I, I think there's some technologies where you're kind of like, I feel like this is going somewhere. It's, it's good to start investing now. Right?

AUSTIN: Yeah, it's, it's, hopefully it works out. I guess you just kind of try to fire off a bunch of ideas to see. Okay, yeah, that could work.

ADRIANA: Yeah.

AUSTIN: And that's what I like too.

ADRIANA: Awesome. Well, we've, we've survived the lightning-ish round questions. Not so fast, not so fast. Lightning round questions. So now onto the meaty bits. So you work at Embrace and if you can explain what is it that Embrace does, that would be super cool.

AUSTIN: Sure, yeah. Just quickly it's. We're an Observability product and we focus mostly on mobile platforms. So Android and iOS we have SDKs for React-Native Unity. We have some clients that are in the game space, but those games are mostly on iOS and Android. And we've recently gone open source, which is, was a big shift in the company to, to kind of come out of the shadows. But even on top of that we've also converted the foundation to be in OpenTelemetry, which is really exciting. OpenTelemetry is kind of a new standard, new ish standard for Observability and it isn't really practice in the mobile space. And so we're excited to hopefully get a lot more people in the mobile space to, you know, join and kind of share their ideas and explore this new standard for what's possible. So I am the one of the lead iOS developers on the team, so I'm focusing mostly on the iOS SDK. But we have, you know, I work very closely with our Android team to make sure that we're both at least following the spec somewhat together as we kind of work through it. Which is, which is fun.

ADRIANA: Cool. That's so awesome. And so when you mentioned that Embrace made the shift to open source. Was it just a matter of like, okay, we're opening up our code base. Did you do like a major re-architecture? Because you also mentioned that you did some like OpenTelemetry integration as recently. Was, was it part of that move to open source as well?

AUSTIN: Yeah. So on the iOS side it was a re-architecture situation, the conversation. And we have the 5.x SDK which is the closed source one, and now our new 6.x SDK that is open source and built on OpenTelemetry. And the big difference there is we wanted to go all in to OpenTelemetry and so there was a lot of conversation of okay, we can add these objects and expose this interface and then shim it kind of back into the existing structures we had under the hood. But it was kind of just like, well, if we're not going all the way, why go at all? So, you know, it's like this. I, I don't want to do anything half assed. I, I want to go into this and really contribute. And it was, you know, that 5.x is built in Objective-C. Uh, and so open sourcing an Objective-C framework in 2024, I don't think was going to gather as much buzz and excitement. No offense to the people that love Objective-C. I, you know, it's, it's what created my career. I can't, I can't say anything bad about it, but, but Swift is, is my favorite language, we've established. And so it was, it was time to really remake the thing with all of our understanding and learnings of Observability and on these, working with these platforms and then build something on top of OpenTelemetry and really start participating in that working group and that community.

ADRIANA: Cool. So when you're using OpenTelemetry on your own product, is that mostly to benefit your developers, like developers within Embrace, or is this also benefiting consumers of your product as well?

AUSTIN: Hopefully everybody. So for me, the consumer is a developer working in their app and they want to observe something. And so we have some instrumentation that's automatic so that you can capture network requests just by installing the SDK. And the OpenTelemetry Swift project has very similar instrumentation as well. And so you can use that project there and get some instrumentation out of the box. And so I think the big benefit that we saw from it internally was now everybody can speak a common language and there's an established set of primitives like a span and a span event to log. These are, these are things that the terms often get overloaded and especially when you're in an organization and you have people of all levels that are, you know, marketing the product or selling the product. If you can have a very tiny set of primitives that we can agree on, then we can speak openly about what it is and what it does and, and use terms that hopefully are familiar to people outside of, of the company and then to hopefully benefit people outside of the company. They're coming in and, and hopefully the learning curve is as shallow as, as possible or as flat as possible. Because the primitives, if they're used to OpenTelemetry, the primitives are the same exact primitives. They're, they're working with spans to create a trace or they're creating logs. And there's no question of what is this, how is this modeled? What do I do to measure this as performance? It's just, oh, no, I need to wrap this with a span. So, yeah, hopefully that is beneficial to both sides and we'll see. Well, you know, it's always a feedback gathering opportunity when, when you put something out there and the fact that it's open source, I'm very, you know, hopeful that we get very good feedback because people can see under the covers and say yeah, you know, this, this doesn't line up how I would expect. Or I can see where this breakdown is occurring. Or this is great.

AUSTIN: I'll be an optimist and say, no, everything's fantastic. You did an awesome job.

ADRIANA: That's so cool. So, so, and then, then the Embrace product itself is, is basically it provides like a, like a UI so that like any, any mobile app can like that's, that's sending telemetry data to it. Can you. It's similar to like other Observability tools, but it's this one specifically tailored for, for mobile then.

AUSTIN: Yeah, yeah. And there are things that are mobile specific like a crash report or Android you have something called an ANR, an application not responding. It's a very common pop up that occurs or application exit info where it's just like the app just quits behind the scenes. The system killed the app. The next time the app launches, the system then provides, hey, we killed your app. Sorry, here's why. That's very mobile. Well, that one is especially Android specific data that we can then collect and save in our dashboard and hopefully explain to the users this is what happened, here's why. Or even just things where a user has quit the app and the user, apparently it seems the user got frustrated or stuck on this page. Go check out that page. Maybe there's a layout issue on the device that that user was using where the submit button was rendered off screen in whatever circumstance. You know, maybe their, their name had pushed it somewhere or their, whatever product they were purchasing had just pushed it off, off screen. It happens. But you know, hopefully they have the tools then to go fix that issue and, and you know, clip some of that text or you know, reflow their layout so that that button is now accessible.

ADRIANA: Cool. So then is it correct for me to assume that Embrace ingests like OTel data in like the native OTLP format or do you guys have like, like an exporter?

AUSTIN: So we're. Yeah, the state of it currently is we have a generic export, so we have instrumentation, we have interfaces to add a span that mimic the OpenTelemetry API. So you can add span, add logs, you can configure if you'd like a generic exporter. And so that's if you have a collector set up already, maybe you have a system somewhere in the cloud that is already ingesting telemetry. And you just want to add a new source to that? Yeah, you just pass us a generic exporter. A lot of those have already been implemented in the OpenTelemetry projects, or you can implement your own if you want to get it to your server in a custom shape and go from there. We by default will upload that data to our backend as well so that you can consume it in our dashboard. And so that's the current state of things. Where we want to go is actually provide the extensibility on the front end of that to ingest more of the data. To say here's the embrace tracer object, that is an OpenTelemetry object, conforms to the OpenTelemetry API. And you have your app that is already instrumented using OpenTelemetry. Just pass that tracer in and all of the instrumentation that you've provided throughout your app will just work. And it'll now flow through our SDK and into our system if you'd like it, or through our generic exporter if you'd like it. And that's kind of the free use. It's just, you don't have to touch our dashboard at all, but you're using our SDK and there's some benefit to the SDK to just maybe recover data if an app crash occurs. Or you just want to use our crash reporting tool and so you can just have that running and then have instrumentation flowing through. So yeah, that's, that's where we're work, what we're working on, you know, this sprint. And so you'll, you'll see that coming soon. If not, you know, already by the time this is out.

ADRIANA: So cool. So then does that mean, do you do, do consumers of Embrace need to use the SDKs then, your SDKs in order to emit telemetry? Or from the sounds of it, it look, they can use OTel then like can you bypass the, your, your own SDKs like the Embrace SDKs?

AUSTIN: We want people to be able to hot swap it of just saying, you know, and, and the way I look at that is, you know, there are three layers that, that we're playing with and it's the OpenTelemetry spec itself, the API that, that is, you know, very strict and very foundational. The semantic conventions that are provided by OpenTelemetry that the SIGs have come up with and agreed upon and promoted. And then the third layer is, I call them the embrace semantics. And it's the things that we've done. The instrumentation how we collect maybe device low power or a low memory warning and it's a custom shape to that telemetry that is still just a span maybe or it might be an event or a log, but we've, it's, it hasn't yet been baked into those OTel semantic conventions. And that's the goal is, is it's going to start, we're going to try to prove its value and its worth and the structure of that and the use case for that, that the shape of that telemetry and then once it's established, participate with the client side SIG, the Android SIG, the Swift SIG to say is this how you would, you know, model a low, low power mode?

ADRIANA: Right.

AUSTIN: And if that's the case, then let's propose it as a, an OTel semantic convention and then just have everybody understand, okay, if you have low power mode events, you know, it might be a span with the name of the span as this and the attributes are this and that and we measure the whole time. And so there's, you know, we, we, that top layer is really just for us to be able to move quickly and provide value to our customers. But as we're doing that, we're constantly talking with, with the SIGs every week to say, you know what, what's working for you? And the, it kind of goes both ways too. Any new semantic conventions that come out, we want to use and, and take on as soon as we can.

ADRIANA: Right.

AUSTIN: Right. But then our, our consumers, hopefully if you know, if they are using the OTel Swift SDK directly and they're happy with it, then great, that's fine. We might in the future put out a package that could sit next to the OTel Swift thing that is just like some Embrace conventions that are little helpers, you know, we extend the span to say, you know, just to make little, make it quicker and easier for developer to measure something like low power mode or disk IO for connecting to an SQLite database locally on the client. That would be just useful and we want to hopefully drive that and be helpful. That's those tools I was talking about of just being helpful that somebody finds inspiring and useful.

ADRIANA: That's awesome. Then what it sounds like to me and correct me if I'm wrong, is trying to make sure that you're as much up to date with the OTel semantic conventions and APIs as much as possible. It sounds like then the Embrace SDK is almost like an implementation, your own implementation of the API, which is by design like an OTel thing that anyone can really implement the, the API with their own SDKs. But also as, but also making sure that you contribute back to the community and hopefully making some upstream contributions to the OTel project that can then be part of that. Reintegrated.

AUSTIN: Yeah. Into the foundation.

ADRIANA: So that, yeah.

AUSTIN: So it's not, you know, then, then you know, our name gets out there and people maybe not just the patterns that we've hopefully established to get out there and people start using them. And that, that would just put a smile on my face like, oh, you found that useful. Great. And, and it's just, yeah, it's all about that learning curve. Mobile developers especially we found aren't used to OpenTelemetry, haven't. It's just not talked about as much or it's not as standardized and so so hopefully by making it easier, it's, it's more accessible and people jump in. So yeah, that's the, the manifesto, I guess.

ADRIANA: Yeah, that's great. It's so interesting because there's so much focus on kind of your common languages for instrumenting in OTel and yet mobile apps are all around us. They're such a ubiquitous part of our lives that when you think about it, it's like, oh, of course there should be instrumentation on our mobile apps, but it's easy to forget about that. So it's cool that that's being tackled and that area is getting some TLC now.

AUSTIN: Yeah, yeah. It's not a new frontier. There have been players in the space for a little bit, but definitely TLC is always appreciated if nothing else.

ADRIANA: Yeah, definitely. And one thing I want to ask you about as well was in, in terms of like implementing OpenTelemetry in your own application internally, how was that for, for, you know, like the, the teams that Embrace. Like, was it how, how would you say the experience was? Like what, what were some of the challenges that you encountered? What are some of the things where you're like, oh my God, this is amazing.

AUSTIN: None of it was too OpenTelemetry specific. Most of it was just now we have a third party dependency that, that we depend on and in a really big way. And OpenTelemetry isn't our only dependency and but it just comes down to the minutia. So we have this generic export where we can, especially for tracing, we export this object called span data. When a span finishes, it gets frozen for a lack of a better term as a span data object and then exported. Right now, at least in the Swift side of things, that was private and we couldn't Access that and create our own and send them off. It caused some development time I guess to, because we had, we had written out and basically re implemented a lot of the SDK to do what we wanted to do. And then we realized oh this, this export won't work.

We can't, we don't have access that final piece. So we actually just, let's scrap it and just use it directly. And now we'll, we had to kind of change our thinking to just use the, the OTel SDK as a third party dependency a little more directly than we initially expected. We just wanted to stay at the API layer and that was a challenge just because it was an assumption that was made that was broken probably two or three months into the project and then had to slow down to make sure that we could do what we needed to do. But now we're past that and you know, works great and it was, you know, better, you know, we, we, we got to a better state and what was great about that is you know, I, I attend the Swift SIG. I went to the Swift SIG and Nacho is, is one of the like lead guys there and he was very helpful, explained what, what his expectation would be and why that is private, why that span data object is private and, and why we shouldn't just open it up. You know, and so that is exactly how the process should go. There was a question, I raised the question.

They you know, came together, discussed and said no, where there, there's a workaround for you and so go, go use the workaround. So, so it was you know, helpful to do that and, but it just, it just, I think that software, general, software development in general is you know, the assumption was broken. Yeah halfway, you know, a time, after some time occurred and now there's you know, not a sunk cost fallacy but just like we're going to have to change the assumption, that original assumption. So let's, let's walk back a bit. But other than that the team understood like the, the, the benefits of OpenTelemetry are so apparent. Like we need a common envelope so that all of our telemetry can, can flow through this same channel. And if we want to start instrumenting a new thing, we shouldn't have to change anything along that channel. It should just be a new span and we've done, done some things to kind of type hint what that span is.

So if we have a network request, our backend can pick it up and see, oh, this is a network request and we're going to treat it as such. And that's been very useful. But that's all within the constraints of OpenTelemetry, which is great because it, it makes, it takes a lot of the decisions away, which means we can focus on the things that we really care about or we really want to try to do.

ADRIANA: Yes, then you don't even have to reinvent the wheel because it's there.

AUSTIN: Yeah, yeah, exactly.

ADRIANA: Nice. Yeah. And do you use, do you use OpenTelemetry then to debug your own product, like your own code?

AUSTIN: We do actually. And this, there's a feature that we just implemented. Feature? Is it a feature if it's not external? I think so, but we implemented, we call it internal logs and it's how the SDK itself works. And so we have errors that occur. We store data into an SQLite database ourselves for local storage and it's a file operation that can fail for, especially on mobile devices for many different reasons. The device can just stop or the device can be out of disk space. And so when that happens we have to do something about it. And we, you know, the disk space one's a little odd because you're trying to save data that you can't, but we send off these internal logs that are just OpenTelemetry logs.

ADRIANA: That's so cool.

AUSTIN: Yeah. And it's the best type of dog food, I guess.

ADRIANA: You know, it's totally, I love, I love hearing these end user stories because, you know, I'm one of the maintainers of the End User SIG and so I think being able to share stories of folks not just in the companies who are like, you know, your typical consumers of OTel, which is like whatever, whatever large enterprise, but also like the, the Observability vendors themselves. Using OpenTelemetry on their own product, I think makes for a very compelling story because especially like, if we want folks to use OpenTelemetry, then it stands to reason that Observability vendors would, it would be very wise to use OpenTelemetry on themselves and to use that to help them troubleshoot their own product.

AUSTIN: Yeah, yeah, absolutely. And even just the performance of like another is the, the startup time of our SDK. It's if, if we're slowing down, you know, we ask that our SDK is, is created and started at the app launch time. Well, that's a critical point in time for an app and you want to show that user the initial content as soon as possible. And if our startup time is blocking and slowing down the app startup time, that's going to cause problems for us and that user. And so we don't want that. So we can trace that. We can just break it down by each operation that we want to show exactly how long it takes. And there's some interesting things that the platforms do that we can hook into. Like how is the process kind of warms up and the. The system actually kind of warms things up after it guesses if the user is going to interact with an app or not. So if a push notification comes in, it's likely that the user is going to tap that push notification and come into the app. And so in certain circumstances, the system will warm that process up. And if we know that launch time of the process, we can actually see how long it took or how long that warming process, how long that process was kept warm, I guess, is what I'm trying to say, before that user has entered the app. And so it's just very interesting things. It's almost not even the performance of our SDK, but just like interesting behavior that the system is doing that I'm curious about. Maybe no one else is interested in this, but I'm just like, okay, this is how Apple is doing stuff under the hood. Let me. Let me take a peek.

ADRIANA: That's very cool. Well, we are coming up on time, but before we wrap up, I wanted to ask if you have any words of wisdom or hot takes to share with our audience.

AUSTIN: If you're working on a project and it's closed source and you're curious about going open source, I would say just do it. It's very useful to get feedback. I have a couple of side projects that I am toiling with. I just need to spend the time to actually finalize them and push out, you know, the little blurb of a readme that they need, but I just need to do it. And so that would be the words of wisdom. Put it out there, get feedback. It's cool. I will use it. It's. And you know, it'll be more fun. I'm sure it will.

ADRIANA: That's awesome. Well, thank you so much, Austin, for geeking out with me today. Y'all don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...

AUSTIN: Peace out and geek out.

ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who incidentally designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.