Developer Voices

The “big data infrastructure” world is dominated by Java, but the data-analysis world is dominated by Python. So if you need to analyse and process huge amounts of data, chances are you’re in for a less-than-ideal time. The impedance mismatch will probably make your life hard somehow. So there are a lot of projects and companies trying to solve that problem. To bridge those two worlds seamlessly, and many of the popular solutions see SQL as the glue. But this week we’re going to look at another solution - ignore Java, treat Kafka as a protocol, and build up all the infrastructure tools you need with a pure Python library. It’s a lot of work, but in theory it would make Python the one language for data storage, analysis and processing, at scale. Tempting, but is it feasible? Joining me to discuss the pros, cons, and massive scope of that approach is Tomáš Neubauer. He started off doing real time data analysis for the Maclaren’s F1 team, and is now deep in the Python mines effectively rewriting Kafka Streams in Python. But how? How much work is actually involved in porting those ideas to Python-land, and how do you even get started? And perhaps most fundamental of all - even if you succeed, will that be enough to make the job easy, or will you still have to scale the mountain of teaching people how to use the new tools you’ve built? Let's find out.– Quix Streams on Github: https://github.com/quixio/quix-streamsQuix Streams getting started guide: https://quix.io/get-started-with-quix-streamsQuix: https://quix.io/ Tomáš on LinkedIn: https://www.linkedin.com/in/tom%C3%A1%C5%A1-neubauer-a10bb144Tomáš on Twitter: https://twitter.com/TomasNeubauer0Kris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins  --#podcast #softwaredevelopment #datascience #apachekafka #streamprocessing

Erlang wears three hats - it’s a language, it’s a platform, and it’s an approach to making software run reliably once it’s in production. Those last two are so interesting I sometimes wonder why those ideas haven’t been ported to every language going.  How much work would it be?This week we’re going to dig right down into that question with Leandro Ostera. He’s been working on Riot - a project to bring the best of Erlang’s runtime system and philosophy to OCaml. But why OCaml? Is it possible to marry together OCaml’s type system with Erlang’s dynamic dispatch systems? And what is it about the recent release of OCaml5 that makes the whole project easier?–Leandro’s Blog: https://www.abstractmachines.dev/Why Typing Erlang is Hard: https://www.abstractmachines.dev/posts/am012-why-typing-erlang-is-hard/Riot: https://riot.ml/Riot source: https://github.com/riot-ml/riotReasonML: https://reasonml.github.io/ReScript: https://rescript-lang.org/Leandro on Twitter: https://twitter.com/leosteraKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins--#podcast #softwaredevelopment #erlang #ocaml #softwaredesign

Discover how Apache Pinot achieves an impressive 200,000 queries per second and the architectural decisions behind it. Tim Berglund explains the roles, optimization, and performance of Pinot, covering queries, data movement, and indexing strategies. Learn about the evolution of databases, query optimization, and the technical details of real-time data applications with Apache Pinot.

Explore the evolution of Neovim from Vim, the power of Lua integration, and the benefits of customizable plugins. Learn about mechanisms over policies in Lula, advancing Lula integration in Neo-vim, and navigating software project challenges. Discover the advantages of Lua scripting in Neovim and the satisfaction of customizing your editing environment.

Done right, a Hackathon can be a fantastic place to be a programmer - you get time and space to build and learn, in a room full of like-minded people, with swag and prizes to sweeten the deal. It’s a great way to pick up new ideas and run with them. But done wrong it can be a waste of time. What’s the difference between a good hackathon and a bad one? What do the good ones do right, and what can we learn from that?This week we’re talking about the Joy of Hacks with Major League Hacking Co-Founder Jon Gottfried. He’s got over 10 years of experience building a Hackathon network that provides the right environment for “structured mucking about with computers”, so we’re going to pick his brains.If you’re ever attending a Hackathon, organising one, or looking for a way to build or contribute to your local programming community, Jon can help guide you to events that work.--Major League Hacking: https://mlh.io/Major League Hacking’s 2024 Event Calendar: https://mlh.io/seasons/2024/eventsGames Week: https://events.mlh.io/events/10848 Jon on Mastodon: https://hachyderm.io/@jonmarkgoJon on LinkedIn: https://www.linkedin.com/in/jonmarkgoJon on Twitter: https://twitter.com/jonmarkgoKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkinsBonus link - The Great American Baking Show 2023: https://www.youtube.com/watch?v=IlWLSAKEedk--#software #podcast #programming #hackathon

Discover the power of property testing in software reliability, where setting rules for code leads to automated testing. Learn advanced techniques beyond unit testing, testing databases, webpages, and more. Get insights from expert Oskar Wickstrom on enhancing code testing effortlessly.

If you ever feel overwhelmed by the number of different programming languages, this week’s episode might just offer you some solace, as we talk about an attempt to reunify many of the most popular languages by focussing on the bread & butter things that every language supports.I’m joined by Martin Johansen, who’s been working on a new tool called Progsbase. With it, he’s created a spec based on all the things programming languages can agree on, and is building a library that can cross-compile between them. Write a program in Java, and it can be automatically translated to PHP, Python and a great deal more.But how far can he take that idea? Is there really enough unity between these languages to build something universal? How do you bridge the divide between manual memory management languages like C and garbage-collected ones like Java? And what would it actually feel like to write code this way? Let’s put Martin’s plan under the spotlight and find out…–Martin on Twitter: https://twitter.com/martinfjKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Progsbase homepage: https://www.progsbase.com/The Spec: https://www.progsbase.com/docs/programs/The Progsbase library repository: https://repo.progsbase.com/The Bug Bounty: https://www.progsbase.com/bug-bounty/–#software #programming #podcast #programminglanguages

Jonathan Schneider, one of the authors of OpenRewrite, discusses the power of automated code-rewriting tools and how they work. Topics include engineering challenges at Netflix, rewriting Terraform, and implementing an internal compiler. They also touch on the decision-making process for using libraries and language support for Gleam and WebAssembly.

SQLite could do with a little competition, so when I invited the co-creator of DuckDB in to talk, I thought we'd be discussing the perils of trying to build a new in-process database engine. I quickly realised things went much deeper than just a tech refresh.Hannes Mühleisen joins me this week to blend his academic credentials as a database researcher with his vehement need to make that research practical. And so we dive into what modern database literature has to say on making queries faster, more parallelizable, and closer to the metal, and how it all comes together in a user-friendly package that’s found its way into my day-to-day workload, and might well help out yours.If you’re curious about the gory details of database queries, how they can take advantage of modern hardware, or how all that research actually turns into a useful tool, Hannes has some great answers.--DuckDB: https://duckdb.org/Database Systems Book: http://infolab.stanford.edu/~ullman/dscb.htmlKris’ first computer: https://en.wikipedia.org/wiki/File:ZX_Spectrum_Plus2_(retouched).jpgVolcano Query Evaluation System [pdf]: https://paperhub.s3.amazonaws.com/dace52a42c07f7f8348b08dc2b186061.pdfMorsel Query Engine [pdf]: https://cs.brown.edu/~kayhan/papers/morsel_cp.pdfUnnesting Arbitrary Queries [pdf]: https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdfPapers Hannes' team have published: https://duckdb.org/why_duckdb#peer-reviewed-papers-and-thesis-worksDuckDB on Mastodon: https://mastodon.social/@duckdbKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Mastodon: https://mastodon.social/@krisajenkins--#softwaredevelopment #podcast #programming #database #duckdb #sql #sqlite

Simon Peyton Jones, a veteran language designer and researcher, talks about building a large language from a small, tightly designed core. He discusses his work on the Verse programming language which blends functional programming with logic languages like Prolog. Simon also discusses the importance of teaching computer science in education and reshaping the way it is taught in England.