The InfoQ Podcast

Wesley Reisz talks to Sid Anand, a data architect at cybersecurity company Agari, about building cloud-native data pipelines. The focus of their discussion is around a solution Agari uses that is built from Amazon Kinesis Streams, serverless functions, and auto scaling groups. Sid Anand is an architect at Agari, and a former technical architect at eBay, Netflix, and LinkedIn. He has 15 years of data infrastructure experience at scale, is a PMC for Apache Airflow, and is also a program committee chair for QCon San Francisco and QCon London. Why listen to this podcast - Real-time data pipeline processing is very latency sensitive - Micro-batching allows much smaller amounts of data to be processed - Use the appropriate data store (or stores) to support the use of the dataIngesting data quickly into a clean database with minimal indexes can be fast - Communicate using a messaging system that supports schema evolution More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2rJU9nB You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2rJU9nB

Wesley Reisz talks to Sachin Kulkarni, Director of Engineering at Facebook, about the engineering challenges for Facebook live, and how it compares to the video upload platform at Facebook. Why listen to this podcast: - Facebook Infrastructure powers the board family of apps including the Facebook app, Messenger and Instagram. It is largely a C++ shop. There is some Java and Python, and the business logic is all done in PHP. The iOS apps are written in Objective C and the Android apps are in Java. - The video infra team at Facebook builds the video infrastructure across the whole company. Projects include a distributed video encoding platform which results in low latency video encoding, video upload and ingest. - Facebook Live does encoding on both the client and the server. The trade-off between encoding on the client side and the server side is mostly around the quality of the video vs. latency and reliability. - Facebook gets around 10x speed-up by encoding data in parallel compared to serial. - They also have an AI-based encoding system which resulted in 20% smaller files than raw H.264. You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2qrseG5 You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2qrseG5 Want more? Read InfoQ: http://bit.ly/2dcHmpu

Wesley Reisz talks to Martijn Verburg, co-founder of the London Java Community and CEO of jClarity, about the JCP EC “no” vote on the Java Platform Module System (JPMS), which is due to be shipped as part of Java 9. The talk about what JPMS offers, how it works, what the no vote means and what happens next. Why listen to this podcast: - Jigsaw isn’t dead - The “no” vote was based on the submission being a bit early, and without expert group consensus that it should be submitted - Since the vote started, several amendments have been made which addressed some of the concerns listed by those who voted “no” - Daily calls with the expert group and interested parties will work to resolve the outstanding issues promptly - A resubmission is due within 30 days with a future vote expected to go through More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2q20esc You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extended shownotes? Check the landing page on InfoQ: http://bit.ly/2q20esc

Wesley Reisz talks to Daniel Bryant on moving from monoliths to micro-services, covering bounded contexts, when to break up micro-services, event storming, practices like observability and tracing, and more. Why listen to this podcast: - Migrating a monolith to micro-services is best done by breaking off a valuable but not critical part first. - Designing a greenfield application as micro-services requires a strong understanding of the domain. - When a request enters the system, it needs to be tagged with a correlation id that flows down to all fan-out service requests. - Observability and metrics are essential parts to include when moving micro-services to production. - A service mesh allows you to scale services and permit binary transports without losing observability. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2pFYBiT You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2pFYBiT

Rossen Stoyanchev talks to Wesley Reisz about blocking and non-blocking architectures, upcoming changes in Spring including Spring WebFlux, the reactive web stack in Spring framework 5, due this summer. He also discusses the differences between rxJava and Reactor. Why listen to this podcast: - Spring Framework 5 is due to be released June 25 2017 - Spring Web Flux provides a web programming model designed for asynchronous APIs - Back-pressure is important in a server environment; less so within a UI environment - It’s possible to use a Spring Web Flux client within a Spring MVC applciation - Managing sets of thread pools is more complicated than having a scalable asynchronous system More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2pPgq0G You can subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2pPgq0G

Why listen to this podcast: - Using a compiler to catch errors at compile time instead of at runtime means much easier refactoring of code. - Incrementally replacing small parts of an existing JavaScript application with Elm is a safer strategy than trying to write an entirely new application in Elm - Elm packages are semantically versioned and gated by the publishing process, so minor versions cannot remove functionality without bumping the major version. - The UI in an Elm application results in messages that transform the immutable state of the application; this allows a debugger to view the state transitions and the messages that triggered them, including record and replay of those messages. - Elm has been benchmarked as being faster than Angular and React whilst being smaller code, which is attributed to the immutable state and pure functional elements. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2qmS2CT You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2qmS2CT

Jean Barmash is Director of Engineering at Compass, Founder & Co-Organizer, NYC CTO School Meetup. Live in New York City. He has over 15 years of experience in software industry, and has been part of 4 startups over the last seven years, 3 as CTO / VPE and one of which he co-founded. Prior to his entrepreneurial adventures, Jean held a variety of progressively senior roles in development, integration consulting, training, and team leadership. He worked for such companies as Trilogy, Symantec, Infusion and Alfresco, consulting to Fortune 100 companies like Ford, Toyota, Microsoft, Adobe, IHG, Citi, BofA, NBC, and Booz Allen Hamilton. Jean will speak at QCon New York 2017: http://bit.ly/2nN7KKo Why listen to this podcast: - The Compass backend is mostly written in Java and Python, with Go increasingly a first class language. The main reason for Go being added was developer productivity. - The app is based on a Microservices architecture with around 40-50 services in total. - Binary RPC, originally Thrift and Finagle, is used as the communication protocol, but the company is gradually moving to gRPC still with Thrift. One advantage that gRPC offers is better Python support than Finagle. - The company has built a code generation framework which takes Thrift and converts it to a RESTful API for clients to consume. - Constraint theory is about how you manage the one constraint in a system or team that prevent you increasing throughput; for example if your software engineering team only has one front end engineer do you ask back-end engineers to pick off some front-end tasks, or bring in a contractor. Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Eric Horesnyi, CEO @streamdata.io, talks to Charles Humble about how hedge funds are applying deep learning as an alternative to the raw speed favoured by HFT to try and curve the market. Why listen to this podcast: - Streamdata.io was originally built for banks and brokers, but more recently hedge funds have begun using the service. - Whilst Hedge Funds like Renaissance Technologies have been using mathematical approaches for some time deep learning is now being applied to markets. Common techniques such as gradient descent and back propagation apply equally well to market analysis. - The data sources used are very broad. As well as market data the network might be using, sentiment analysis from social networks, social trading data, as well as more unusual data such as retail data, and IoT sensors from farms and factories. - By way of contrast High Frequency Trading focusses on latency. From an infrastructure stand-point you can play with propagation time, Serilization (the thickness of the pipe), and Processing time for any active component in chain. - One current battleground in HFT is around using FPGA to build circuits dedicated to feed handlers. Companies such as Novasparks are specialists in this area. More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2nv71M8 Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2nv71M8 You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Greg Murphy is the COO of Gamesparks, a cloud-based platform providing and a rich mobile back-end service for game developers to engage with their users. Greg takes us inside Gamesparks discussing the architecture, machine learning and what it’s like to launch in the China market. Why listen to this podcast: Gamesparks Engagement Engine Tuning the Gaming Experience The Architecture SDK’s and Real-time Data Transfers Server-side Scripts Managing Noisy neighbours and Security The Developer Experience Deploying Across Three Cloud Providers Machine Learning The China Market Notes and links can be found on: http://bit.ly/2neCjEV You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Wesley Reisz talks to Slava Oks, who has worked at Microsoft for over 20 years on flagship products, including SQL Server. He also led the kernel team who worked on the Midori operating system. More recently, he has worked on bringing SQL Server to Linux. Why listen to this podcast: - Microsoft SQL Server runs on Linux through a containerised approach called Drawbridge - Drawbridge implements a Linux loader and a minimal set of ABI calls to allow an in-process NT user mode kernel to run - SQL Server runs on top of a SQL platform layer (called SQL OS) that could be ported to run on Drawbridge - SQL Server had supportability commands added to allow the state of the system to be measured with SQL calls - A number of efficiency gains were applied to both the Drawbridge components and the SQL Server code to bring performance to within 20% of the equivalent process running on Windows Notes and links can be found on: http://bit.ly/2kVfatX Drawbridge ABIs Security LibOS Linux SQL OS Supportability Some assembly required Addressing Performance More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2kVfatX You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq