Building a Signature Scanner and Scaling Services with Message Brokers

Episode 29: In this episode of Critical Thinking - Bug Bounty Podcast sit down with Assetnote Engineer Sean Yeoh, and pick his brain about what he's learned on his development journey. We talk about the place and importance of message brokers, and which ones we like best, as well as his engineering philosophy regarding bottleneck prevention and the importance of pursuing optimization. Don't miss this episode of terrific technical tips!

Follow us on twitter at: @ctbbpodcast

We're new to this podcasting thing, so feel free to send us any feedback here: info@criticalthinkingpodcast.io

Shoutout to YTCracker for the awesome intro music!

------ Links ------

Follow your hosts Rhynorater & Teknogeek on twitter:

https://twitter.com/0xteknogeek

https://twitter.com/rhynorater

Today’s Guest:

https://twitter.com/seanyeoh

Assetnote

https://www.assetnote.io/

https://twitter.com/assetnote

XKCD automation graph

https://xkcd.com/1319/

Github repository

https://github.com/alex/what-happens-when

Article about Queues

https://archive.is/Nan4e

NATS

https://nats.io/

MongoDB

https://www.mongodb.com/

Timestamps:

(00:00:00) Introduction

(00:01:18) Story of Assetnote

(00:05:20) Message Brokers and event-driven architectures

(00:11:15) Preventing bottlenecks and pursuing optimization

(00:21:35) Using a profiler

(00:28:30) Choosing a Message Broker

(00:33:00) Kubernetes and Conntrack Limits

(00:37:13) Databases

(00:46:30) Bug bounty tips: Sub-domain vs. IP Address

(00:51:15) Engineering quandaries

(00:53:38) DNS Wildcards