Efficient Data Collection Process from APIs

2min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

To gather detailed trade data daily, the speaker developed a program to download it since the required data wasn't available through the API. They faced a challenge with the short-lived login cookie, so they implemented a two-step login process using their one password integration for security. After obtaining a JSON web token, they could access the API to fetch and save the required data. For some instances, they needed to use fetch requests, adjust methods, and headers. They mentioned the 'fetch cookie on node' plugin as a helpful tool eliminating the manual cookie tracking hassle, by automatically adding the cookie to subsequent requests.

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease.

Show Notes

00:00 Welcome to Syntax!
03:13 Brought to you by Sentry.io.
05:00 What is scraping?
Socialblade.
08:01 Examples of past scrapers.
Canadian Tire.
10:06 Cloud app downloader.
16:13 Other use cases.
16:58 Scraping 101.
17:28 Client Side.
19:08 Private API.
Proxyman.
22:40 Server rendered.
23:27 Initial state.
24:57 What format is the data in?
Google Puppeteer Extension.
27:08 Working with the DOM.
27:12 Linkedom npm package.
29:02 querySelector everything.
31:28 How to find the elements without classes.
34:08 Use XPath selectors for select by word.
34:53 Make them as flexible as you can. Classes change!
35:10 AI is good at this!
36:26 File downloading.
38:20 Working with protected routes.
Proxyman.
40:41 Programatically retrieve authentication keys because they are short-lived.
Fetch Cookie.
43:20 Deal-breakers.
Mechanical Turk.
44:58 What happened with Amazon?
Uniqlo Self-Checkout
46:42 Wes’ portable refrigerator utopia.
47:25 Sick Picks & Shameless Plugs.