The Essential Guide to Crawling: Practical Techniques for Crawling Websites

This is a book I picked up because I needed crawling for a toy project. But the project soon fell through (?), so I never actually needed to build a crawler myself. Despite this anticlimactic turn of events, I couldn’t just put the book down. The difficulty of web crawling ultimately comes down to a battle against poorly built web pages. And this book deals with exactly that point. The more I read, the more I had to reflect on whether I had really been building web pages properly all this time. I recommend that every web developer try building a crawler at least once.

Naoki Takezoe, Takako Shimamoto, Shunsuke Tadokoro, Takahiro Hagino, Momoko Kawakami, 2018, The Essential Guide to Crawling: Practical Techniques for Crawling Websites, translated by Yoon In-sung, WikiBooks