« Changelog History


Infinity Crawler v0.3.0 Release Notes

Release Date: 2019-08-17 // over 5 years ago

  • 👌 Improved support for in-page Robots configuration (#7)
    ➕ Added fragment support for crawl data (#7)
    ➕ Added support for request timeouts (#7)
    ➕ Added support for max pages to crawl (#7)
    Obey no-follow on anchors (#7)
    🛠 Various other minor tweaks and fixes (#7)


Previous changes from v0.2.0

  • Replaced crawl contents stream with a string, allowing easier access to contents.