Infinity Crawler v0.3.0 Release Notes

Release Date: 2019-08-17 // over 4 years ago
  • ๐Ÿ‘Œ Improved support for in-page Robots configuration (#7)
    โž• Added fragment support for crawl data (#7)
    โž• Added support for request timeouts (#7)
    โž• Added support for max pages to crawl (#7)
    Obey no-follow on anchors (#7)
    ๐Ÿ›  Various other minor tweaks and fixes (#7)

Previous changes from v0.2.0

  • Replaced crawl contents stream with a string, allowing easier access to contents.