Popularity
3.1
Growing
Activity
7.3
Growing
168
10
26

Programming language: C#
License: MIT License
Tags: Crawler     Spider     Netstandard    
Latest version: v0.3.0

Infinity Crawler alternatives and similar packages

Based on the "Tools" category.
Alternatively, view Infinity Crawler alternatives based on common mentions on social networks and blogs.

Do you think we are missing an alternative of Infinity Crawler or a related project?

Add another 'Tools' Package

README

[Icon](images/icon.png)

Infinity Crawler

A simple but powerful web crawler library for .NET

Build Codecov NuGet

Features

  • Obeys robots.txt (crawl delay & allow/disallow)
  • Obeys in-page robots rules (X-Robots-Tag header and <meta name="robots" /> tag)
  • Uses sitemap.xml to seed the initial crawl of the site
  • Built around a parallel task async/await system
  • Swappable request and content processors, allowing greater customisation
  • Auto-throttling (see below)

Licensing and Support

Infinity Crawler is licensed under the MIT license. It is free to use in personal and commercial projects.

There are support plans available that cover all active Turner Software OSS projects. Support plans provide private email support, expert usage advice for our projects, priority bug fixes and more. These support plans help fund our OSS commitments to provide better software for everyone.

Polite Crawling

The crawler is built around fast but "polite" crawling of website. This is accomplished through a number of settings that allow adjustments of delays and throttles.

You can control:

  • Number of simulatenous requests
  • The delay between requests starting (Note: If a crawl-delay is defined for the User-agent, that will be the minimum)
  • Artificial "jitter" in request delays (requests seem less "robotic")
  • Timeout for a request before throttling will apply for new requests
  • Throttling request backoff: The amount of time added to the delay to throttle requests (this is cumulative)
  • Minimum number of requests under the throttle timeout before the throttle is gradually removed

Other Settings

  • Control the UserAgent used in the crawling process
  • Set additional host aliases you want the crawling process to follow (for example, subdomains)
  • The max number of retries for a specific URI
  • The max number of redirects to follow
  • The max number of pages to crawl

Example Usage

using InfinityCrawler;

var crawler = new Crawler();
var result = await crawler.Crawl(new Uri("http://example.org/"), new CrawlSettings {
    UserAgent = "MyVeryOwnWebCrawler/1.0",
    RequestProcessorOptions = new RequestProcessorOptions
    {
        MaxNumberOfSimultaneousRequests = 5
    }
});


*Note that all licence references and agreements mentioned in the Infinity Crawler README section above are relevant to that project's source code only.