Description
This is intended for use on flickering tests, where the reason for failure is an external dependency and the failure is transient, e.g:
- HTTP request over the network
- Database call that could deadlock, timeout etc...
Whenever a test includes real-world infrastructure, particularly when communicated with via the internet, there is a risk of the test randomly failing so we want to try and run it again. This is the intended use case of the library.
If you have a test that covers some flaky code, where sporadic failures are caused by a bug, this library should not be used to cover it up!
xRetry alternatives and similar packages
Based on the "Testing" category.
Alternatively, view xRetry alternatives based on common mentions on social networks and blogs.
-
Bogus
:card_index: A simple fake data generator for C#, F#, and VB.NET. Based on and ported from the famed faker.js. -
xUnit
xUnit.net is a free, open source, community-focused unit testing tool for the .NET Framework. -
SpecFlow
#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration -
Fluent Assertions
A very extensive set of extension methods that allow you to more naturally specify the expected outcome of a TDD or BDD-style unit tests. Targets .NET Framework 4.7, as well as .NET Core 2.1, .NET Core 3.0, .NET 6, .NET Standard 2.0 and 2.1. Supports the unit test frameworks MSTest2, NUnit3, XUnit2, MSpec, and NSpec3. -
AutoFixture
AutoFixture is an open source library for .NET designed to minimize the 'Arrange' phase of your unit tests in order to maximize maintainability. Its primary goal is to allow developers to focus on what is being tested rather than how to setup the test scenario, by making it easier to create object graphs containing test data. -
NBomber
Modern and flexible load testing framework for Pull and Push scenarios, designed to test any system regardless a protocol (HTTP/WebSockets/AMQP etc) or a semantic model (Pull/Push). -
Machine.Specifications
Machine.Specifications is a Context/Specification framework for .NET that removes language noise and simplifies tests. -
Compare-Net-Objects
What you have been waiting for :+1: Perform a deep compare of any two .NET objects using reflection. Shows the differences between the two objects. -
Verify
Verify is a snapshot tool that simplifies the assertion of complex data models and documents. -
WireMock.Net
WireMock.Net is a flexible library for stubbing and mocking web HTTP responses using advanced request matching and response templating. Based on the functionality from http://WireMock.org, but extended with more functionality. -
GenFu
GenFu is a library you can use to generate realistic test data. It is composed of several property fillers that can populate commonly named properties through reflection using an internal database of values or randomly created data. You can override any of the fillers, give GenFu hints on how to fill them. -
Canopy
f# web automation and testing library, built on top of Selenium (friendly to c# also) -
Expecto
A smooth testing lib for F#. APIs made for humans! Strong testing methodologies for everyone! -
NFluent
Smooth your .NET TDD experience with NFluent! NFluent is an ergonomic assertion library which aims to fluent your .NET TDD experience (based on simple Check.That() assertion statements). NFluent aims your tests to be fluent to write (with a super-duper-happy 'dot' auto-completion experience), fluent to read (i.e. as close as possible to plain English expression), but also fluent to troubleshoot, in a less-error-prone way comparing to the classical .NET test frameworks. NFluent is also directly inspired by the awesome Java FEST Fluent assertion/reflection library (http://fest.easytesting.org/) -
xBehave.net
✖ An xUnit.net extension for describing each step in a test with natural language. -
NSpec
A battle hardened testing framework for C# that's heavily inspired by Mocha and RSpec. -
SpecsFor
SpecsFor is a light-weight Behavior-Driven Development framework that focuses on ease of use for *developers* by minimizing testing friction. -
LightBDD
BDD framework allowing to create easy to read and maintain tests. -
ArchUnitNET
A C# architecture test library to specify and assert architecture rules in C# for automated testing. -
Fine Code Coverage
Visualize unit test code coverage easily for free in Visual Studio Community Edition (and other editions too) -
snapshooter
Snapshooter is a snapshot testing tool for .NET Core and .NET Framework -
Xunit.Gherkin.Quick
BDD in .NET Core - using Xunit and Gherkin (compatible with both .NET Core and .NET) -
SimpleStubs
*SimpleStubs* is a simple mocking framework that supports Universal Windows Platform (UWP), .NET Core and .NET framework. SimpleStubs is currently developed and maintained by Microsoft BigPark Studios in Vancouver. -
ExpressionToCode
Generates valid, readable C# from an Expression Tree. -
NScenario
Dead simple library for annotating steps of test case scenarios. -
NCrunch
An automated continuous & concurrent testing tool for Visual Studio. [$] -
ReportPortal
AI-powered Test Automation Dashboard. Acquire, aggregate and analyze test reports to ascertain release health.
Static code analysis for 29 languages.
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of xRetry or a related project?
Popular Comparisons
README
xRetry
Retry flickering test cases for xUnit and SpecFlow in dotnet.
When to use this
This is intended for use on flickering tests, where the reason for failure is an external dependency and the failure is transient, e.g:
- HTTP request over the network
- Database call that could deadlock, timeout etc...
Whenever a test includes real-world infrastructure, particularly when communicated with via the internet, there is a risk of the test randomly failing so we want to try and run it again. This is the intended use case of the library.
If you have a test that covers some flaky code, where sporadic failures are caused by a bug, this library should not be used to cover it up!
Usage: SpecFlow 3
Add the xRetry.SpecFlow
nuget package to your project.
Above any scenario or scenario outline that should be retried, add a @retry
tag, e.g:
@retry
Scenario: Retry three times by default
When I increment the default retry count
Then the default result should be 3
This will retry the test up to 3 times by default. You can optionally specify a number of times
to retry the test in brackets, e.g. @retry(5)
.
You can also optionally specify a delay between each retry (in milliseconds) as a second
parameter, e.g. @retry(5,100)
will run your test 5 times until it passes, waiting 100ms
between each attempt.
Note that you must not include a space between the parameters, as Cucumber/SpecFlow uses
a space to separate tags, i.e. @retry(5, 100)
would not work due to the space after the comma.
Usage: xUnit
Add the xRetry
nuget package to your project.
Facts
Above any Fact
test case that should be retried, replace the Fact
attribute, with
RetryFact
, e.g:
private static int defaultNumCalls = 0;
[RetryFact]
public void Default_Reaches3()
{
defaultNumCalls++;
Assert.Equal(3, defaultNumCalls);
}
This will retry the test up to 3 times by default. You can optionally specify a number of times
to retry the test as an argument, e.g. [RetryFact(5)]
.
You can also optionally specify a delay between each retry (in milliseconds) as a second
parameter, e.g. [RetryFact(5, 100)]
will run your test 5 times until it passes, waiting 100ms
between each attempt.
Theories
If you have used the library for retrying Fact
tests, using it to retry a Theory
should be very intuitive.
Above any Theory
test case that should be retried, replace the Theory
attribute with RetryTheory
, e.g:
// testId => numCalls
private static readonly Dictionary<int, int> defaultNumCalls = new Dictionary<int, int>()
{
{ 0, 0 },
{ 1, 0 }
};
[RetryTheory]
[InlineData(0)]
[InlineData(1)]
public void Default_Reaches3(int id)
{
defaultNumCalls[id]++;
Assert.Equal(3, defaultNumCalls[id]);
}
The same optional arguments (max retries and delay between each retry) are supported as for facts, and can be used in the same way.
Viewing retry logs
By default, you won't see whether your tests are being retried as we make this information available
via the xunit diagnostic logs but test runners will hide these detailed logs by default.
To enable them you must configure your xUnit test project to have diagnosticMessages
set to true
in the xunit.runner.json
.
See the xUnit docs for a full setup guide of their config file, or see
this projects own unit tests which has been set up with this enabled.
Contributing
Feel free to open a pull request! If you want to start any sizeable chunk of work, consider opening an issue first to discuss, and make sure nobody else is working on the same problem.
Running locally
To run locally, always build xRetry.SpecFlowPlugin
before the tests, to ensure MSBuild
uses the latest version of your changes.
If you install make
and go to the build
directory, you can run the following from the
terminal to build, run tests and create the nuget packages:
make ci
If that works, all is well!
Licence
[MIT](LICENSE)
*Note that all licence references and agreements mentioned in the xRetry README section above
are relevant to that project's source code only.