v3.0.0-beta.1November 14, 2020
v3.0.0-alpha.3September 19, 2020
🛠 Fixes #246 "Enable
RollForward = Majorto run on later .NET Core versions": For the 2.x line, the Fixie.Console installed by DotNetCliToolReference respects major version roll-forward, so that consumers no longer have to install and old SDK just for invoking tests compiled with more recent SDKs.
🚀 The previous release supported .NET Core 3 test projects for most use cases, but as a happy coincidence. No action had to be taken in Fixie's implementation, as the existing support for .NET Core 2 carried forward naturally.
👍 Fixie 2.2.1 includes official support for such test projects. However, because
netcoreapp3.0has already reached End-of-Life,
netcoreapp3.0is not supported. As a Long-Term-Support release,
netcoreapp3.1+ is supported. If you need to target
netcoreapp3.0, we recommend sticking with Fixie 2.2.0 until you can upgrade to a target framework supported by Microsoft.
✨ Enhancements for .NET Core Test Projects:
- 👌 Improved readability of test failure stack traces for
- 🐎 When consumers target
netcoreapp3.1+, internal messaging serialization uses
System.Text.Json.JsonSerializerinstead of the deprecated
DataContractJsonSerializer, for its simpler API and improved performance.
- 🐎 Update VS Test Platform dependencies to take advantage of their performance improvements.
- 🛠 Fixed the ability to run tests under the debugger in Visual Studio's Test Explorer for
netcoreapp3.1+ test projects.
✨ Enhancements when running on Azure DevOps:
- ✅ When a test assembly completes, the test run as a whole is marked as completed. Doing so lights up several elements in the Azure DevOps "Tests" screen: assembly-level success/failure icons, assembly-level durations, and high-level statistics.
- ✅ When posting skipped test results to Azure DevOps, use the 'outcome' value 'Warning'. This allows the Azure DevOps UI to properly categorize the result, to display a warning icon by the result, and to pass through the skip reason as a visible message. Don't skip tests.
- 🏗 #211: Improved logging and error handling around Azure DevOps API failures, allowing test runs to complete with full details in the build log and an accurate exit code even when the Azure DevOps API fails completely.
- ✅ Because AppVeyor relies on test case name + the assembly
FileNameto distinguish unique results, include the target framework in that
FileName. This way, test results are correctly distinguished from each other during test runs of multitargeted test assemblies.
- 📦 Modernize deprecated elements in NuGet packages.
- 👌 Improved readability of test failure stack traces for
📚 When running under Azure DevOps, if you have configured your Pipeline to allow access to the Azure DevOps test reporting API, Fixie will report each test result so that they will appear on the build's "Tests" screen. If you forget that configuration, the console output will provide guidance based on the Azure DevOps documentation (https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables#systemaccesstoken)
✅ When running tests in Visual Studio under Test Explorer:
- 👌 Improved error reporting, both in the Output pane and in the tree of test results, to help with troubleshooting.
- 👌 Improved performance when selecting a large number of individual tests to run in Test Explorer. Previously, we performed an excessive amount of reflection and redundant evaluation of which test classes and test methods need to be executed.
- ↔ Integrate with Test Explorer's test progress icons, so that the user can identify long-running tests.
- ✅ Graceful handling of catastrophic failures such as tests which perform Environment.Exit(...) or which throw StackOverflowException. Naturally such catastrophic errors force the test run to end early, and that is out of our control, but now we accurately report what happened in both the Test Explorer tree and in the 'Tests' Output pane.
v2.1.0July 24, 2019
🚀 Release 2.0.2 attempted to integrate with breaking changes in Visual Studio 15.8. It failed, however, to properly deal with the following combination:
- ✅ Test assembly targets Full .NET Framework
- 🐎 Enabling the new option Tools \ Options \ Tests \ "For improved performance, only use test adapters in test assembly folder..."
✅ In this scenario, Test Explorer would fail to locate a dependency (Mono.Cecil) causing those tests to disappear from Test Explorer.
🚀 With this release, Tests Explorer is expected to behave well regardless of the new Tools \ Options \ Tests setting.
✅ Handling Stale Test Discovery Information
✅ Because test runners like Test Explorer can request specific tests to run using stale discovery data (such as selecting a test to run which has been renamed but not yet recompiled), we now take care to only execute requested tests which in fact exist.
✅ Test Exception Handling
✅ We simplified the way that test exceptions are captured and propagated while preserving original stack traces.
✅ For async tests, this means favoring task.GetAwaiter().GetResult() over task.Wait(), simplifying our implementation and providing a more readable stack trace.
We leverage ExceptionDispatchInfo.Throw(), though we take care to omit the portion of the resulting stack trace that verbosely describes Fixie's own calling of that method.
🐛 Bugs Fixed
- When formatting output under TeamCity, ensure all non-ASCII characters are properly escaped.
- ✅ When asynchronous F# test cases return the F#-specific
Async<'T>type, ensure that execution has completed before determining the case's result.
- ✅ Ensure that execution of any asynchronous test case completes before determining the case's result, whether or not the
asynckeyword is present, even when the
Taskobject is returned from a method declared with
objectas the return type.
- ✅ Asynchronous test cases which return the
nulltask always pass, as there is no more work to perform.
- ✅ Asynchronous test cases fail with a clear explanation, instead of blocking indefinitely, when the
Taskreturned was never started.
Huge thanks to @petejohanson!
✅ Visual Studio 15.8 included an intentional breaking change to the way Test Explorer discovers third party "test adapters" like Fixie.VisualStudio.TestAdapter.dll.
Historically, Test Explorer would search for .NET Core adapters only in the build output folder, while it would search for .NET Framework adapters in both the build output folder and the referenced NuGet packages. When Fixie 2.0.0 was released, we took advantage of the NuGet package search in order to avoid having to wastefully copy the adapter DLL to the build outpuf folder for .NET Framework builds.
🚀 Visual Studio 15.8 includes a breaking change in the interest of performance. The search no longer considers NuGet package contents when discovering test adapters. This release fixes Test Explorer integration by including the same DLL copy for .NET Framework projects as we've been doing for .NET Core projects.
✅ As a side effect,
dotnet testintegration is likewise improved, though we still recommend using