« Changelog History


language-ext v3.4.14-beta Release Notes

Release Date: 2020-03-05 // about 4 years ago

  • An ongoing thorn in my side has been the behaviour of Traverse and Sequence for certain pairs of monadic types (when nested). These issues document some of the problems:

    #552
    #651

    The Traverse and Sequence functions were previously auto-generated by a T4 template, because for 25 monads that's 25 * 25 * 2 = 1250 functions to write. In practice it's a bit less than that, because not all nested monads should have a Traverse and Sequence function, but it is in the many hundreds of functions.

    Because the same issue kept popping up I decided to bite the bullet and write them all by hand. This has a number of benefits:

    • 🏗 The odd rules of various monads when paired can have bespoke code that makes more sense than any auto-generated T4 template could ever build. This fixes the bugs that keep being reported and removes the surprising nature of Traverse and Sequence working most of the time, but not in all cases.
    • 🐎 I'm able to hand-optimise each function based on what's most efficient for the monad pairing. This is especially powerful for working with Traverse and Sequence on list/sequence types. The generic T4 code-gen had to create singleton sequences and the concat them, which was super inefficient and could cause stack overflows. Often now I can pre-allocate an array and use a much faster imperative implementation with sequential memory access. Where possible I've tried to avoid nesting lambdas, again in the quest for performance but also to reduce the amount of GC objects created. I expect a major performance boost from these changes.
    • The lazy stream types Seq and IEnumerable when paired with async types like Task, OptionAsync, etc. can now have bespoke behaviour to better handle the concurrency requirements (These types now have TraverseSerial and SequenceSerial which process tasks in a sequence one-at-a-time, and TraverseParallel and SequenceParallel which processes tasks in a sequence concurrently with a window of running tasks - that means it's possible to stop the Traverse or Sequence operation from thrashing the scheduler.

    Help

    🐎 Those are all lovely things, but the problem with writing several hundred functions manually is that there's gonna be bugs in there, especially as I've implemented them in the most imperative way I can to get the max performance out of them.

    🏗 I have just spent the past three days writing these functions, and frankly, it was pretty soul destroying experience - the idea of writing several thousand unit tests fills me with dread; and so if any of you lovely people would like to jump in and help build some unit tests then I would be eternally grateful.

    Sharing the load on this one would make sense. If you've never contributed to an open-source project before then this is a really good place to start!

    I have...

    • 🚀 Released the updates in 3.4.14-beta - so if you have unit tests that use Traverse and Sequence then any feedback on the stability of your tests would be really helpful.
    • ✅ Created a github project for managing the cards of each file that needs unit tests. It's the first time using this, so not sure of its capabilities yet, but it would be great to assign a card to someone so work doesn't end up being doubled up.
    • The code is in the hand-written-traverse branch.
    • The folder with all the functions is transformers/traverse

    Things to know

    • Traverse and Sequence take a nested monadic type of the form MonadOuter<MonadInner<A>> and flips it so the result is MonadInner<MonadOuter<A>>
    • If the outer-type is in a fail state then usually the inner value's fail state is returned. i.e. Try<Option<A>> would return Option<Try<A>>.None if the outer Try was in a Fail state.
    • If the inner-type is in a fail state then usually that short-cuts any operation. For example Seq<Option<A>> would return an Option<Seq<A>>.None if any of the Options in the Seq were None.
    • 👻 Where possible I've tried to rescue a fail value where the old system returned Bottom. For example: Either<Error, Try<A>>. The new system now knows that the language-ext Error type contains an Exception and can therefore be used when constructing Try<Either<Error, A>>
    • 🏁 All async pairings are eagerly consumed, even when using Seq or IEnumerable. Seq and IEnumerable do have windows for throttling the consumption though.
    • 🚚 Option combined with other types that have an error value (like Option<Try<A>>, Option<Either<L, R>>, etc.) will put None into the resulting type (Try<Option<A>>(None), Either<L, Option<A>>(None) if the outer type is None - this is because there is no error value to construct an Exception or L value - and so the only option is to either return Bottom or a success value with None in it, which I think is slightly more useful. This behaviour is different from the old system. This decision is up for debate, and I'm happy to have it - the choices are: remove the pairing altogether (so there is no Traverse or Sequence for those types) or return None as described above

    ✅ Obviously, it helps if you understand this code, what it does and how it should work. I'll make some initial tests over the next few days as guidance.

    Discussion will be here