Are there any software #testing frameworks that support testing outcomes other than PASS and FAIL?
-
Are there any software #testing frameworks that support testing outcomes other than PASS and FAIL? E.g. “message successfully received but content needed to be lossily reformatted”? Wondering how APIs would look like. #programming
-
@J12t Realize this isn’t exactly what you’re asking, but the way I would achieve that in a “traditional” testing framework would be: One test confirms response != null, that test is passing, second test confirms that data format is as expected, that test fails.
-
@philip For Fediverse testing, I"m looking to produce reports that say something like Pass / Fail / Degraded where Degraded is things like "objects regardless of type have become a Note". Thinking of maybe having two types of AssertionErrors; still has the problem that if they are thrown as exceptions, a test will stop right there.
-
@J12t Minitest, one of the standard Ruby libs, has pass/fail/exception/skip as the possible states, plus some additional info about them.
You can also assert some things about the exception if you want by catching-and-checking it. Or assert other things from outside the test framework -- a Minitest skip, failure or exception is a Ruby exception, for instance, and can be handled as such.
-
@codefolio Seems the approach boils down to throws / does not throw Exception, and if it does, categorize Exceptions into various buckets so they get reported in different ways.
-
@J12t @philip The testing mantra is, every test should test just one thing, and I don't see a good reason to depart from that rule here. If there are two things that can go wrong, run the test twice. Efficiency shouldn't be a concern at this point of the pipeline.
"Degraded" doesn't really say anything actionable to me as a developer or a user. I'd be better off just looking at a straight percentage of failed tests, and diving into the details of the ones that "fail". I'd certainly much rather have a clear yes/no answer to each individual question.
I certainly wouldn't choose my testing framework based on the requirement that it doesn't just pass or fail. Best to treat your difficulty in finding such a thing as a message.