I think this is the most clever solution I've seen.

Thank you so much for your explanation and the function explaining it. I tested it and it worked with my assumptions in the way your functions explained. I shall look more into bitwise operations as well.

This kind of cheating is punishable, don't do this again.

.

Yeah, this kata is cool and all but it even rejected my code just because it had a blank newline, like yeah wth

Most entertainig part was to write a script that generates the solution

May I quote myself from before?

And you have to write the best random tests you can think of anyway.

More is better. Of course better is also better.

YES, you also have to write the best fixed tests you can think of.

I was just talking that if bug can happen, increasing number of same tests, with random data set and values in same domain doesn't actually help that much to catch it. It can help in some cases, but one needs to be lucky :) What can help for sure is to add more edge-cases covering for tests.

So for example to catch bug with sum called w/o arguments we need to change tests so that they will call it w/o arguments :) The thing is that, again, we need to extend data domain, not it's quantity. Generating current random test-cases in for loop won't help to catch this at all.

I hear you.

You do not rely on it only, but it does have its place.

Eliminate all possible bugs with fixed tests, and someone will invent a new bug in a solution that passes your tests (or your kata task is probably really very limited and uninteresting).

Mind you, if random testing (fuzz testing) exposes an edge condition you have no fixed test for (yet), you should definitely add a fixed test for it. But (1) that's often an iterative process and (2) you never know if you've covered all possible edge conditions.

I just noticed that you don't test with sum() at all. sum() should be 0 I guess. You can define sum to always take at least one argument, but then somebody will filter out zeroes in an of course misguided attempt to make his calculation more efficient, and be left with zero arguments, some day. Should he blow up the planet because he just went out of spec?

Really, I wish you good luck eliminating "things happening". Random testing won't eliminate it - but it will help reducing it. And I remain unconvinced you're going to fully eliminate it.

I just don't like relying on random in tests. Suffered from this a lot during E2E and integration testing, due to their nature. You're talking about thing called "fuzz testing" (https://en.wikipedia.org/wiki/Fuzzing), and still it's more about picking random data domain, than random data in just in bigger quantities. This kind of tests doesn't guarantee 100% coverage of control flow. In practice I've learned a lesson that if thing can happen, but the probability is extremely low - it will happen anyway, just give it a time. So it doesn't makes sense to lower this probability, you need to eliminate it.

Non-negative integers only is perfectly acceptable and indeed it allows users to focus on normal cases, which is generally a good thing.

That still leaves open the possibility of solutions having bugs you're not going to find with limited testing. With random tests, there's always the chance of slipping through the cracks, but with enough of 'em, the possibility is smaller, going to zero. With fixed tests, it's an all-or-nothing proposition - it either finds the bug, or it doesn't. If you haven't foreseen a particular bug, maybe because of it being too subtle, or it depending on something you don't do in your reference solution but some user out there thought was a good idea at the time, it probably won't be caught. This puts an onus on the random tests as well, to generate as many different inputs as reasonably possible, to have the highest possible chance of finding bugs.

Two recent examples of testing done not entirely good enough:

• User had to sort a list of values according to some criteria. User solution was only tested with lists of values all different from each other (both in fixed and random tests). At least one solution returned not list.sort() but list.uniq().sort(), if you get my drift. This was not called for according to the description, but it was never caught.
• Same kata: this very subtly buggy solution. With the specified input domain, it could not even be tested against! but the solution definitely is wonky.

Does all of this mandate hundreds of random tests? No, but the chance of finding a bug increases (a little) with every random test. So, within reason, more is better. And you have to write the best random tests you can think of anyway.

So it's not about amount of data, but about it's quality. Currently I test only against positive integers. My point was that increasing number of tests with current data will not reveal subtle bugs for sure, It'll only increase probability of this. And since it's random, user/tester can be lucky enough. On the other hand, putting "special" numbers like +/-Infinity and NaNs (also -0 :P) into the game will reveal some bugs in some solutions. That's a good argument. But also forcing coders to workaround edge-cases usually makes kata less enjoyable since solutiuns become ugly.

How about this: I'll put criteria into description that the sum function will be tested only against natural numbers and zero?

For example: description specifies "numbers". Include negative numbers, and non-integers numbers, possibly Infinity and NaN (they are Numbers!).

( When working with floating point numbers, make sure to assertApproxEquals instead of assertEquals ! )

It may cull out solutions with subtle bugs. I couldn't tell you what those subtle bugs might be, because they're subtle.

An honest solver will tell you when (and possibly why) a random test fails, so you can include a fixed test for that particular edge condition. There just may be edge conditions you haven't thought of.

( This is why it's advantegeous to have different reference and example solutions, and not just a bit different but actually, really, competely different. This is not documented anywhere that I know of BTW. It's just something I try to do in my own kata. )