Remove dependencies between tests

[untitaker]

I think we should get rid of the tests dependency on each other, i.e. wipe the server after each test. This is slow but will let us keep our sanity.

https://github.com/remotestorage/api-test-suite/pull/32/files#r44957676

[gregkare]

Agreed. It was easier to do it like this to get started but now it's getting confusing (and it will keep getting more confusing as we add more specs)

[raucao]

I don't see how that's possible, because you can only use other methods than the one to be tested in order to tear down what you did. If you don't have access to the server itself, you cannot test DELETE without a working PUT. You cannot test GET without a working PUT. You cannot clean up PUT without a working DELETE. And so on and so forth...

Am I missing something?

[untitaker]

You would explicitly set up the appropriate state for each test. That presumes very basic upload, update and delete to work, otherwise everything fails. But that's not different from now.

On 16 November 2015 23:15:10 CET, Sebastian Kippe notifications@github.com wrote:

I don't see how that's possible, because you can only use other methods
than the one to be tested in order to tear down what you did. If you
don't have access to the server itself, you cannot test DELETE without
a working PUT. You cannot test GET without a working PUT. You cannot
clean up PUT without a working DELETE. And so on and so forth...

Am I missing something?

---

Reply to this email directly or view it on GitHub:
#33 (comment)

Sent from my phone. Please excuse my brevity.

[raucao]

You don't see how you still have dependencies throughout the whole suite with that?

[untitaker]

No.

Assuming wiping the server doesn't fail (in which case the testsuite should probably abort), you always start with an empty storage.

On 16 November 2015 23:19:01 CET, Sebastian Kippe notifications@github.com wrote:

You don't see how you still have dependencies throughout the whole
suite with that?

---

Reply to this email directly or view it on GitHub:
#33 (comment)

Sent from my phone. Please excuse my brevity.

[raucao]

You cannot start with an empty storage when DELETEs fail. And implementing DELETEs before PUTs is also not really something that makes sense. No?

[raucao]

By the way, the point is you can't even know if something was deleted just by way of its return code. That's what I mean by still having dependencies and not being able to be sure about anything.

[untitaker]

Server authors are still expected to read the spec, and not do TDD using the testsuite. With that in mind it IMO doesn't make sense to talk about order of implementing methods. I only see the suite as a helper to get details right.

On 16 November 2015 23:22:45 CET, Sebastian Kippe notifications@github.com wrote:

You cannot start with an empty storage when DELETEs fail. And
implementing DELETEs before PUTs is also not really something that
makes sense. No?

---

Reply to this email directly or view it on GitHub:
#33 (comment)

Sent from my phone. Please excuse my brevity.

[untitaker]

How can't you rely on a conformant server's return code?

On 16 November 2015 23:24:27 CET, Sebastian Kippe notifications@github.com wrote:

By the way, the point is you can't even know if something was deleted
just by way of its return code. That's what I mean by still having
dependencies and not being able to be sure about anything.

---

Reply to this email directly or view it on GitHub:
#33 (comment)

Sent from my phone. Please excuse my brevity.

[raucao]

How can't you rely on a conformant server's return code?

Because that's only one of the things you test for correctness. The return code alone does't tell you much, except that someone implemented the return code. That's the main problem with a suite like this: you're testing a blackbox, and the very methods you want to test are the same ones you need for setup and teardown of specs.

Any PR is welcome, of course. I just don't see how it would be less confusing and messy by adding more interdependent specs and have the whole thing take about 10 times longer. I'll gladly be convinced of the opposite, as always. :)