Running tests in separate processes

[JoelLarson]

The feature I am inquiring about is runInSeparateProcess from PHPUnit. I am using Mockery to stub out some classes with the alias and overload classes found here.

So my questions are:

• Is this type of functionality something that is desired?
• What would be the desired implementation to do this? (I'm not really interested in seeing it done with annotations)

I would have no problem implementing the feature, but I would like some feedback before I start.

---

I am also aware of peridot-concurrency which works very similarly to what I want, however I would like for a test to have a dedicated process to allow for overloading the classes in an autoloader without affecting the rest of the tests.

[brianium]

Thanks for opening the issue! I'm not 100% sure on the mechanics for this, but it seems doable - especially with some of the enhancements we have on the way with 2.0.

1.x is a little limited when it comes to isolating and and running single it blocks, but with 2.0 we are working on completely changing how the --grep functionality works. We are looking at something similar to Mocha's concept of tagging for use with --grep.

This should make process isolation for a single it much easier, and should allow multiple it blocks to run concurrently in the peridot concurrency plugin.

For version 1.x, I imagine this could be done via a plugin that uses a similar concept?

it('should be run in @isolation', function() {

});

// peridot.php
$emitter->on('test.start', function ($test) {
    if (preg_match('/@isolation/', $test->getDescription())) {
        // run in isolation via process? forking?
        // prevent test from running in main process - not sure if there is an elegant way to do this yet
    } 
});

It seems like the functionality that needs to be added to core is the ability to halt a test (similar to what suites do already?) so the test doesn't run twice. Some place early in the test's run method. Seems like that would be super useful in general for plugins and adhoc functionality :)

I'll open an issue for that as well and reference this one.

Thoughts?

[JoelLarson]

Having @isolation in the test description seems kind of strange to me. It doesn't read very well and I feel the isolation is more of an implementation detail of the tests rather than what it is testing.

I came up with a solution yesterday that I was hoping would work, but it seems it'll have to be rethought slightly.

The idea was something like:

<?php

describe('ObjectBuilder', function () {
    context('when building', function () {
        it('should be assembled correctly', new IsolatedTest(function () {
            expect(1)->to->equal(1);
        }));
    });
});

I attempted to write IsolatedTest with an __invoke() method hoping that Closure::bind() would treat the __invoke() like a closure, but alas it didn't work. An alternative would be to rewrite the core slightly to allow for it, or possibly use the emitter hooks to watch for the test and run it.

The IsolatedTest class would be able to encapsulate the process of forking and setting up everything without having to scatter that knowledge across the codebase.

[brianium]

Interesting. I wonder if there is a language that could be created for this, as a custom DSL perhaps? A different function could facilitate using the IsolatedTest class. Again language is the key, but what about something like:

describe('ObjectBuilder', function () {
    context('when building', function () {
        independently('should be assembled correctly', function () {
            expect(1)->to->equal(1);
        });
    });
});

And independently (or some other aptly named function) could construct an IsolatedTest and insert it into the test hierarchy?

[JoelLarson]

I really like the independently DSL. However, I still don't feel it reads very well. What about something like this?

describe('ObjectBuilder', function () {
    context('when building', function () {
        independently(function () {
            it('should be assembled correctly', function () {
                expect(1)->to->equal(1);
            });
        });
    });
});

At the same time though, the isolation doesn't really describe the test. It just serves as meta on the test to tell it how to execute, rather than describing that the test should work as expected in isolation.

Maybe using annotations is the way to go then?

describe('ObjectBuilder', function () {
    context('when building', function () {
        /**
         * @runInSeparateProcess
         */
        it('should be assembled correctly', function () {
            expect(1)->to->equal(1);
        });
    });
});```

[brianium]

Hmmm.. it's either describe test behavior with annotations or with functions. I personally think a DSL describes test behavior better than annotations, and I suspect it will be easier to implement, and would probably introduce less overhead.

Maybe there is a better word than independently? I think either implementation would be useful, but my personal vote would be dsl :) @austinsmorris do you have any input on this?

[austinsmorris]

What if the DSL was updated so that it() had an optional third argument?

describe('ObjectBuilder', function () {
    context('when building', function () {
        it('should be assembled correctly', function () {
            expect(1)->to->equal(1);
        }, Blah::ISOLATION);
    });
});

[JoelLarson]

I had that thought come across as well, though for extendability reasons I figured wrapping the test in a method would be better than that. Then you wouldn't have to modify the test builder whenever something is added.

Also a third parameter doesn't feel very accommodating when you want to run multiple tests in isolation. It would have to be duplicated or carried into other DSL functions.

[brianium]

Going to document some examples here just to keep things grouped.

I can't really find a good example of this being done in similar frameworks. The cleanest path seems to be a custom dsl or grep functionality. I found this (albeit old) concept for rspec here - and they provide an iso_it function.

I am not too keen on supporting invokable objects because it seems like it would complicate sharing and inheriting of scopes. Using grep functionality seems like a clean way to do it (and is what I am leaning towards), and will be pretty easy once #156 is added and #114 is complete

describe('thing', function () {
  it('should use constant as @isolated', function () {
     // test a thing
  });
});

yes it still muddies the language a bit, but the need to run in isolation seems like an exception, and really shouldn't be too common in a test base. If it is common, maybe something can be put in a beforeEach that says "hey run all of these in isolation"

[AustP]

I was looking for this same functionality. Upon finding this open issue, I created my own plugin that allows running tests in a separate process. I prefer the Gherkin language for BDD so the plugin follows that for the DSL. The function that runs the tests in a separate process is isolatedScenario.

https://github.com/AustP/peridot-gherkin-plugin