PIO Remote builds tests and then executes them all, resulting in only the last-built test executing

[tanishqaggarwal]

Temporary solution

Please use -r, --force-remote option for pio remote test command.

---

Configuration

Operating system: MacOS X

PlatformIO Version (platformio --version): 4.1.0

Description of problem

When testing on remote platforms with Teensy, if there are many tests specified for a single environment, the remote agent builds all the tests and then runs them. Unfortunately this only results in a

See post here: https://community.platformio.org/t/pio-remote-testing-bug/10496

Steps to Reproduce

1. Write some unit tests for the Teensy environment and have an environment run all of them
2. Run them on a remote agent: platformio remote --agent <name> test -e <env>

Actual Results

The code for the tests is built in series, but only the test built last is actually uploaded to the device for testing. See
https://travis-ci.org/pathfinder-for-autonomous-navigation/CommonSoftware/jobs/610692665?utm_medium=notification&utm_source=github_status

Expected Results

The tests are executed one-by-one.

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:teensy]
platform = teensy
board = teensy35
framework = arduino
test_build_project_src = true
src_filter = +<*>
test_filter = *
upload_protocol = teensy-cli

Workaround

Use the --force-remote option in the remote test command.

[sticilface]

It looks like this issue is still here. I am unable to run more than one unit test using PIO remote..

My test folder contains 3 tests, simple, complex, datatest.

Simple:

#include "unity.h"
#include <Arduino.h>

void setUp(void) {
}

void tearDown(void) {
}

void test_function_simple(void) {
}

int runUnityTests(void) {
  UNITY_BEGIN();
  RUN_TEST(test_function_simple); 
  return UNITY_END();
}


int main(void) {
  return runUnityTests();
}


void setup() {
  Serial.begin(115200); 
  delay(2000);
  runUnityTests();
}

void loop() {}

Complex:

#include "unity.h"
#include <Arduino.h>

void setUp(void) {
}

void tearDown(void) {
}

void test_function_complex(void) {
}

int runUnityTests(void) {
  UNITY_BEGIN();
  RUN_TEST(test_function_complex); 
  return UNITY_END();
}

int main(void) {
  return runUnityTests();
}

void setup() {
  Serial.begin(115200); 
  delay(2000);
  runUnityTests();
}
void loop() {}

Datatest:

• This is my main test, with about 20 functions to be tested

int runUnityTests(void) {
  UNITY_BEGIN();

  RUN_TEST(test_function_bool); 
  RUN_TEST(test_function_uint8_t);
  RUN_TEST(test_function_uint16_t);
  RUN_TEST(test_function_uint32_t);
  RUN_TEST(test_function_float);
  RUN_TEST(test_function_string); 
  RUN_TEST(test_function_last);
  RUN_TEST(test_function_walkall);
  RUN_TEST(test_function_device2_send_bool); 
  RUN_TEST(test_function_device2_send_uint8_t); 
  RUN_TEST(test_function_device2_send_uint16_t); 
  RUN_TEST(test_function_device2_send_uint32_t); 
  RUN_TEST(test_function_device2_send_float); 
  RUN_TEST(test_function_device2_send_string); 
  RUN_TEST(test_function_callback_1); 
  RUN_TEST(test_function_callback_2); 
  RUN_TEST(test_function_callback_3); 
  RUN_TEST(test_function_callback_4); 

  return UNITY_END();
}

When I run the tests..

 *  Executing task in folder Test Board: platformio remote test --environment mega_A_LEDS 

Building project locally
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 3 tests

Processing embedded/test_simple in mega_A_LEDS environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------------------------- mega_A_LEDS:embedded/test_simple [PASSED] Took 0.69 seconds -------------------------------------------------------------------------------------------------------

Processing embedded/test_datatest in mega_A_LEDS environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------------------------ mega_A_LEDS:embedded/test_datatest [PASSED] Took 0.64 seconds ------------------------------------------------------------------------------------------------------

Processing embedded/test_complex in mega_A_LEDS environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------------------------- mega_A_LEDS:embedded/test_complex [PASSED] Took 0.55 seconds -------------------------------------------------------------------------------------------------------

================================================================================================================ 0 test cases: 0 succeeded in 00:00:01.874 ================================================================================================================
Testing project remotely
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 3 tests

Processing embedded/test_complex in mega_A_LEDS environment
--------------------------------------------------------------------------------
Building & Uploading...
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e950f (probably m328p)

avrdude: processing -U flash:w:.pio/build/mega_A_LEDS/firmware.hex:i
avrdude: reading input file .pio/build/mega_A_LEDS/firmware.hex for flash
         with 2638 bytes in 1 section within [0, 0xa4d]
         using 21 pages and 50 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 2638 bytes flash ...
Writing | ################################################## | 100% 1.01s
avrdude: 2638 bytes of flash written
avrdude: verifying flash memory against .pio/build/mega_A_LEDS/firmware.hex
Reading | ################################################## | 100% 0.67s
avrdude: 2638 bytes of flash verified

avrdude done.  Thank you.

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/embedded/test_complex/complex.cpp:20: test_function_complex        [PASSED]
--------- mega_A_LEDS:embedded/test_complex [PASSED] Took 6.74 seconds ---------

Processing embedded/test_simple in mega_A_LEDS environment
--------------------------------------------------------------------------------
Building & Uploading...
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e950f (probably m328p)

avrdude: processing -U flash:w:.pio/build/mega_A_LEDS/firmware.hex:i
avrdude: reading input file .pio/build/mega_A_LEDS/firmware.hex for flash
         with 2638 bytes in 1 section within [0, 0xa4d]
         using 21 pages and 50 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 2638 bytes flash ...
Writing | ################################################## | 100% 1.01s
avrdude: 2638 bytes of flash written
avrdude: verifying flash memory against .pio/build/mega_A_LEDS/firmware.hex
Reading | ################################################## | 100% 0.67s
avrdude: 2638 bytes of flash verified

avrdude done.  Thank you.

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/embedded/test_complex/complex.cpp:20: test_function_complex        [PASSED]
--------- mega_A_LEDS:embedded/test_simple [PASSED] Took 6.48 seconds ---------

Processing embedded/test_datatest in mega_A_LEDS environment
--------------------------------------------------------------------------------
Building & Uploading...
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e950f (probably m328p)

avrdude: processing -U flash:w:.pio/build/mega_A_LEDS/firmware.hex:i
avrdude: reading input file .pio/build/mega_A_LEDS/firmware.hex for flash
         with 2638 bytes in 1 section within [0, 0xa4d]
         using 21 pages and 50 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 2638 bytes flash ...
Writing | ################################################## | 100% 1.01s
avrdude: 2638 bytes of flash written
avrdude: verifying flash memory against .pio/build/mega_A_LEDS/firmware.hex
Reading | ################################################## | 100% 0.67s
avrdude: 2638 bytes of flash verified

avrdude done.  Thank you.

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/embedded/test_complex/complex.cpp:20: test_function_complex        [PASSED]
-------- mega_A_LEDS:embedded/test_datatest [PASSED] Took 6.74 seconds --------

=================================== SUMMARY ===================================
Environment    Test                    Status    Duration
-------------  ----------------------  --------  ------------
mega_A_LEDS    embedded/test_complex   PASSED    00:00:06.744
mega_A_LEDS    embedded/test_simple    PASSED    00:00:06.484
mega_A_LEDS    embedded/test_datatest  PASSED    00:00:06.735
================== 3 test cases: 3 succeeded in 00:00:19.963 ==================
 *  Terminal will be reused by tasks, press any key to close it. 

You can see that it has build all 3 tests but then it uploads the same one (complex) as that is the only test function returned by the test

writing 2638 bytes flash ...

Given that this issue is 5 years old is there any chance of fixing it?