Tests that use staging-fuel.ignitionrobotics.org are failing

[chapulina]

Environment

• OS Version: all
• Source or binary build? all

Description

• Expected behavior: All tests pass
• Actual behavior: Tests are failing

Steps to reproduce

Run UNIT_ign_src_TEST. Many tests there assume that the staging server has some users and models.

Output

Example test:

gz-fuel-tools/src/ign_src_TEST.cc

Lines 107 to 124 in 21c5eb5

	TEST_F(CmdLine, ModelListConfigServerPretty)
	{
	EXPECT_TRUE(listModels("https://staging-fuel.ignitionrobotics.org"));
	EXPECT_NE(this->stdOutBuffer.str().find(
	"https://staging-fuel.ignitionrobotics.org"),
	std::string::npos) << this->stdOutBuffer.str();
	EXPECT_NE(this->stdOutBuffer.str().find("owners"), std::string::npos)
	<< this->stdOutBuffer.str();
	EXPECT_NE(this->stdOutBuffer.str().find("models"), std::string::npos)
	<< this->stdOutBuffer.str();
	EXPECT_EQ(this->stdOutBuffer.str().find("https://fuel.ignitionrobotics.org"),
	std::string::npos) << this->stdOutBuffer.str();
	EXPECT_EQ(this->stdOutBuffer.str().find(
	"https://staging-fuel.ignitionrobotics.org/1.0/"), std::string::npos)
	<< this->stdOutBuffer.str();
	}

And the failure, note the Either failed to fetch model list, or server has no models yet.

[125.129s] 7: [ RUN      ] CmdLine.ModelListConfigServerPretty
[125.476s] 7: ^[[1;31m[Err] [RestClient.cc:327] ^[[0m^[[1;31mError in REST request^[[0m^[[1;31m^[[0m
[125.476s] 7:
[125.476s] 7: libcurl: (60) SSL: no alternative certificate subject name matches target host name 'staging-fuel.ignitionrobotics.org'
[125.476s] 7: ^[[1;33m[Wrn] [FuelClient.cc:317] ^[[0m^[[1;33mFailed to fetch models from server, returning cached models.^[[0m^[[1;33m^[[0m
[125.476s] 7: ^[[1;33mURL: https://staging-fuel.ignitionrobotics.org
[125.476s] 7: Version: 1.0
[125.476s] 7: API key: ^[[0m
[125.476s] 7: ^[[1;33m^[[0m
[125.476s] 7: ^[[1;33m[Wrn] [LocalCache.cc:105] ^[[0m^[[1;33mServer directory does not exist [^[[0m^[[1;33m/staging-fuel.ignitionrobotics.org^[[0m^[[1;33m]^[[0m
[125.476s] 7: /home/chapulina/dev_focal/ws_garden/src/ign-fuel-tools/src/ign_src_TEST.cc:114: Failure
[125.476s] 7: Expected: (this->stdOutBuffer.str().find("owners")) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615
[125.476s] 7: Fetching model list from https://staging-fuel.ignitionrobotics.org...
[125.477s] 7: Either failed to fetch model list, or server has no models yet.
[125.477s] 7:
[125.477s] 7: [  FAILED  ] CmdLine.ModelListConfigServerPretty (347 ms)

Suggested solution

Some possible solutions:

• Make sure that server used on the test is always populated
• Change the test to a non-staging server
• Change the test to use a mock server that doesn't rely on the actual internet

[mjcarroll]

Change the test to use a mock server that doesn't rely on the actual internet

I think this should be the case for any UNIT_ test in fuel-tools. I noticed that the Model tests have blown up in time recently because they iterate over all of the available models in fuel.

I think it would still be appropriate to have them in INTEGRATION_ tests, but the tests that access external network resources should be clearly labelled.