[ZH] Implement Headless Mode #651

helmutbuhler · 2025-04-11T12:00:32Z

This PR adds a commandline option -headless which runs the game without any graphics.
This is necessary to implement a silent replay checker (see issue #565). Note that I have that replay checker already implemented, but I will push that in another PR. This is just preparation for that.

Note that right now, if you pass in -headless, the game just runs the Shellmap. Since it's headless, you won't be able to see or do anything. But if you run it in Debug, you can enjoy looking at the log file getting bigger.

Headless mode is implemented here by essentially only running GameLogic, and not the GameClient. Unfortunately, GameLogic depends on the GameClient in some small hacky ways, which makes this a non-trivial task. See also this comment I added:

// TheSuperHackers @logic-client-separation helmutbuhler 04/11/2025
// Some information about the architecture and headless mode:
// The game is structurally separated into GameLogic and GameClient.
// The Logic is responsible for everything that affects the game mechanic and what is synchronized over
// the network. The Client is responsible for rendering, input, audio and similar stuff.
// 
// Unfortunately there are some places in the code that make the Logic depend on the Client.
// (Search for @logic-client-separation)
// That means if we want to run the game headless, we cannot just disable the Client. We need to disable
// the parts in the Client that don't work in headless mode and need to keep the parts that are needed
// to run the Logic.
// The following describes which parts we disable in headless mode:
//
//	GameEngine:
//		TheGameClient is partially disabled:
//			TheKeyboard = NULL
//			TheMouse = NULL
//			TheDisplay is partially disabled:
//				m_3DInterfaceScene = NULL
//				m_2DScene = NULL
//				m_3DScene = NULL
//				(m_assetManager remains!)
//			TheWindowManager = NULL
//			TheIMEManager = NULL
//			TheTerrainVisual is partially disabled:
//				TheTerrainTracksRenderObjClassSystem = NULL
//				TheW3DShadowManager = NULL
//				TheWaterRenderObj = NULL
//				TheSmudgeManager = NULL
//				TheTerrainRenderObject is partially disabled:
//					m_treeBuffer = NULL
//					m_propBuffer = NULL
//					m_bibBuffer = NULL
//					m_bridgeBuffer = NULL
//					m_waypointBuffer = NULL
//					m_roadBuffer = NULL
//					m_shroud = NULL
//		TheRadar = RadarDummy

As it is now, this PR sets all the client components as listed above to NULL and adds all the appropriate NULL-checks (which are sadly quite a few).
For TheRadar I added a dummy implementation to avoid the NULL-checks in the many callers (this doesn't make much sense in the other components though).

If we decide to fix these separation issues properly in the future, we will be able to revert many of the changes in this PR in the future.

Alternatively, we can also decide to just hide the window in headless mode and initialize DX without using it. Then almost all the changes in this PR become obsolete as well. The question is then whether this will still run in CI.

I would prefer to keep the changes in this PR and to clean up these separation issues properly long-term. Having logic and client properly separated can help in the future when we decide to increase the framerate of the client, or if we want to implement a headless server. It's also nice to automatically test that the logic works without the client.

…th logging enabled (if you define RELEASE_DEBUG_LOGGING in Debug.h)

…in GameLogic::startNewGame to improve CRC Logging

…e with -SaveDebugCRCPerFrame. Also remove Bool dumped (wasn't really used and was in the way for this new code)

…d and not intended because it spams the crclogging with mostly useless data.

…ewGame instead of GameLogic::prepareNewGame so it works when loading a game.

# Conflicts: # GeneralsMD/Code/GameEngine/Source/GameLogic/Object/Object.cpp

GeneralsMD/Code/GameEngine/Include/Common/Radar.h

feliwir · 2025-04-11T12:10:17Z

GeneralsMD/Code/GameEngine/Source/GameClient/GUI/ControlBar/ControlBar.cpp

@@ -1652,7 +1654,7 @@ const Image *ControlBar::getStarImage(void )
 	else
 		m_lastFlashedAtPointValue = ThePlayerList->getLocalPlayer()->getSciencePurchasePoints();

-	GameWindow *win= TheWindowManager->winGetWindowFromId( NULL, TheNameKeyGenerator->nameToKey( "ControlBar.wnd:ButtonGeneral" ) );
+	GameWindow *win= TheWindowManager ? TheWindowManager->winGetWindowFromId( NULL, TheNameKeyGenerator->nameToKey( "ControlBar.wnd:ButtonGeneral" ) ) : NULL;


Couldn't you do a check for headless at the beginning as you do in ControlBar::update ?

# Conflicts: # cmake/config.cmake

…eCode into headless_sh

GeneralsMD/Code/GameEngine/Source/Common/RandomValue.cpp

helmutbuhler · 2025-04-15T20:52:48Z

Alright, PR #544 is merged and I updated this PR, it's ready for review now. Please read the first post before you review though.

helmutbuhler · 2025-04-19T18:59:42Z

Quick update with merge conflict fix and an added previously missed null pointer check

xezon

I would like to see more null implementations where so possible. I suspect the null checks were only placed to avoid crashing, right? I suspect it still runs a lot of logic that is not necessary to run in headless?

GeneralsMD/Code/GameEngine/Source/GameClient/GUI/GUICallbacks/Diplomacy.cpp

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameLogic/W3DTerrainLogic.cpp

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DView.cpp

xezon · 2025-04-21T08:26:05Z

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DDisplay.cpp

@@ -457,12 +457,15 @@ W3DDisplay::~W3DDisplay()

 	// shutdown
 	Debug_Statistics::Shutdown_Statistics();
-	W3DShaderManager::shutdown();
+	if (!TheGlobalData->m_headless)


This looks suspicious. I would expect W3DDisplay to not even be created in Headless. It looks to be necessary for asset manager and ww math.

Yeah, it's unfortunate, but TerrainLogic uses the Asset Manager to get the Terrain data. When we move that data to the TerrainLogic, we'll be able to skip creation of W3DDisplay in headless mode and remove all those checks in that file.

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/BaseHeightMap.cpp

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DAssetManager.cpp

helmutbuhler · 2025-04-21T09:47:39Z

@xezon
Thanks for the review! Hopefully I can implement those requested changes soon.
But in the meantime, maybe it would be a good idea to also test whether these changes allow the game to run in a Github actions container without gpu support. Unfortunately I don't have much experience there, maybe someone else can do that?

# Conflicts: # GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DTerrainVisual.cpp

…Class

xezon

I do wonder if more of the null checks can be avoided with early checks

if (TheGlobalData->m_headless)
  return;

or

if (!TheGlobalData->m_headless)
  doRendering();

The advantage of these checks is

avoid more unecessary computations
make our headless related checks easier to find

I am also fond of the dummy implementations.

xezon · 2025-04-26T16:48:57Z

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameLogic/W3DTerrainLogic.cpp

@@ -281,6 +281,8 @@ Real W3DTerrainLogic::getGroundHeight( Real x, Real y, Coord3D* normal ) const
 {
 #define USE_THE_TERRAIN_OBJECT
 #ifdef USE_THE_TERRAIN_OBJECT
+	// TheSuperHackers @logic-client-separation helmutbuhler 04/11/2025


Are these logic-client-separation comments complete or are some potentially missing?

xezon · 2025-04-26T16:58:42Z

GeneralsMD/Code/GameEngine/Include/GameClient/GameClient.h

+//					m_waypointBuffer = NULL
+//					m_roadBuffer = NULL
+//					m_shroud = NULL
+//		TheRadar = RadarDummy


RadarDummy, DummyGameWindow, DummyGameWindowManager are not consistently named.

I suggest to put "Dummy" at the end so that it groups properly with the normal class.

Radar RadarDummy GameWindow GameWindowDummy GameWindowManager GameWindowManagerDummy

Alternatively can also use "Null" instead of "Dummy". Either name is fine.

xezon · 2025-04-26T17:19:50Z

GeneralsMD/Code/GameEngine/Include/GameClient/GameWindowManager.h

@@ -380,6 +380,49 @@ extern WindowMsgHandledType PassMessagesToParentSystem( GameWindow *window,
 																															 WindowMsgData mData2 );


+// TheSuperHackers @feature helmutbuhler 24/04/2025
+// GameWindow that does nothing. Used for Headless Mode.


I suggest to move these dummies into new header + source files because they are a bit on the larger side.

xezon · 2025-04-26T17:45:44Z

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DView.cpp

@@ -616,7 +616,7 @@ void W3DView::setCameraTransform( void )
 	buildCameraTransform( &cameraTransform );
 	m_3DCamera->Set_Transform( cameraTransform );

-	if (TheTerrainRenderObject) 
+	if (TheTerrainRenderObject && W3DDisplay::m_3DScene != NULL) 


Is a call into W3DView::setCameraTransform needed at all? Can this call not be prevented higher up?

xezon · 2025-04-26T17:46:31Z

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameLogic/W3DGhostObject.cpp

@@ -457,7 +458,7 @@ void W3DGhostObject::restoreParentObject(void)
 				//removed by the ghost object manager, so restore it.  If we have a render
 				//object that is in the scene, then it was probably added because the model
 				//changed while the object was ghosted (for damage states, garrison, etc.).
-				if (robj->Peek_Scene() == NULL)
+				if (robj->Peek_Scene() == NULL && W3DDisplay::m_3DScene != NULL)


Is a call to W3DGhostObject::restoreParentObject required? Perhaps this call can be prevented higher up?

xezon · 2025-04-26T17:48:46Z

GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DTerrainVisual.cpp

@@ -655,13 +658,17 @@ Bool W3DTerrainVisual::load( AsciiString filename )
 	 it = NULL;
 	}
 	// add our terrain render object to the scene
-	W3DDisplay::m_3DScene->Add_Render_Object( m_terrainRenderObject );
+	if (W3DDisplay::m_3DScene != NULL)


Looking at all these m_3DScene tests, I wonder if a few m_headless tests can be placed higher up the call stack or around these m_3DScene usages to avoid the bulk of rendering logic. Assuming that the rendering does not affect game logic, it should be possible to strip a lot of the render updates, no?

helmutbuhler added 22 commits March 29, 2025 19:08

Make CRC-related commandline arguments work when compiling Release wi…

e4b6f18

…th logging enabled (if you define RELEASE_DEBUG_LOGGING in Debug.h)

Add comments to Xfer.h

fb4881f

Add some comments to crc command line arguments

10991d6

Resest Frame Counter earlier in GameLogic::prepareNewGame instead of …

320cd7c

…in GameLogic::startNewGame to improve CRC Logging

Improved CRC Debugging: Optionally save crc data into a file per fram…

2e0a25c

…e with -SaveDebugCRCPerFrame. Also remove Bool dumped (wasn't really used and was in the way for this new code)

Improved logging for mismatch debugging.

adfd721

Make condition for checking TheCRCFirstFrameToLog consistent.

ba13902

Add comment regarding g_keepCRCSaves

1b2f5a8

Remove transform crclogging in Object::crc. This was likely copypaste…

fb5d977

…d and not intended because it spams the crclogging with mostly useless data.

Delete CRC Frame files properly

46a4ab4

Improve comments

9f7287e

Reset framecounter and call CRCDebugStartNewGame in GameLogic::startN…

4a22c73

…ewGame instead of GameLogic::prepareNewGame so it works when loading a game.

Fix TEAM_ID_INVALID

e536826

Some code cleanup in CRCDebug

21b4851

Merge branch 'hb_sh' into improve_crc_logging_ea

88a5b8e

# Conflicts: # GeneralsMD/Code/GameEngine/Source/GameLogic/Object/Object.cpp

Add define NORMAL_LOG_IN_CRC_LOG to control normal logging in crc logs.

2be0870

Add cmake options to control logging

3813f09

Add m_headless global flag to optionally run the game in headless mode.

83e7ec1

Add some comments where logic-client-separation is broken.

16aa073

Add some more checks for headless

4229f1d

Add comment about logic-client architecture and headless mode

c7c8b1d

Add -headless commandline option

f69e41f

feliwir reviewed Apr 11, 2025

View reviewed changes

helmutbuhler added 7 commits April 13, 2025 20:05

Fix headless when shutting down the engine.

7498de0

Rename RadarHeadless to RadarDummy

bf0a2e5

Simplify RadarDummy

7df3c68

Change TheWindowManager Check in ControlBar::getStarImage

8d325d9

Merge branch 'main' into improve_crc_logging_ea

741a9a3

# Conflicts: # cmake/config.cmake

Readd logging configuration to cmake files

bb744e7

Merge branch 'improve_crc_logging_ea' into headless_sh

bf19c8c

helmutbuhler added 3 commits April 15, 2025 22:25

Merge branch 'improve_crc_logging_ea' into headless_sh

40a9407

Merge branch 'main' of https://github.com/TheSuperHackers/GeneralsGam…

5fba809

…eCode into headless_sh

Merge branch 'main' of https://github.com/TheSuperHackers/GeneralsGam…

73adde5

…eCode into headless_sh

helmutbuhler commented Apr 15, 2025

View reviewed changes

GeneralsMD/Code/GameEngine/Source/Common/RandomValue.cpp Show resolved Hide resolved

helmutbuhler marked this pull request as ready for review April 15, 2025 20:51

Add null checks for W3DDisplay::m_3DScene

e50eec4

helmutbuhler requested a review from xezon April 19, 2025 18:58

xezon force-pushed the main branch from 3d4b5c7 to 8e5bb95 Compare April 19, 2025 20:38

Merge branch 'main' into headless_sh

232da78

helmutbuhler force-pushed the headless_sh branch from 03f4f73 to 232da78 Compare April 19, 2025 23:19

xezon added Enhancement Is new feature or request Major Severity: Minor < Major < Critical < Blocker labels Apr 21, 2025

xezon reviewed Apr 21, 2025

View reviewed changes

helmutbuhler added 12 commits April 24, 2025 10:00

Merge branch 'main' into headless_sh

016187a

# Conflicts: # GeneralsMD/Code/GameEngineDevice/Source/W3DDevice/GameClient/W3DTerrainVisual.cpp

Move m_headless check for W3DView to GameClient

d6c8339

Some minor code moving around

c26971e

Add missing headless check for Sizzle Video

8eb9730

Add missing headless checks for m_bibBuffer in BaseHeightMapRenderObj…

5951bc8

…Class

Add DummyGameWindowManager to avoid null checks

39ae00c

Remove TheWindowManager NULL checks

8759b91

Remove more TheWindowManager NULL checks

cf8963d

Fix stupid dates in comments

d371c76

Remove one more TheWindowManager NULL check

9513645

Fix some white space stuff

1cf0c69

Add comments for DummyGameWindowManager

d8bcd57

xezon reviewed Apr 26, 2025

View reviewed changes

jaapdeheer mentioned this pull request May 2, 2025

[GEN][ZH] Fix limit of 100 in Replay Menu #760

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZH] Implement Headless Mode #651

[ZH] Implement Headless Mode #651

helmutbuhler commented Apr 11, 2025 •

edited

Loading

feliwir Apr 11, 2025

helmutbuhler commented Apr 15, 2025

helmutbuhler commented Apr 19, 2025

xezon left a comment

xezon Apr 21, 2025

helmutbuhler Apr 24, 2025

helmutbuhler commented Apr 21, 2025 •

edited

Loading

xezon left a comment

xezon Apr 26, 2025

xezon Apr 26, 2025

xezon Apr 26, 2025

xezon Apr 26, 2025

xezon Apr 26, 2025

xezon Apr 26, 2025

[ZH] Implement Headless Mode #651

Are you sure you want to change the base?

[ZH] Implement Headless Mode #651

Conversation

helmutbuhler commented Apr 11, 2025 • edited Loading

Choose a reason for hiding this comment

helmutbuhler commented Apr 15, 2025

helmutbuhler commented Apr 19, 2025

xezon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helmutbuhler commented Apr 21, 2025 • edited Loading

xezon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helmutbuhler commented Apr 11, 2025 •

edited

Loading

helmutbuhler commented Apr 21, 2025 •

edited

Loading