Specifying -nvidia 0 results in GPU 1 being selected -nvidia 1 selects GPU 0

[papabear99]

Describe the bug
Specifying -nvidia 0 results in GPU 1 being selected -nvidia 1 selects GPU 0

To Reproduce
Steps to reproduce the behavior:

1. Add -nvidia 0 to start command
2. Start Livepeer
3. View console output and observe "Transcoding on these Nvidia GPUs: [1]" Also confirmed in use that the incorrect GPU is selected and used for transcoding.
4. This behavior also happens in reverse. -nvidia 1 selects GPU 0.

Expected behavior
The GPU specified by the -nvidia flag should choose the GPU specified.

Desktop (please complete the following information):

• OS: WIndows 10 and Windows Server 2019
• Version Livepeer 0.5.24 - 0.5.27 (probably other versions as well but I have tried it on these versions)

Additional context
The behavior is present using both combined O+T and stand alone T configurations
I have not tried this on machines with more than 2 GPUs so I don't know what happens if -nvidia 2 is specified

[RyanC92]

I did notice this and I thought it was my system doing that. glad I'm not the only one.

[thomshutt]

I haven't dug into which GPU actually gets used yet, but I haven't been able to reproduce Step [3] of the reported behaviour:

$ livepeer -transcoder -orchSecret=123 -orchAddr=123 -nvidia 0
I0303 11:32:54.434922   47524 livepeer.go:274] ***Livepeer is running on the offchain network***
I0303 11:32:54.436043   47524 livepeer.go:319] Transcoding on these Nvidia GPUs: [0]

$ livepeer -transcoder -orchSecret=123 -orchAddr=123 -nvidia 1
I0303 11:34:06.017142   47564 livepeer.go:274] ***Livepeer is running on the offchain network***
I0303 11:34:06.020906   47564 livepeer.go:319] Transcoding on these Nvidia GPUs: [1]

and looking at the code, this part more or less just parses the command line arguments and logs them, so I'm not sure how the log line could end up being wrong:

go-livepeer/cmd/livepeer/livepeer.go

Lines 315 to 319 in bd5e342

	devices, err := common.ParseNvidiaDevices(*nvidia)
	if err != nil {
	glog.Fatalf("Error while parsing '-nvidia %v' flag: %v", *nvidia, err)
	}
	glog.Infof("Transcoding on these Nvidia GPUs: %v", devices)

go-livepeer/common/util.go

Lines 444 to 449 in bd5e342

	func ParseNvidiaDevices(nvidia string) ([]string, error) {
	if nvidia == "all" {
	return detectNvidiaDevices()
	}
	return strings.Split(nvidia, ","), nil
	}

I'll carry on checking whether the GPU selected is/isn't the one that actually gets used, but could you double check what you're seeing logged out please @papabear99? If you're still able to reproduce then a full log of the startup might be useful

[papabear99]

Yes the issue is still present. If GPU 0 is specified in the start command as-nividia 0the log (as yours above) shows GPU 0 is selected but when I look at GPU activity using nvidia-smi or the task manager I can see that GPU 1 is actually the GPU that is being used. It works the other way if -nividia 1 is specified.

On a single GPU node -nvidia all shows and uses GPU 0

Note: I don't if this is Windows specific as I don't have any Transcoders running on Linux

@Nightwolf92 Does it happen for you on Linux?

[thomshutt]

Thanks @papabear99, if you're seeing the correct GPU in the log line now then that's good - I'll carry on with investigating why that differs from the GPU that actually gets used

[RyanC92]

@papabear99 @thomshutt
I am only seeing this on Windows, it seems to be fine on Linux.

my config

transcoder
orchSecret <secret>
reward false
maxSessions 20
orchAddr <ip>
cliAddr 127.0.0.1:<port>
nvidia 1 #As you can see nvidia 1 is selected
monitor 
v 6

GPU 0 is the 1070, this is what is being utilized in the config file. GPU 1 is my 3080 TI. However you can see that although I put nvidia 1 as the option in the config, its utilizing GPU 0. if I were to put nvidia 0 it would use GPU 1

[thomshutt]

I've spun up a Windows Server 2022 machine in Google Cloud, with 4 Nvidia Tesla T4s attached.

I ran the following commands to stand up a local Broadcaster / Orchestrator / Transcoder environment:

Broadcaster

./livepeer -datadir ~/temp/livepeer/TEST_B -broadcaster -cliAddr :7937 -httpAddr :8936 -orchAddr 127.0.0.1:8935 -v 6

Orchestrator

./livepeer -datadir ~/temp/livepeer/TEST -orchestrator -orchSecret foo -serviceAddr 127.0.0.1:8935 -v 6

Transcoder

./livepeer -datadir ~/temp/livepeer/TEST_T1 -transcoder -orchSecret foo -orchAddr 127.0.0.1:8935 -cliAddr :7936 -nvidia 0 -v 6

I see the expected log line on startup

I0308 15:53:50.331431    1468 livepeer.go:319] Transcoding on these Nvidia GPUs: [0]

and the correct GPU being used when I send a job in to be transcoded (also works correctly when specifying the other GPUs):

[RyanC92]

I've spun up a Windows Server 2022 machine in Google Cloud, with 4 Nvidia Tesla T4s attached.

I ran the following commands to stand up a local Broadcaster / Orchestrator / Transcoder environment:

Broadcaster

./livepeer -datadir ~/temp/livepeer/TEST_B -broadcaster -cliAddr :7937 -httpAddr :8936 -orchAddr 127.0.0.1:8935 -v 6

Orchestrator

./livepeer -datadir ~/temp/livepeer/TEST -orchestrator -orchSecret foo -serviceAddr 127.0.0.1:8935 -v 6

Transcoder

./livepeer -datadir ~/temp/livepeer/TEST_T1 -transcoder -orchSecret foo -orchAddr 127.0.0.1:8935 -cliAddr :7936 -nvidia 0 -v 6

I see the expected log line on startup

I0308 15:53:50.331431    1468 livepeer.go:319] Transcoding on these Nvidia GPUs: [0]

and the correct GPU being used when I send a job in to be transcoded (also works correctly when specifying the other GPUs):

Hm. not sure why yours is proper whereas ours isnt.. See below.