Difference between two tensors using the same device

[ShayLevy]

I’m working on the fundamentals exercise. I'm trying to understand why tensor a and b yield different results when the device assignment is as follows(assuming device=cuda):

torch.manual_seed(1234)
a = torch.rand(size=(2,3)).to(device)
b = torch.rand(size=(2,3), device=device)

Thanks!

[Haji-Suleman]

I was working on the same problem and observed the same thing on Colab.
The problem is that torch.manual_seed() seeds only the CPU random number generator, while CUDA has a random number generator of its own.

torch.manual_seed(1234)
torch.cuda.manual_seed_all(1234)

This will definitely Solve your problem

[ShayLevy]

Thanks Haji.

Apparently not. I get False for all three attempts

Initially I tried testing with torch.cuda.manual_seed, same result.
I didn't know torch.cuda.manual_seed_all, after reading about it I see that it is used for multi-GPU setups, colab has only one gpu (in my case).

device="cuda"

torch.manual_seed(1234)
a = torch.rand(size=(2,3)).to(device)
b = torch.rand(size=(2,3), device=device)
print(a==b)

tensor([[False, False, False],
[False, False, False]], device='cuda:0')

torch.cuda.manual_seed(1234)
a = torch.rand(size=(2,3)).to(device)
b = torch.rand(size=(2,3), device=device)
print(a==b)

tensor([[False, False, False],
[False, False, False]], device='cuda:0')

torch.cuda.manual_seed_all(1234)
a = torch.rand(size=(2,3)).to(device)
b = torch.rand(size=(2,3), device=device)
print(a==b)

tensor([[False, False, False],
[False, False, False]], device='cuda:0')

[Haji-Suleman]

Bro your code is not correct

Generating on cpu then moving to a gpu not equals Generating on gpu
code :
torch.manual_seed(1234)

a = torch.rand((2,3)).to(device)
b = torch.rand((2,3)).to(device) # Also CPU → GPU

print(a == b) # Now Trueeeee

Now this will work

[Haji-Suleman]

I know you must understand the concept if there is any other issue you can tell me

[ShayLevy]

What am I missing out?

device="cuda"
torch.manual_seed(1234)
print(f"cuda available: {torch.cuda.is_available()}")

a = torch.rand((2,3)).to(device)
b = torch.rand((2,3)).to(device) # Also CPU → GPU

print(a == b) 

cuda available: True
tensor([[False, False, False],
[False, False, False]], device='cuda:0')

[ohecker-logicalredstone]

You need to redefine the manual seed when generating both tensors. When u use torch.manual_seed(), It only works 1 time like:

torch.manual_seed(42)
A = torch.rand(size=(2, 3) # Independent against CPU or GPU, also, it is assigned that seed)`
B = torch.rand(size=(2, 3) # Now the torch.manual_seed(...) wont work because it is already used 1 time

It must be:

torch.manual_seed(42)
A = torch.rand(size=(2, 3) # Independent against CPU or GPU)
torch.manual_seed(42)
B = torch.rand(size=(2, 3) #Now it is redefined, so it will generate that seed agian

[ShayLevy]

My initial question was about why a == b is False. Adding another manual seed after the a tensor doesn't change the result

device="cuda"
torch.manual_seed(1234)
a = torch.rand(size=(2,3)).to(device)
b = torch.rand(size=(2,3), device=device)

[ohecker-logicalredstone]

Can you give me some sample code of yours? (FYI:

device="cuda"
torch.manual_seed(1234)
a = torch.rand(size=(2,3)).to(device)
b = torch.rand(size=(2,3), device=device)

is wrong)

[ShayLevy]

This is actually the code I was writing when I worked on the 00_pytorch_fundamentals exercise.

I wrote torch.rand(size=(2,3), device=device) and something didn't work and looked at Daniel's code and saw that I should have wrote torch.rand(size=(2,3)).to(device)

So i tried to find the difference between the two cause the result was different

[Haji-Suleman]

Try This one
NOTE: CPU and GPU use different random generators.
a was created on CPU first, then moved to GPU.
b was created directly on GPU.
Same seed ≠ same result if tensors are created on different devices.
Mistake:
You thought one torch.manual_seed() controls both CPU and GPU randomness.
Why your fix works:
You set the seed again before creating the tensor on the same device, so both tensors match.

CODE:
import torch
device = "cuda"
torch.manual_seed(1234)
a = torch.rand(size=(2, 3)).to(device)
torch.manual_seed(1234)
b = torch.rand(size=(2, 3)).to(device)
a==b

OUTPUT: tensor([[True, True, True],
[True, True, True]], device='cuda:0')

[ShayLevy]

Got it, thanks!

[Haji-Suleman]

Fine

[mrdbourke]

Thank you @Haji-Suleman and @ohecker-logicalredstone for helping @ShayLevy !