None backend - ssh key problems

[spinus]

I can't deploy the box. Nix expression is almost empty. NixOps versions I tried are 1.3.1 and 1.4. Nixpkgs on host machine is unstable, on the virtual machine nixos-version = 16.03.1083.d745044 (Emu).

I can ssh to the box using ssh key.

nixops deploy
jenkins-slave1> generating new SSH keypair... done
Enter passphrase for key '/home/spinus/.ssh/id_rsa': 
Host key fingerprint is SHA256:aS+mlf1IQUQRygUL7il89qISapZc1IP4yaDbgFh+JwE
+--[ED25519 256]--+
|      . .o*o     |
|  E  . o =       |
|  ..o . + .      |
| o.+.+ . o       |
|+o= +.* S .      |
|= o=o+.o + .     |
|.=oo o. * +      |
|o=o  . = o o     |
|o  .. .   . .    |
+----[SHA256]-----+
jenkins-slave1> setting state version to 16.03
jenkins-slave1> waiting for SSH................................................

So, looks like in the first step nixops could connect to the machine and put ssh key there but in the second step stuff it's not working :[

I can ssh to the box using ssh and nixops ssh

[domenkozar]

I saw something similar in the past. I think this is happening due to _ssh_public_key_deployed modified in #364.

I long knew keeping such states would just cause trouble on the long run. You'll have to find out what really happened there and report it so we can fix it.

[spinus]

@domenkozar any idea what's the best solution to debug it? I tried to add --debug but it's not that useful, no debug logging. Best way is to add logging statements to the code or run the code through the debugger or something else?

[domenkozar]

@spinus I think it's best to placeimport pdb;pdb.set_trace() inside that code and see what's going on. Debugging isn't verbose enough.

[spinus]

Thanks. I'll try.

[dezgeg]

Are you by any chance trying to deploy using a hostname that is just an alias in ~/.ssh/config? I found out that won't work as nixops tries to explicitly that targetPort on targetHost is open.

[spinus]

@dezgeg yes, I'm doing that. I have to do that as I'm connecting to this vm through ssh-proxy/ssh-hop.
It's interesting as ssh command is working, so that would mean that each "command" is implementing ssh connection differently.

[dezgeg]

Yes. This is the problematic part:

nixops/nixops/backends/__init__.py

Line 262 in b6e37b7

nixops.util.wait_for_tcp_port(self.get_ssh_name(), self.ssh_port, callback=lambda: self.log_continue("."))

[spinus]

@dezgeg nice, good finding.
@domenkozar, @rbvermaa do you think this would be ok to replace it with equivalent of ssh <host> 'exit 0'?

[spinus]

@domenkozar , @rbvermaa would you mind to take a look?

I switched the check to ssh check rather tcp check. As it uses some internal ssh subsystem logging, the output is slightly different than before, it produces some log lines instead of adding dots (I think this should not be the issue probably).

I added timeout as well, but I may remove it if you wish (it's not essential to that change).