Unstable IDE service in one container

[jankeromnes]

The Cloud9 IDE in a brand new Servo container on OVH1 started misbehaving today (thanks a lot to @birtles for reporting it and helping troubleshoot it!)

The IDE was basically unusable, Editor and Terminal not working properly, and it was displaying errors like:

Failed to write to 'state.settings'. EDISCONNECT: Agent disconnected .Reconnecting...

and

Language server could not be loaded; some language features have been disabled

Opening the DevTools revealed that many requests were getting a 499 error status (on VFS URLs like https://ovh1.janitor.technology/vfs/1/9cbOePWuii5bALJ2?access_token=token).

Inspecting the Cloud9 SDK error log with less /tmp/c9sdk-stderr---supervisor-ejOQPD.log revealed contiuous VFS errors, likely to be the cause of this malfunction:

Error: 'nak' utility failed with exit code 1 and stderr: 'path.js:424
    var path = (i >= 0) ? arguments[i] : process.cwd();
                                                 ^

    var path = (i >= 0) ? arguments[i] : process.cwd();
                                                 ^

Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._resolveLookupPaths (module.js:249:17)
    at Function.Module._resolveFilename (module.js:316:31)
    at Function.Module._load (module.js:276:25)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/user/.c9sdk/node_modules/nak/bin/nak:2:1)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
'
    at EventEmitter.<anonymous> (/home/user/.c9sdk/plugins/c9.vfs.server/filelist.js:99:31)
    at emitTwo (events.js:126:13)
    at EventEmitter.emit (events.js:214:7)
    at Parent.onExit (/home/user/.c9sdk/node_modules/vfs-socket/consumer.js:280:17)
1:8)
    at emitOne (events.js:116:13)
    at Transport.emit (events.js:211:7)
    at /home/user/.c9sdk/node_modules/smith/smith.js:96:14
    at parse (/home/user/.c9sdk/node_modules/smith/smith.js:252:21)
    at Socket.onData (/home/user/.c9sdk/node_modules/smith/smith.js:101:9)
Error: 'nak' utility failed with exit code 1 and stderr: 'path.js:424
    var path = (i >= 0) ? arguments[i] : process.cwd();
                                                 ^
Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._resolveLookupPaths (module.js:249:17)
    at Function.Module._resolveFilename (module.js:316:31)
    at Function.Module._load (module.js:276:25)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/user/.c9sdk/node_modules/nak/bin/nak:2:1)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
'
t.js:99:31)
    at emitTwo (events.js:126:13)
    at EventEmitter.emit (events.js:214:7)
17)
1:8)
    at emitOne (events.js:116:13)
    at Transport.emit (events.js:211:7)
    at /home/user/.c9sdk/node_modules/smith/smith.js:96:14
    at parse (/home/user/.c9sdk/node_modules/smith/smith.js:252:21)
    at Socket.onData (/home/user/.c9sdk/node_modules/smith/smith.js:101:9)
VFS Exception in function 'registerHandlers':
 Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._nodeModulePaths (module.js:190:15)
    at createRequire (jsonalyzer_server:275:33)
    at loadPlugin (jsonalyzer_server:253:19)
    at registerHandler (jsonalyzer_server:131:5)
    at async.forEachSeries.callback.summaries (jsonalyzer_server:107:13)
    at iterate (jsonalyzer_server:328:13)
    at Object.async.forEachSeries (jsonalyzer_server:344:9)
    at Object.registerHandlers (jsonalyzer_server:104:11)
Scheduling process exit
VFS Exception in function 'registerHandlers':
 Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._nodeModulePaths (module.js:190:15)
    at createRequire (jsonalyzer_server:275:33)
    at loadPlugin (jsonalyzer_server:253:19)
    at registerHandler (jsonalyzer_server:131:5)
    at async.forEachSeries.callback.summaries (jsonalyzer_server:107:13)
    at iterate (jsonalyzer_server:328:13)
    at Object.async.forEachSeries (jsonalyzer_server:344:9)
    at Object.registerHandlers (jsonalyzer_server:104:11)
Scheduling process exit
VFS Exception in function 'registerHandlers':
 Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._nodeModulePaths (module.js:190:15)
    at createRequire (jsonalyzer_server:275:33)
    at loadPlugin (jsonalyzer_server:253:19)
    at registerHandler (jsonalyzer_server:131:5)
    at async.forEachSeries.callback.summaries (jsonalyzer_server:107:13)
    at iterate (jsonalyzer_server:328:13)
    at Object.async.forEachSeries (jsonalyzer_server:344:9)
    at Object.registerHandlers (jsonalyzer_server:104:11)
Scheduling process exit
[...]

By searching around, this issue looks similar to npm/npm#10983

We were able to work around this issue by simply creating a new container, which worked fine. Hopefully this was just a temporary bug, associated with that particular Cloud9 SDK version, and we won't see again.

But if you do see this bug again, please comment on this issue.

[jankeromnes]

This seems like a very rare occurrence, so I'll close this bug. Please re-open it if it happens again.

[pjbrunet]

I have the same problem :-( Coincidentally, I'm also using OVH. So maybe OVH is screwing with something. Here are the symptoms...

Same problem... "Failed to write to 'state.settings'"
No ability to save layouts.
Also the terminals stopped working, no bash.

'nak' utility was terminated by signal SIGKILL
at EventEmitter. (/home/c9/c9sdk/plugins/c9.vfs.server/filelist.js:103:31)
at EventEmitter.emit (events.js:98:17)
at Parent.onExit (/home/c9/c9sdk/node_modules/vfs-socket/consumer.js:280:17)
at Parent.Agent._onMessage (/home/c9/c9sdk/node_modules/smith/smith.js:471:8)
at Transport.emit (events.js:95:17)
at /home/c9/c9sdk/node_modules/smith/smith.js:96:14
at parse (/home/c9/c9sdk/node_modules/smith/smith.js:252:21)
at Socket.onData (/home/c9/c9sdk/node_modules/smith/smith.js:101:9)
at Socket.emit (events.js:95:17)
at Socket. (_stream_readable.js:765:14)

I tried to clone the latest version, same problem.

I'm going to assume this is OVH's fault.

[jankeromnes]

Oh no, apologies for the instability. 😐 And thanks a lot for reporting it, I'll re-open this issue.

Failed to write to 'state.settings'"
No ability to save layouts.
Also the terminals stopped working, no bash.

This looks like a connectivity problem to the container. Usually a hard-refresh of the tab solves it (temporary connection problem between front-end and back-end), but maybe sometimes it doesn't, which would mean there is a bug with the IDE itself.

'nak' utility was terminated by signal SIGKILL

Hmm, this doesn't look too dramatic (nak is a search module in Cloud9 IDE, but everything else should work fine without it) but I'm not sure what's killing it.

I suspect this error is related to the Out-Of-Memory errors we've been having lately (when OVH1 uses more than 100% of its 32GB RAM), because in these cases Linux's OOM-killer starts killing processes at random.

Some suggestions:

• Please feel free to share specific container IDs (e.g. 355a586a7006c061 or a full URL like https://ovh1.janitor.technology/355a586a7006c061/8089/ide.html) as we may be able to fix any hard-refresh-surviving problems by restarting the container (or worst case rebooting the OVH1 sever, which causes a brief ~5-10min outage, so not too bad)
• If you see Cloud9 IDE specific problems, please feel free to use our alternative IDE Theia (the URL is the same but ends with /8090/ instead of /8089/ide.html, i.e. https://ovh1.janitor.technology/355a586a7006c061/8090/ with our previous example). We'll soon add links to these URLs in our UI, but Theia IDE should already work fine.

[pjbrunet]

OK guys, after 5 days of pulling out my hair, I found a solution. So maybe OVH on Cloud9 was not the problem.

It turns out, my phone's VPN has a new setting called "killswitch" that was blocking my laptop's VPN. Longer explanation here https://stackoverflow.com/a/50653495/722796

Of course it's possible my laptop's VPN is the solution, but I think more likely, the "killswitch" was somehow mucking up my laptop's Internet connection. Since getting my laptop's VPN running again, everything is running smoother. Like last night I couldn't save a configuration file on my VPS and I tried three different editors with ssh and sshfs, all of them were glitchy, but with my laptop's VPN running properly, I was able to save the file no problem. (Weirdly, I could save a short file last night like "test test test" but could not save a larger file that was just a few pages long!)

So it seems my hotspot works better if it's allowed to pass my laptop's data through without hitting my phone's VPN. Granted, I'm not 100% sure what happened, because I don't know exactly how my phone's hotspot works in conjunction with my phone's VPN, and because I also upgraded my phone and my laptop's OS on the same day, lots of variables. However, that one VPN configuration setting made everything work perfectly again, whereas before everything was working 99% but 1% of everything was breaking.