Authentication

[Kixunil]

One of the use cases of this crate suggests use in production to debug possible problems. That's very understandable but I see it as a potential security vulnerability. If the port is exposed an attacker (e.g. a compromised user on the same system) may be able to obtain valuable information.

The obvious solution is to implement a basic authentication mechanism. A simple challenge-response with a secret token and HMAC would go long way to achieve security. Another possibility is to allow binding Unix sockets and just rely on filesystem permissions (but this AFAIK only works on Linux).

[hawkw]

I agree that authentication is important if enabling the console in production.

However, I think it seems out of scope for the console-subscriber crate itself to implement the authentication mechanism(s). Users may want to use any number of different authentication and access management systems depending on their deployment model, what their organization is using to authenticate access for other systems, personal preferences, and so on.

Instead of actually implementing our own authentication mechanism, I think we should make sure that it's possible for users to easily add the authentication mechanisms of their choice to the console-subscriber gRPC server. We should also have examples and other documentation explaining how to add your preferred authentication mechanism to the gRPC server.

The gRPC server is implemented using the tonic crate. tonic supports adding authentication to servers via gRPC interceptors or using tower middleware. The console-subscriber::Server type implements the tonic generated trait for the Instrument gRPC service, so it should be possible to add authentication to the server using the mechanisms tonic supports. We should add examples and other documentation explaining this.

The client-side part of authentication is more difficult, though. The client is typically the tokio-console command-line application, so users aren't typically connecting clients from code they write --- it's just a binary, so you can't easily layer in your own auth mechanism. So, I think it will be necessary to add support for various authentication mechanisms to the tokio-console binary, perhaps controllable via CLI flags.

An alternative is to not implement auth directly in the console client or server, and instead suggest production users rely on external authentication mechanisms. For example, by default, the console-subscriber server listens on localhost, rather than 0.0.0.0. This means it's not opened up to the entire internet. Instead, in order to connect to a remote console-subscriber server that's only listening on localhost, users would need to port forward the connection to the host the console-enabled application is running on. This might be done using SSH port forwarding, kubectl port-forward if running in Kubernetes, or similar mechanisms. In that case, whatever existing authentication mechanisms control access to the host (e.g. SSH authorized_keys in the SSH case, Kubernetes RBAC in the Kubernetes case, etc) would also control access to the console.

[Kixunil]

Yeah, agree with making it pluggable. I think the console situation can be resolved by having some basic good default. Having something good handy (maybe as a separate crate) so that users don't have to think about it too much (or worse, screw up rolling their own) would be nice anyway.

People who want to use something advanced can just build their own by implementing appropriate traits.

Notes:

• I found tonic interceptors slightly hard to understand - good docs would be helpful
• binding on localhost is not enough in multi-user environment - there are possible attacks abusing race conditions around binding ports. Unix socket fixes this on Linux.
• Basic security should probably be turned on by default (console_subscriber::init()) to not end up with log4shell-style situation.

[hawkw]

• Basic security should probably be turned on by default (console_subscriber::init()) to not end up with log4shell-style situation.

I agree that we should offer some security by default when using the simple entry-point API. The question is just, what auth mechanism can we use in this case that works out of the box without requiring complicated configuration? It's also important for the default init function to work out of the box for local debugging without requiring the user to perform steps like generating a token and so on. Ideally, there would be something that should be configured on most systems that we could use...perhaps we could do a SSH handshake based on what's already in the .ssh directories on the host and client systems? That would also give us basic authorization --- the server could only allow clients whose public keys are already in the system's authorized_keysfile.

Another option would be to have an explicit init_unauthenticated() or something, for the local debugging use-case. If we went down that route, I think it would be more okay to have the init function not work out of the box if authentication hasn't been configured in some way.

[hawkw]

Honestly, I'm not totally sure how important having a default authentication mechanism is, as I imagine most users deploying console-enabled software in prod won't use whatever scheme we provide by default, and will instead prefer to integrate with their existing auth systems. Maybe what we need to do is just rename init to init_unauthenticated or something similarly scary sounding, so that it becomes clear that it's only for local debugging and shouldn't be enabled in production, and then just let production users bring their own auth mechanism.

[hawkw]

There's still the question, though, of what authentication mechanisms the console CLI needs to support...and since we have to actually support them in the CLI, we may as well have nice APIs for building a server with the supported authentication mechanisms enabled. Then, the user can just plug in code for whatever authorization policy they want based on the selected authentication mechanism...

[Kixunil]

IMO the easiest and sufficiently secure way is to just generate a random 128-bit secret and store it in some predictable location (e.g. dirs_next::data_dir().unwrap().join("tokio-console/secret")). Then just use HMAC challenge-response.

I imagine most users deploying console-enabled software in prod won't use whatever scheme we provide by default

I don't see any problem using the simple token scheme, maybe I'd adjust the location of the secret file and prefer unix socket.
The best way of figuring out which authentication mechanisms people want to use is ask them. :)

[hawkw]

Yeah, I suppose we can always do a simple token scheme and allow layering in other authz/authn systems that the user might require on top of that.

[hawkw]

We probably also want to have an example showing how to serve the console endpoint over TLS. That ought to be opt-in because the user would need to actually provide TLS key material for the server.

[BatmanAoD]

There's still the question, though, of what authentication mechanisms the console CLI needs to support...and since we have to actually support them in the CLI, we may as well have nice APIs for building a server with the supported authentication mechanisms enabled.

Perhaps #227 would make baked-in authentication for the tokio-console binary unnecessary, as you could just require users deploying in production to compile their own (T)UI and layer in whatever auth they need that way?

[BatmanAoD]

We probably also want to have an example showing how to serve the console endpoint over TLS. That ought to be opt-in because the user would need to actually provide TLS key material for the server.

The tokio-console help string doesn't seem to mention TLS; is there a way to configure that on the client side, currently?

[hds]

@BatmanAoD By specifying an https endpoint you should be able to connect with TLS.

However that will require that the underlying hyper stuff trusts the console-subscriber server. Normally this can be configured via your system trust store, but I'm not sure about your use case.

[BatmanAoD]

@hds Sorry, should have tried that first! I'm used to tools not always respecting the URL scheme. Thanks!

[Kixunil]

Looks like on server side this still needs some kind of proxy to add TLS. Unfortunately it's racy unless network namespaces are used. I think just making it generic over transport stream would resolve this - one can use TLS directly, Unix socket or whatever else.

[hds]

I think you can do that with the server parts here can't you?

https://docs.rs/console-subscriber/latest/console_subscriber/struct.Server.html#method.into_parts