WIP - New interface for modules to given greater control over connection (L4/TLS/etc) establishment

[phillip-stephens]

Description and Motivation

Some users of ZGrab modules, especially those who are doing scanning of L7 hosts as part of a larger scanning pipeline, desire having more control over the connections ZGrab establishes/uses in order to scan.

This PR intends to modify the interface between protocol modules and the ZGrab framework in order to give users the ability to have full control over ZGrab's connections. This could include:

• setting the source IP/Port for Layer 4 connections on TCP/UDP
• Use a pre-existing connection for scanning a given target
• Upon a successful TLS handshake with a target, to update some telemetry with an outside service of the state of this handshake.

In sum, we want to give the user the ability to take full control of establishing a connection to a scanned host. We also want all existing functionality of ZGrab as a CLI tool to remain unaffected.

Protocol Differences in needs for a Connection

First, let's delineate between 2 classes of protocol.

Transport-Agnostic

The majority of protocols are transport-agnostic and can run on any L4 connection or a TLS-wrapped connection. There may be standard norms (HTTPS on TCP port 443, NTP on port 123 over UDP, etc) but the protocol will work over any if both sides agree.

Non-Transport-Agnostic

Some L7 protocols have certain demands of the underlying transport. For example, HTTPS needs to be able to follow re-directs to HTTP sites and vice-versa. The HTTP module will usually require both a L4 connection and a TLS-wrapped connection to be able to follow both of these types of redirects.

There are other protocols as well, some (SMTP for ex.) want to establish an L4 connection, send a STARTTLS packet, and then proceed with a TLS handshake. We could not just hand this module a pre-existing, post-handshake TLS connection and expect the protocol to work.

Dialer Groups

In order to accomplish this in a clean abstraction, we've settled on the idea of a DialerGroup that is taken as input in every call to Scan(). Modules will provide a "default" dialer group setting the dialers to be as they'd commonly expect. Transport-agnostic modules will set the TransportAgnosticDialer to their default L4/TLS settings, minding any flags the user sets.

For example, NTP would set the dialer to be a UDP dialer connecting to port 123, by default.

From module.go:

type DialerGroup struct {
	// ProtocolAgnosticDialer should be used by most modules that do not need control over the transport layer.
	// It abstracts the underlying transport protocol so a module can  deal with just the L7 logic. Any protocol that
	// doesn't need to know about the underlying transport should use this.
	// If the transport is a TLS connection, the dialer should provide a zgrab2.TLSConnection so the underlying log can be
	// accessed.
	ProtocolAgnosticDialer func(ctx context.Context, target *ScanTarget) (net.Conn, error)
	// L4Dialer will be used by any module that needs to have a TCP/UDP connection. Think of following a redirect to an
	// http:// server, or a module that needs to start with a TCP connection and then upgrade to TLS as part of the protocol.
	// The layered function is needed since we set DialerGroups at Scanner.Init, but modules like HTTP will modify the
	// Dialer based on the target.
	L4Dialer func(target *ScanTarget) func(ctx context.Context, network, addr string) (net.Conn, error)
	// TLSWrapper is a function that takes an existing net.Conn and upgrades it to a TLS connection. This is useful for
	// modules that need to start with a TCP connection and then upgrade to TLS later as part of the protocol.
	// Cannot be used for QUIC connections, as QUIC connections are not "upgraded" from a L4 connection
	TLSWrapper func(ctx context.Context, target *ScanTarget, l4Conn net.Conn) (*TLSConnection, error)

Protocols like HTTP will instead use the L4Dialer and TLSWrapper to specify the two types of connections they require.

Users that require full control over connections can set these dialers to whichever they wish and the Scan() function will utilize them as specified.

The new Scan() function looks as follows:

	// Scan connects to a host. The result should be JSON-serializable
	Scan(ctx context.Context, t *ScanTarget, dialer *DialerGroup) (ScanStatus, any, error)

How it all comes together

The overall flow will look as follows:

1. Framework starts, passing the user's scan flags to the module.
2. During module Init(scanFlags), a scan module will use the user-provided scan flags (entries like what dst port to scan on, TLS flags to control aspects of the TLS connection, etc) to configure the default dialer group the module will usually use. So in the case of NTP, it'll set the TransportAgnosticDialer to a UDP dialer connecting to port 123.
3. Prior to scanning a target, the Framework will call GetDefaultDialerGroup to get a module's dialers for "default" operation.
4. The Framework will then pass these dialers into Scan(ctx context.Context, t *ScanTarget, dialer *DialerGroup).
5. The module will use whatever dialers are passed into Scan() to connect to the target.

This let's users of the modules themselves inject their own dialers into Scan() in Step 5.

In conclusion, this interface let's modules define a certain "default behavior" as each L7 protocol has differing norms on what connections it usually uses, while also giving developers who use these modules in larger scanning systems the ability to fully control all aspects of a connection.

[zakird]

It seems a bit weird that this is specific to a target if it's the default. Can we just call this once at the initiation of ZGrab?

[phillip-stephens]

Well, the DefaultDialer is used for the "base case" when protocols don't care about their transport. And in the case that we want to run the protocol over TLS, we'll need the target.Domain for SNI in the handshake.

What I have currently is the modules provide their DefaultDialer in Init, so only once. This defines their default behavior since only the modules know the value of the specific Flags. The Fox module has a good, simple example: here.

[zakird]

I think this should be more about getting default options from the module not getting a DailerGroup. Semantically, it doesn't make sense to ask a scanner for a Dialer.