[knx] add initial support for KNX secure [WIP]

* add support for KNX IP Secure, new options SECURETUNNEL and SECUREROUTER
* add config options for keyring file and password, and credentials for
  secure connections
* add passive (listening only) access for KNX data secure frames, #8872
* add tests for security functions
* add useCEMI option for newer serial devices like KNX RF sticks,
  kBerry, etc., inspired by #10407
* update user documentation
* handle exceptions during initial read, fixes #7239
* reduce number of warnings

Signed-off-by: Holger Friedrich <mail@holger-friedrich.de>

Author: holgerfriedrich

bundles/org.openhab.binding.knx/README.md

@@ -29,7 +29,7 @@ The IP Gateway is the most commonly used way to connect to the KNX bus. At its b
 
 | Name                | Required     | Description                                                                                                  | Default value                                        |
 |---------------------|--------------|--------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
-| type                | Yes          | The IP connection type for connecting to the KNX bus (`TUNNEL` or `ROUTER`)                                  | -                                                    |
+| type                | Yes          | The IP connection type for connecting to the KNX bus (`TUNNEL`, `ROUTER`, `SECURETUNNEL` or `SECUREROUTER`)                                  | -                                                    |
 | ipAddress           | for `TUNNEL` | Network address of the KNX/IP gateway. If type `ROUTER` is set, the IPv4 Multicast Address can be set.       | for `TUNNEL`: \<nothing\>, for `ROUTER`: 224.0.23.12 |
 | portNumber          | for `TUNNEL` | Port number of the KNX/IP gateway                                                                            | 3671                                                 |
 | localIp             | No           | Network address of the local host to be used to set up the connection to the KNX/IP gateway                  | the system-wide configured primary interface address |
@@ -52,6 +52,7 @@ The *serial* bridge accepts the following configuration parameters:
 | responseTimeout     | N        | Timeout in seconds to wait for a response from the KNX bus                                                   | 10            |
 | readRetriesLimit    | N        | Limits the read retries while initialization from the KNX bus                                                | 3             |
 | autoReconnectPeriod | N        | Seconds between connect retries when KNX link has been lost, 0 means never retry                             | 0             |
+| useCemi             | N        | Use newer CEMI message format, could be useful for newer devices like KNX RF sticks, kBerry, etc.        | false         |
 
 ## Things
 
@@ -199,6 +200,36 @@ Each configuration parameter has a `mainGA` where commands are written to and op
 The `dpt` element is optional. If ommitted, the corresponding default value will be used (see the channel descriptions above).
 
 
+## KNX Secure
+
+> Note: Support for KNX Secure is partly implemented for openHAB and should be considered as experimental.
+
+### KNX IP Secure
+
+KNX IP Secure protects the traffic between openHAB and your KNX installation. It requires either a KNX Secure Router or a Secure IP Interface with security features enabled in ETS tool.
+
+For *Secure routing* mode, the so called `backbone key` needs to be configured in openHAB. It is created by the ETS tool and cannot be changed via the ETS user interface. There are two possible ways to provide the key to openHAB:
+
+- The backbone key can be extracted Security report (ETS, Reports, Security, look for a 32-digit key) and specified in parameter `backboneKey`.
+- The backbone key is included in ETS keyring export (ETS, project settings, export keyring). Keyring file is configured using `keyringFile` (put it in `config\misc` folder of the openHAB installation) and also requires `keyringPasswort`.
+
+For *Secure tunneling* with a Secure IP Interface (or a router in tunneling mode), more parameters are required. A unique device authentication key, and a specific tunnel identifier and password need to be available. It can be provided to openHAB in two different ways:
+
+- All information can be looked up in ETS and provided separately: `tunnelDevAuth`, `tunnelPasswort`. `tunnelUser` is a number which is not directly visible in ETS, but can be looked up in keyring export or deduced (typically 2 for the first tunnel of a device, 3 for the second one, ...)
+- All necessary information is included in ETS keyring export (ETS, project settings, export keyring). Keyring file is configured using `keyringFile` (put it in `config\misc` folder of the openHAB installation) and `keyringPasswort`. In addition, `tunnelSourceAddr` needs to be set to uniquely identify the tunnel in use.
+
+
+### KNX Data Secure
+
+Data secure protects the content of messages on the KNX bus. In a KNX installation, both classic and secure group addresses can coexist.
+
+openHAB typically ignores messages with secure group addresses, unless data secure is configured.
+
+> NOTE: openHAB currently does fully support passive (listening) access to secure group addresses. Write access to secured GAs is done via the "GO diagnostics" feature described in KNX AN170 and is currently limited. Expect a timeout if a data value is written too often. Initial/periodic read will fail, avoid automatic read (< in thing definition).
+
+All necessary information to decode secure group addresses is included in ETS keyring export (ETS, project settings, export keyring). Keyring file is configured using `keyringFile` (put it in `config\misc` folder of the openHAB installation) and also requires `keyringPasswort`.
+
+
 ## Examples
 
 The following two templates are sufficient for almost all purposes.

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/KNXBindingConstants.java

@@ -18,6 +18,7 @@
 import java.util.Set;
 import java.util.stream.Stream;
 
+import org.eclipse.jdt.annotation.NonNullByDefault;
 import org.openhab.core.thing.ThingTypeUID;
 
 /**
@@ -26,6 +27,7 @@
  *
  * @author Karel Goderis - Initial contribution
  */
+@NonNullByDefault
 public class KNXBindingConstants {
 
     public static final String BINDING_ID = "knx";
@@ -51,6 +53,14 @@ public class KNXBindingConstants {
     public static final String LOCAL_SOURCE_ADDRESS = "localSourceAddr";
     public static final String PORT_NUMBER = "portNumber";
     public static final String SERIAL_PORT = "serialPort";
+    public static final String USE_CEMI = "useCemi";
+    public static final String KEYRING_FILE = "keyringFile";
+    public static final String KEYRING_PASSWORD = "keyringPassword";
+    public static final String ROUTER_BACKBONE_GROUP_KEY = "routerBackboneGroupKey";
+    public static final String TUNNEL_USER_ID = "tunnelUserId";
+    public static final String TUNNEL_USER_PASSWORD = "tunnelUserPassword";
+    public static final String TUNNEL_DEVICE_AUTHENTICATION = "tunnelDeviceAuthentication";
+    public static final String TUNNEL_SOURCE_ADDRESS = "tunnelSourceAddress";
 
     // The default multicast ip address (see <a
     // href="http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xml">iana</a> EIBnet/IP

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/KNXTypeMapper.java

@@ -17,7 +17,6 @@
 import org.openhab.core.types.Type;
 
 import tuwien.auto.calimero.datapoint.Datapoint;
-import tuwien.auto.calimero.process.ProcessEvent;
 
 /**
  * This interface must be implemented by classes that provide a type mapping
@@ -45,7 +44,8 @@ public interface KNXTypeMapper {
      * maps a datapoint value to an openHAB command or state
      *
      * @param datapoint the source datapoint
-     * @param data the datapoint value as an ASDU byte array (see <code>{@link ProcessEvent}.getASDU()</code>)
+     * @param data the datapoint value as an ASDU byte array (see
+     *            <code>{@link tuwien.auto.calimero.process.ProcessEvent}.getASDU()</code>)
      * @return a command or state of openHAB
      */
     @Nullable

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/channel/AbstractSpec.java

@@ -14,8 +14,6 @@
 
 import org.eclipse.jdt.annotation.NonNullByDefault;
 import org.eclipse.jdt.annotation.Nullable;
-import org.openhab.binding.knx.internal.client.InboundSpec;
-import org.openhab.binding.knx.internal.client.OutboundSpec;
 
 import tuwien.auto.calimero.GroupAddress;
 import tuwien.auto.calimero.KNXFormatException;
@@ -57,7 +55,8 @@ protected final GroupAddress toGroupAddress(GroupAddressConfiguration ga) {
     /**
      * Return the data point type.
      * <p>
-     * See {@link InboundSpec#getDPT()} and {@link OutboundSpec#getDPT()}.
+     * See {@link org.openhab.binding.knx.internal.client.InboundSpec#getDPT()} and
+     * {@link org.openhab.binding.knx.internal.client.OutboundSpec#getDPT()}.
      *
      * @return the data point type.
      */

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/channel/ListenSpecImpl.java

@@ -17,6 +17,7 @@
 import java.util.Collections;
 import java.util.List;
 
+import org.eclipse.jdt.annotation.NonNullByDefault;
 import org.eclipse.jdt.annotation.Nullable;
 import org.openhab.binding.knx.internal.client.InboundSpec;
 
@@ -28,6 +29,7 @@
  * @author Simon Kaufmann - initial contribution and API.
  *
  */
+@NonNullByDefault
 public class ListenSpecImpl extends AbstractSpec implements InboundSpec {
 
     private final List<GroupAddress> listenAddresses;

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/channel/TypeColor.java

@@ -40,7 +40,8 @@ class TypeColor extends KNXChannelType {
 
     @Override
     protected Set<String> getAllGAKeys() {
-        return Stream.of(SWITCH_GA, POSITION_GA, INCREASE_DECREASE_GA, HSB_GA).collect(toSet());
+        final var tmp = Stream.of(SWITCH_GA, POSITION_GA, INCREASE_DECREASE_GA, HSB_GA).collect(toSet());
+        return tmp;
     }
 
     @Override

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/channel/TypeDimmer.java

@@ -40,7 +40,8 @@ class TypeDimmer extends KNXChannelType {
 
     @Override
     protected Set<String> getAllGAKeys() {
-        return Stream.of(SWITCH_GA, POSITION_GA, INCREASE_DECREASE_GA).collect(toSet());
+        final var tmp = Stream.of(SWITCH_GA, POSITION_GA, INCREASE_DECREASE_GA).collect(toSet());
+        return tmp;
     }
 
     @Override

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/channel/TypeRollershutter.java

@@ -53,6 +53,7 @@ protected String getDefaultDPT(String gaConfigKey) {
 
     @Override
     protected Set<String> getAllGAKeys() {
-        return Stream.of(UP_DOWN_GA, STOP_MOVE_GA, POSITION_GA).collect(toSet());
+        final var tmp = Stream.of(UP_DOWN_GA, STOP_MOVE_GA, POSITION_GA).collect(toSet());
+        return tmp;
     }
 }

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/client/AbstractKNXClient.java

@@ -39,6 +39,7 @@
 import tuwien.auto.calimero.GroupAddress;
 import tuwien.auto.calimero.IndividualAddress;
 import tuwien.auto.calimero.KNXException;
+import tuwien.auto.calimero.KnxRuntimeException;
 import tuwien.auto.calimero.datapoint.CommandDP;
 import tuwien.auto.calimero.datapoint.Datapoint;
 import tuwien.auto.calimero.device.ProcessCommunicationResponder;
@@ -54,6 +55,7 @@
 import tuwien.auto.calimero.process.ProcessCommunicatorImpl;
 import tuwien.auto.calimero.process.ProcessEvent;
 import tuwien.auto.calimero.process.ProcessListener;
+import tuwien.auto.calimero.secure.KnxSecureException;
 import tuwien.auto.calimero.secure.SecureApplicationLayer;
 import tuwien.auto.calimero.secure.Security;
 
@@ -78,6 +80,7 @@ public abstract class AbstractKNXClient implements NetworkLinkListener, KNXClien
     private final int readRetriesLimit;
     private final StatusUpdateCallback statusUpdateCallback;
     private final ScheduledExecutorService knxScheduler;
+    protected final Security openhabSecurity;
 
     private @Nullable ProcessCommunicator processCommunicator;
     private @Nullable ProcessCommunicationResponder responseCommunicator;
@@ -91,6 +94,9 @@ public abstract class AbstractKNXClient implements NetworkLinkListener, KNXClien
     private final Set<GroupAddressListener> groupAddressListeners = new CopyOnWriteArraySet<>();
     private final LinkedBlockingQueue<ReadDatapoint> readDatapoints = new LinkedBlockingQueue<>();
 
+    private boolean firstConnect;
+    private long lastDisconnectSysMillis;
+
     @FunctionalInterface
     private interface ListenerNotification {
         void apply(BusMessageListener listener, IndividualAddress source, GroupAddress destination, byte[] asdu);
@@ -101,6 +107,7 @@ private interface ListenerNotification {
 
         @Override
         public void detached(DetachEvent e) {
+            lastDisconnectSysMillis = System.currentTimeMillis();
             logger.debug("The KNX network link was detached from the process communicator");
         }
 
@@ -127,14 +134,18 @@ public void groupReadResponse(ProcessEvent e) {
     };
 
     public AbstractKNXClient(int autoReconnectPeriod, ThingUID thingUID, int responseTimeout, int readingPause,
-            int readRetriesLimit, ScheduledExecutorService knxScheduler, StatusUpdateCallback statusUpdateCallback) {
+            int readRetriesLimit, ScheduledExecutorService knxScheduler, StatusUpdateCallback statusUpdateCallback,
+            Security openhabSecurity) {
         this.autoReconnectPeriod = autoReconnectPeriod;
         this.thingUID = thingUID;
         this.responseTimeout = responseTimeout;
         this.readingPause = readingPause;
         this.readRetriesLimit = readRetriesLimit;
         this.knxScheduler = knxScheduler;
         this.statusUpdateCallback = statusUpdateCallback;
+        this.openhabSecurity = openhabSecurity;
+        firstConnect = true;
+        lastDisconnectSysMillis = System.currentTimeMillis();
     }
 
     public void initialize() {
@@ -145,7 +156,9 @@ public void initialize() {
 
     private boolean scheduleReconnectJob() {
         if (autoReconnectPeriod > 0) {
-            connectJob = knxScheduler.schedule(this::connect, autoReconnectPeriod, TimeUnit.SECONDS);
+            // schedule connect job, for the first connection ignore autoReconnectPeriod and use 1 sec
+            connectJob = knxScheduler.schedule(this::connect, firstConnect ? 1 : autoReconnectPeriod, TimeUnit.SECONDS);
+            firstConnect = false;
             return true;
         } else {
             return false;
@@ -173,6 +186,12 @@ private synchronized boolean connect() {
         if (isConnected()) {
             return true;
         }
+        long now = System.currentTimeMillis();
+        if ((now - lastDisconnectSysMillis) < 1000) {
+            logger.debug("fast reconnect");
+            // SECURETUNNEL: TCP Connections need additional efforts to avoid grabbing all tunnel connections
+            // the interface, this is implemented in IPClient
+        }
         try {
             releaseConnection();
 
@@ -195,7 +214,7 @@ private synchronized boolean connect() {
             this.processCommunicator = processCommunicator;
 
             ProcessCommunicationResponder responseCommunicator = new ProcessCommunicationResponder(link,
-                    new SecureApplicationLayer(link, Security.defaultInstallation()));
+                    new SecureApplicationLayer(link, openhabSecurity));
             this.responseCommunicator = responseCommunicator;
 
             link.addLinkListener(this);
@@ -206,7 +225,7 @@ private synchronized boolean connect() {
             statusUpdateCallback.updateStatus(ThingStatus.ONLINE);
             connectJob = null;
             return true;
-        } catch (KNXException | InterruptedException e) {
+        } catch (KNXException | InterruptedException | KnxSecureException e) {
             logger.debug("Error connecting to the bus: {}", e.getMessage(), e);
             disconnect(e);
             scheduleReconnectJob();
@@ -215,18 +234,19 @@ private synchronized boolean connect() {
     }
 
     private void disconnect(@Nullable Exception e) {
+        lastDisconnectSysMillis = System.currentTimeMillis();
+
         releaseConnection();
         if (e != null) {
-            String message = e.getLocalizedMessage();
             statusUpdateCallback.updateStatus(ThingStatus.OFFLINE, ThingStatusDetail.COMMUNICATION_ERROR,
-                    message != null ? message : "");
+                    "" + e.getLocalizedMessage());
         } else {
             statusUpdateCallback.updateStatus(ThingStatus.OFFLINE);
         }
     }
 
     @SuppressWarnings("null")
-    private void releaseConnection() {
+    protected void releaseConnection() {
         logger.debug("Bridge {} is disconnecting from the KNX bus", thingUID);
         readDatapoints.clear();
         busJob = nullify(busJob, j -> j.cancel(true));
@@ -244,7 +264,7 @@ private void releaseConnection() {
         });
     }
 
-    private <T> T nullify(T target, @Nullable Consumer<T> lastWill) {
+    private <T> @Nullable T nullify(T target, @Nullable Consumer<T> lastWill) {
         if (target != null && lastWill != null) {
             lastWill.accept(target);
         }
@@ -275,6 +295,7 @@ private String toDPTValue(Type type, String dpt) {
         return typeHelper.toDPTValue(type, dpt);
     }
 
+    // datapoint is null at end of the list, warning is misleading
     @SuppressWarnings("null")
     private void readNextQueuedDatapoint() {
         if (!connectIfNotAutomatic()) {
@@ -285,7 +306,15 @@ private void readNextQueuedDatapoint() {
             return;
         }
         ReadDatapoint datapoint = readDatapoints.poll();
+
         if (datapoint != null) {
+            // TODO #8872: allow write access, currently only listening mode
+            if (openhabSecurity.groupKeys().containsKey(datapoint.getDatapoint().getMainAddress())) {
+                logger.debug("outgoing secure communication not implemented, explicit read from GA '{}' skipped",
+                        datapoint.getDatapoint().getMainAddress());
+                return;
+            }
+
             datapoint.incrementRetries();
             try {
                 logger.trace("Sending a Group Read Request telegram for {}", datapoint.getDatapoint().getMainAddress());
@@ -299,6 +328,11 @@ private void readNextQueuedDatapoint() {
                     logger.warn("Giving up reading datapoint {}, the number of maximum retries ({}) is reached.",
                             datapoint.getDatapoint().getMainAddress(), datapoint.getLimit());
                 }
+            } catch (KnxRuntimeException e) {
+                // KnxRuntimeException is _not_ a subclass of KnxException. Fail gracefully for this case as well,
+                // fixes #7239
+                logger.warn("Error reading datapoint {}: {}", datapoint.getDatapoint().getMainAddress(),
+                        e.getMessage());
             } catch (InterruptedException e) {
                 logger.debug("Interrupted sending KNX read request");
                 return;
@@ -392,7 +426,8 @@ public final boolean unregisterGroupAddressListener(GroupAddressListener listene
 
     @Override
     public boolean isConnected() {
-        return link != null && link.isOpen();
+        final var tmpLink = link;
+        return tmpLink != null && tmpLink.isOpen();
     }
 
     @Override
@@ -449,6 +484,12 @@ private void sendToKNX(ProcessCommunication communicator, KNXNetworkLink link, G
             return;
         }
 
+        // TODO #8872: allow write access, currently only listening mode
+        if (openhabSecurity.groupKeys().containsKey(groupAddress)) {
+            logger.debug("outgoing secure communication not implemented, write to GA '{}' skipped", groupAddress);
+            return;
+        }
+
         Datapoint datapoint = new CommandDP(groupAddress, thingUID.toString(), 0, dpt);
         String mappedValue = toDPTValue(type, dpt);
 

bundles/org.openhab.binding.knx/src/main/java/org/openhab/binding/knx/internal/client/CustomKNXNetworkLinkIP.java

@@ -12,6 +12,8 @@
  */
 package org.openhab.binding.knx.internal.client;
 
+import org.eclipse.jdt.annotation.NonNullByDefault;
+
 import tuwien.auto.calimero.KNXException;
 import tuwien.auto.calimero.knxnetip.KNXnetIPConnection;
 import tuwien.auto.calimero.link.KNXNetworkLinkIP;
@@ -24,9 +26,11 @@
  * @author Simon Kaufmann - initial contribution and API
  *
  */
+@NonNullByDefault
 public class CustomKNXNetworkLinkIP extends KNXNetworkLinkIP {
 
     public static final int TUNNELING = KNXNetworkLinkIP.TUNNELING;
+    public static final int TUNNELINGV2 = KNXNetworkLinkIP.TunnelingV2;
     public static final int ROUTING = KNXNetworkLinkIP.ROUTING;
 
     CustomKNXNetworkLinkIP(final int serviceMode, KNXnetIPConnection conn, KNXMediumSettings settings)