Seq Kubernetes Operator

[wasabii]

Would anybody else be interested in this? I've been having a terrible time trying to get the Seq helm chart deployed in K8s, with proper configuration, etc. Like, getting it installed isn't a problem. But then figuring out how to run something to add users, change the default password, and then apply configuration, has been a pretty terrible.

[mloskot]

Would anybody else be interested in this?

Yes. I've been thinking about it myself for a while.

In fact, I have developed an little operator, written in Python based on https://github.com/flant/shell-operator, for managing Seq API keys. My goal was to synchronise Seq API keys from Kubernetes secrets. This allowed me to manage Seq API keys with an external key vault, next automatically import them into Kubernetes Secrets using https://external-secrets.io, then import/update/delete into/in Seq from Kubernetes secrets. I also introduced custom annotations for my Kubernetes secrets with Seq API keys like seq-apikey-title and seq-apikey-properties which values are used to set the title and properties on the API keys created in Seq e.g. with application specific properties which allow to filter logs, etc.
I used the shell-operator, because it's PoC. I cannot share code of that operator, but if I'm allowed to publish it in future, I will share it here.

This experience made me think about a full-fledged operator for managing the lifecycle of Seq, like the Flux Operator or others.

I've been having a terrible time trying to get the Seq helm chart deployed in K8s (...) add users, change the default password, and then apply configuration, has been a pretty terrible.

I fully understand your point. The currently available Seq stack is not quite Kubernetes-native and if one aims for automation to configure Seq, it does take some gymnastics to manage Seq deployment. That, and an existing context, made me decide to do it with Terraform:

1. Generate Seq admin password and store it in Azure Key Vault

2. Create ExternalSecret using kubectl_manifest to import the Seq admin password into Kubernetes

3. Create Certificate using kubectl_manifest too (this assumes pre-existing deployment of https://cert-manager.io, obviously.

4. Create Kubernetes ConfigMap using kubernetes_config_map_v1 with Bash script init-1-firstrun-adminpassword.sh generated from a template in order to be used as Seq Init Scripts. Now, THIS generated script is the MOST important part of the whole idea, that allows some automation. Here it is:

   #!/bin/bash
   # Seq init script generates firstRun.adminPasswordHash from SEQ_FIRSTRUN_ADMINPASSWORD,
   # just like the Seq deployment was given SEQ_FIRSTRUN_ADMINPASSWORDHASH instead.
   # This hack is required in order to:
   # 1. Avoid prompt for password change during initial login to Seq UI
   # 2. Allow re-use of password from Kubernetes Secret by agents external to Seq,
   #    like the seq-apikey-operator or other which need to generate dedicated
   #    Seq API key in order to perform any operations with Seq CLI or API.
   #
   set -e
   SCRIPT_NAME="$(basename "$0")"
   
   LOGFILE="/var/log/seqinit/${SCRIPT_NAME%.sh}.log"
   mkdir -p "$(dirname "${LOGFILE}")"
   
   function echolog { echo "$(printf '%(%F %T)T') [${SCRIPT_NAME}] INFO $*" | tee -a "${LOGFILE}"; }
   function echoerr { echo "$(printf '%(%F %T)T') [${SCRIPT_NAME}] ERROR $*" | tee -a "${LOGFILE}"; }
   
   echolog "Logging ${SCRIPT_NAME} actions to ${LOGFILE}"
   echolog "Listing Seq environment variables"
   env | grep SEQ_
   cfg_admin_username=${SEQ_FIRSTRUN_ADMINUSERNAME:-"admin"}
   cfg_auth_ingestion=${SEQ_FIRSTRUN_REQUIREAUTHENTICATIONFORHTTPINGESTION:-"true"}
   cfg_no_auth=${SEQ_FIRSTRUN_NOAUTHENTICATION:-"false"}
   if [[ -n "${SEQ_FIRSTRUN_ADMINPASSWORD}" ]]; then
       echolog "Generating admin password hash from SEQ_FIRSTRUN_ADMINPASSWORD"
       cfg_admin_password_hash=$(echo "${SEQ_FIRSTRUN_ADMINPASSWORD}" | seqsvr config hash)
   elif [[ -n "${SEQ_FIRSTRUN_ADMINPASSWORDHASH}" ]]; then
       echolog "Using admin password hash from SEQ_FIRSTRUN_ADMINPASSWORDHASH"
       cfg_admin_password_hash="${SEQ_FIRSTRUN_ADMINPASSWORDHASH}"
   else
       echoerr "SEQ_FIRSTRUN_ADMINPASSWORD or SEQ_FIRSTRUN_ADMINPASSWORDHASH is missing"
       exit 1
   fi
   echolog "Setting firstRun.adminUsername"
   seqsvr config set -k firstRun.adminUsername -v "${cfg_admin_username}"
   echolog "Setting firstRun.adminPasswordHash"
   seqsvr config set -k firstRun.adminPasswordHash -v "${cfg_admin_password_hash}"
   echolog "Setting firstRun.noAuthentication"
   seqsvr config set -k firstRun.noAuthentication -v "${cfg_no_auth}"
   echolog "Setting firstRun.requireAuthenticationForHttpIngestion"
   seqsvr config set -k firstRun.requireAuthenticationForHttpIngestion -v "${cfg_auth_ingestion}"
   echolog "Listing firstRun.* settings in Seq.json file:"
   seqsvr config list | grep firstRun
   echolog "Done"

5. Deploy Seq using helm_release with values parametrised this way

     values = [
       templatefile("${path.module}/templates/seq/values.yaml.tftpl", {
         admin_password_secret_name  = local.seq_admin_password_secret_name
         admin_password_secret_key   = "password" # see ExternalSecret above
         init_scripts_configmap_name = local.seq_init_scripts_configmap_name
         dashboard_fqdn              = local.seq_dashboard_fqdn
         ingestion_fqdn              = local.seq_ingestion_fqdn
         storage_size                = local.seq_storage_size
         tls_certificate_secret_name = local.seq_tls_secret_name # see kubectl_manifest.seq_cert_manager_certificate
       })
     ]

   where the values.yaml.tftpl is this:

   extraVolumes:
   - name: seq-init-scripts
     configMap:
       name: ${init_scripts_configmap_name}
   
   extraVolumeMounts:
   - name: seq-init-scripts
     mountPath: /seqinit
   
   firstRunAdminPasswordSecret:
     name: ${admin_password_secret_name}
     key: ${admin_password_secret_key}
   
   firstRunRequireAuthenticationForHttpIngestion: true
   
   image:
     pullPolicy: IfNotPresent
     repository: datalust/seq
     tag: "2025"
   
   ingestion:
     ingress:
       enabled: true
       hosts:
       - ${ingestion_fqdn}
       path: /
   ui:
     ingress:
       enabled: true
       hosts:
       - ${dashboard_fqdn}
       path: /
   
   ingress:
     className: kong
     tls:
     - hosts:
       - ${dashboard_fqdn}
       - ${ingestion_fqdn}
       secretName: ${tls_certificate_secret_name}
   
   persistence:
     enabled: true
     size: ${storage_size}
     storageClass: managed-csi
   
   resources:
     limits:
       memory: 4Gi # Standard_D2d_v5 has RAM of 8 GiB
   
   nodeSelector:
     kubernetes.io/os: linux

6. Finally, my custom Seq API keys operator automagically generates a privileged admin API key using the password stored in Kubernetes secrets (see above), according to https://datalust.co/docs/command-line-client#connecting-without-an-api-key. This admin API key is then used to execute seqcli apikey create|update|remove to manage application specific API keys and any other Seq admin operations.

That would be it. I hope it helps.

Kubernetes Operator

However, I'm very keen in developing Kubernetes operator and here are the basic available options, that I have been thinking about:

1. Operator written in C# using the official API client library for Seq and KubeOps. Piggy-backing on the official API client seems like simplest option.
2. Develop Go client for Seq, then develop Kubernetes Operator for Seq in Go. This is more laborious option, but 1) Go client may turn out to be a very useful and desired project on its own 2) Go as the Kubernetes native language seems to have better potential for contributions from Kubernetes experts.

Terraform Provider

By the way, I've been planning to resurrect the Seq provider for Terraform originally developed by Innovation Norway, which I contacted a while ago and managed to fork it, https://github.com/mloskot/terraform-provider-datalust-seqseq, but I haven't managed to find time to work on it yet.

The Innovation Norway also provided me with their old Go client for Seq, https://github.com/mloskot/go-seq-obsolete, which possible could serve as basis for an updated Go client for Seq developed as part of the option 2. above.

Plan?

I think, it would also be good to know what is the sentiment about the Kubernetes operator for Seq within the Seq team.
If the Seq team is interested in such project or has planned, then joining the efforts from the Seq team and the Seq community could be a most productive way forward.

[wasabii]

Yeah, so if I did this, it'd probably be in C# using KubeOps. I've used that before. I wrote an auth0 operator in it. It works fine. And I don't really want to do Go.

I'd probably start by considering it as management-only as a first pass: being able to create and manage entities in an existing Seq instance given a hostname and initial password. Once that worked good enough, I'd graft on the ability to deploy new Seq instances. But, for me, that would be an optional second-step since I currently use Flux and HelmRelease to deploy Seq just fine.

[mloskot]

The idea of Seq Configurator operator sounds good to me too. I don't mind writing C#, so I'd be interested in testing/contributing too.

[wasabii]

https://github.com/alethic/seq-operator

[wasabii]

I imported my basic structure from the Auth0 operator I have, but removed a bunch of stuff.

[mloskot]

Excellent, let's move the detailed discussions there.

Naming is hard...I'd probably rename seq-operator as seq-config-operator to make its purpose clear and allow spawning seq-operator for full-fledged operator in future, without breaking the simpler one. Just a brainstorm.

[wasabii]

alethic/seq-operator#2

[KodrAus]

Sorry we haven't commented yet on what's been happening here; but have been following along. My experience from maintaining the Helm repository so far has been that we don't currently have the bandwidth or experience internally to dedicate to maintaining product-level Kubernetes integration. The Helm chart itself is largely unguided and I've never been particularly satisfied with it or Helm in general. Am I right that an operator would be a successor to the Helm chart? The fact that the operator you've been building @wasabii is largely C#-based rather than YAML templating is certainly very appealing.

For us, I think we should be aiming to push the product and its tooling in a direction that makes these higher-level stack-specific integrations as easy as possible to pull together. What I think that means in the immediate term is ensuring seqcli has broad coverage of Seq's API and behaves in a way that's consistent with how these kinds of tools expect, and filling in API automation gaps in the container itself.

If there's anything you run into that makes this difficult or that you need to work around please do create issues for them and we'll see if we can improve things 👍

[wasabii]

Am I right that an operator would be a successor to the Helm chart?

For some people.

The Kubernetes eco system isn't that straightforward. Some people expect well built Helm charts. Other people want an operator to do configuration. Some people want an operator to do deployment and configuration. Some people want all of that to work both cluster-level and per-namespace.

[wasabii]

I'll use RabbitMQ as an example. They have a well done container image, that some people just include as their own Pod as part of their product. They also have a Helm chart, which does a single node or cluster installation. Some people use that. They can be namespace isolated. They also have a Cluster Operator, which can facilitate different things: either private isolated teams (tenants) deploying their own cluster using a RabbitMQCluster resource, or having a central cluster instance, but allowing each team to deploy custom Kubernetes resources for their own isolated VHost on the same cluster (single cluster multiple tenants, or multiple clusters). Then they also have a Topology Operator which can be used to add things to a RabbitMQ instance: queues, exchanges, users, etc, whether that instance is deployed by hand, deployed by Helm, external to Kubernetes itself, or deployed by the previous Cluster Operator.

It's all over the map what works for people. K8s can be small. But it can also be massive.

[wasabii]

Seq itself doesn't have too much of a per-instance isolation model built in. So one of those models is probably not too useful. But I can see a few possibilities:

a) User has an external Seq instance, hosted somewhere else, but wants to configure it using K8s resources. Like an ApiKey resource. They might use the operator to configure it.
b) User is just installing Seq on their cluster, but it's sort of a single purpose cluster, and the user isn't super advanced. They probably just want a Helm chart that Just Works.
c) User wants to embed Seq into their product in their product's Helm chart. They could use a chart Include. But they could also just ignore the chart and add their own StatefulSet into their own product refering to the official pod instance. If their cluster has ArgoCD or Flux, maybe they want to use App of Apps pattern.
d) User is a member of a product team inside of a larger corp, and wants to incldue Seq in their product, but hasn't got buy in for any organization-level participation: he might just install the Helm chart in his own namespace, and have his own services use it.
e) User runs a medium sized cluster. He has a bunch of products running in his cluster, but he wants to provide some Seq-as-a-service to those apps. There's not really any data-isolation going on. He might use the operator to install a single Instance, but give access to each other product to add an ApiKey resource pointing to that central cluster. Each product group allocates its own ApiKey. He probaly wants this Seq-as-a-service to integrat with his IdP too, so he wants to use the operator to configure that. Basic permissions, but a single instance, with fairly closely watched requirements.
f) User runs a massive multi-tenant K8s cluster, with many disparent teams below him. He wants to produce Seq as a service, but allow product groups below him to deploy their own clusters. He might grant permission to his tenants to create their own Seq Instance records, with their own configuration. Additionally he might use something like Crossplane or Kyverno to govern that process.

[mloskot]

@wasabii

The Kubernetes eco system isn't that straightforward.
(...)
But I can see a few possibilities:
(...)

That is perfectly captured!

[wasabii]

Just to let ya'll here know, the operator I built in like a week is up and working. ;)