[Proposal][Discuss] Gitea Cluster

[lunny]

How does a Gitea deployment scale? Gitea cluster should resolve part of it.

Currently when running several Gitea instances which shared database, git storage. There is still something needs to resolve.

• Crons: Now every Gitea instance will run all the crons. It is duplicated work and will waste CPU and disk. The idea is the cron tasks should be spliced into all the Gitea instances.
• Migrating: You cannot stop the running migrating task. Because you don't know which Gitea instance is running it.
• Git Storage: A shared/copied git storage is required. If every Gitea instance will only store part of the repositories and when requests come, they will be routed to the right Gitea instance. Integrating Gitaly is a possible resolution.

comment by @wxiaoguang

• If there is no session-stick or ip-stick, I guess it would trigger database deadlock more frequently due to transaction conflicts.
• Actions Crons: And it conflicts with actions cron, there will be multiple tasks.
• Packages: And it would trigger docker's duplicate insert bug again because there is only a workaround (in process mutex) at the moment.
• UI notifications: I guess "event source" doesn't work with cluster either
• Locks: Some packages depend on the ExclusivePool pool, which is also in-process now.
  Use global lock instead of NewExclusivePool to allow distributed lock between multiple Gitea instances #31813(based on Introduce globallock as distributed locks #31908)

[6543]

• for cron I propose: cron only create tasks, witch are represented in DB (like it's done with migration tasks)

• for tasks: each instance should have a unique ID (GUID), if an instances fetch tasks from DB and alter there state by changing Status to running + adding GUID & PID into table

• there must be some way gitea instances can speac to each other by GUID as Identifyer to:

  • send cancle/pause/continue "signals"

• propose a hardbeat to recover & cleanup tasks of crashed gitea instances:

type Hardbeat struct {
  guid        int64
  beat        int64
  recoverGUID int64 // empty if nothing crashed 
}


on process.Manager creation start hardbeatFunc()

func hardbeatFunc() {
  for {
    x.Where(guid=getGUID()).Update(&Hardbeat{beat=unixtime.Now()})
    
    for _, crash := x.Select(guids with timeout && recoverGUID == "") {
      x.Where(crash.guid).Update(recoverGUID = getGUID())
      // make sure no other instance has taken the recover step - and 
      if !x.Exist(guid=crash.guid, recoverGUID=getGUID()) {ret error}
      
      now reset all tasks with crash.guid

    }
    
    sleep 20sec

  }
}

modules/tasks task will need to be refactor to have an easy interface:
task.Signal(task.CANCEL, guid, pid) <- if guid is not of running instance, send it to the specific one ...
task.Run(t *task)
...

[lafriks]

Some kind of git storage layer would be needed imho (something like gitlab has)

[6543]

I would fokus on tasks since git data via shared-storage work quite well at the moment

[lunny]

I would fokus on tasks since git data via shared-storage work quite well at the moment

It is but in fact it's expensive. So a distributed git data storage layer still be a necessary feature of Gitea in future.

[Codeberg-org]

I would fokus on tasks since git data via shared-storage work quite well at the moment

+1

Safe distributed/concurrent gitea is surely the highest priority from a user point of view, as off-the-shelf options for distributed SQL databases and distributed file systems are readily available.

[6543]

Roadmap:

1. master elec
2. log & com for processManager com
3. tasks

master elec

done by DBMS: who get SQL select-update query in first

• need hearbeat table in DB
• GUID creation in process.GetManager()

~7msg types

• CANCEL - cancle processes
• ACK
• LIST - get running processes
• IMRUNNING
• PING - are you alive
• STATUS - get status of a specific process
• CREATE - opt. create process on other instance

msg com

some sort of https://nats.io/, https://activemq.apache.org/cross-language-clients, ... over DB, Redis, ... ?

sidenotes

• only master should start cron jobs
• ? trigger webhooks ?

[gary-mazz]

Interesting discussion. I think this started back in 2017 #2959.

There needs to be recognition of 2 cluster use cases: Load Balancing and High Availability (HA) with 2 types of location configurations: local and remote.

The more distant the cluster participant, data shifts from synchronous (near -real-time) to delayed; creating a spectrum of data synchronization quality levels from highly consistent to eventually consistent.

Technologies picked should be able to operate at a distance as well as on local prem without reconfiguration. Secure communications via tunneling and certificate based authentication between nodes should also be considered.

The "tricky part" is figuring out where to put the replication. Since gitea supports multiple databases, and each employs different and incompatible replication mechanisms, a formalized middle-ware layer is likely required to replicate data. The mid-layer replications also allows different db backend configuration (eg postgresql and Mysql) to provide transparent replication.

Replications will need some type lockout strategy for check-in/outs and zips operations during replication activity. The options are:

1. lockout client access until replication activity is completed
2. Lockout replication updates (cache operations) until client activity is complete.
3. Fail client operations if replication updates touches files in client operations
4. Delay/pause client operation until replication activities and status checks are complete. (for remote site failover and load balancing..)

With remote site load balancing, it is possible to have check-in collisions causing inconsistencies. The use cases that cause these conditions:

1. system clocks fall out of sync between servers (both local and remote locations)
2. remote site load balancing loses replication network connection(s) (two headed monster)
3. normal networking and load delays, cause race conditions between servers (occurs both local and remote configurations)

I hope this helps some of your design decisions

PS: don't forget config files change pushes..

[lafriks]

We should probably also need some kind of git repository access layer so that they could be distributed across cluster with local storage

[imacks]

Just want to contribute my own experience using Gitea for the last couple of years.

Our first attempt was to run dockerized Gitea in kube, with storage back end provided by NFS. We rely on kube healthcheck to restart an unresponsive Gitea instance, which can run on any tainted host managed by kube. This solves the reliability issue somewhat, though there will be a period of unavailability while the container restarts.

Our v2 setup swaps out NFS for ceph CSI in kube. R/W performance improves dramatically. We also use S3 compat layer in ceph to store LFS data.

My most pressing desire for v3 is HA. We can be less ambitious and work on single local cluster first. There can be a dedicated pod for running cron tasks, so Gitea can concentrate on doing git and webserver stuff. We can also use s3 for storage exclusively for its sync capabilities.