Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have a task template when starting a job with it , it would call other tasks and query the status of these tasks. After a few hours running ,the RSS of semaphore process can raise to nearly 6GB. So I dig into the pprof :
╰─➤ go tool pprof ./semaphore mem-1748411057.prof
File: semaphore
Build ID: bc74ae534f682663ad480d52d875d15dff56ceea
Type: inuse_space
Time: 2025-05-28 13:44:17
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 3440.74MB, 81.64% of 4214.56MB total
Dropped 58 nodes (cum <= 21.07MB)
Showing top 10 nodes out of 74
flat flat% sum% cum cum%
741.16MB 17.59% 17.59% 757.16MB 17.97% net/textproto.readMIMEHeader
473.54MB 11.24% 28.82% 582.55MB 13.82% encoding/json.(*decodeState).literalStore
456.68MB 10.84% 39.66% 456.68MB 10.84% github.com/gorilla/context.Set
398.12MB 9.45% 49.10% 398.12MB 9.45% net/http.(*Request).WithContext
366.10MB 8.69% 57.79% 366.10MB 8.69% github.com/gorilla/mux.extractVars
297.57MB 7.06% 64.85% 769.61MB 18.26% github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.func1
206.02MB 4.89% 69.74% 343.53MB 8.15% context.withCancel
181.02MB 4.30% 74.03% 181.02MB 4.30% net/url.parse
171.51MB 4.07% 78.10% 171.51MB 4.07% context.WithValue
149.02MB 3.54% 81.64% 262.02MB 6.22% github.com/semaphoreui/semaphore/api.authenticationHandler
(pprof) list github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.fun
c1
Total: 4.12GB
ROUTINE ======================== github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.func1 in /go/src/semaphore/api/projects/tasks.go
297.57MB 769.61MB (flat, cum) 18.26% of Total
. . 105:func GetTaskMiddleware(next http.Handler) http.Handler {
. . 106: return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
. . 107: project := context.Get(r, "project").(db.Project)
. . 108: taskID, err := helpers.GetIntParam("task_id", w, r)
. . 109:
. . 110: if err != nil {
. . 111: util.LogErrorF(err, log.Fields{"error": "Bad request. Cannot get task_id from request"})
. . 112: w.WriteHeader(http.StatusBadRequest)
. . 113: return
. . 114: }
. 472.04MB 115:
. . 116: task, err := helpers.Store(r).GetTask(project.ID, taskID)
. . 117: if err != nil {
. . 118: util.LogErrorF(err, log.Fields{"error": "Bad request. Cannot get task from database"})
. . 119: w.WriteHeader(http.StatusBadRequest)
. . 120: return
. . 121: }
297.57MB 297.57MB 122:
. . 123: context.Set(r, "task", task)
. . 124: next.ServeHTTP(w, r)
. . 125: })
. . 126:}
. . 127:
And I do find out that there was memory leak in gorilla/context. for Get function , it just get a value from *http.Request which won't be a problem. But for Set function, it will store an object in a global map object map[*http.Request]map[interface{}]interface{} , so it won't be released after the response is send back to client.
Unfortunately the task object seems bigger than other semaphore object , so it consume more memory . So I clear the context object just after it finished the api call dedicately.For other objects , the memory problem seems no so obviously, if did cause a memory leak, we can call a gc to purge context objects which created 10 minutes ago.
And as suggest by gorilla/context:
You should use the http.Request.Context() feature in Go 1.7.
Fixed the memory leak while using gorilla/context.Set. Clear the global map object with current request after tasks api return or manually release all the expired objects in context using api of gc.