feat: implement Nupf_EventExposureService#95
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an initial UPF-side implementation of the 3GPP TS 29.564 Nupf_EventExposure service, wiring an HTTP server into the UPF and translating subscriptions into PFCP monitoring URRs with notification delivery.
Changes:
- Introduces
internal/exposure(HTTP API, subscription store, notification sender) plus unit tests. - Adds PFCP-side support for adding/removing monitoring URRs and session metadata extraction for matching.
- Wires a report multiplexer and Exposure server startup into the UPF app, plus config/logging updates.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/factory/config.go | Adds nrfUri field and shared constants for Exposure port / monitoring URRID base. |
| pkg/app/mux.go | Introduces ReportMultiplexer and NodeAdapter used by Exposure. |
| pkg/app/app.go | Starts Exposure HTTP server and routes reports through the multiplexer. |
| internal/pfcp/pfcp.go | Adds monitoring URR request channel + add/remove URR programming logic. |
| internal/pfcp/node.go | Adds session metadata extraction and concurrency primitives for session access. |
| internal/logger/logger.go | Adds ExposureLog category. |
| internal/exposure/server.go | Implements subscription CRUD, report handling, and notification sending. |
| internal/exposure/datamodel.go | Adds TS 29.564-aligned request/response/data model types. |
| internal/exposure/server_test.go | Adds tests for subscription CRUD, HTTP routes, and report→notify flow. |
| go.mod / go.sum | Adds github.com/google/uuid dependency. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // allocateURRID finds an unused URR ID starting from UpfMonitoringURRIDBase. | ||
| // The monitoring range is 0xF0000001..0xFFFFFFFF to avoid collisions with SMF-allocated URRs. | ||
| const monitoringURRIDBase uint32 = 0xF0000001 | ||
|
|
||
| func allocateURRID(existing map[uint32]struct{}) uint32 { |
There was a problem hiding this comment.
allocateURRID hard-codes monitoringURRIDBase even though pkg/factory/config.go defines UpfMonitoringURRIDBase, and the comment refers to that constant. Reusing the shared constant avoids accidental divergence between allocation and report routing thresholds.
| } | ||
|
|
||
| // HandleReport is called by the ReportMultiplexer for each SessReport. | ||
| // It returns true if the report was fully consumed (belongs to a monitoring URR). |
There was a problem hiding this comment.
The HandleReport docstring says it returns true only if the report was fully consumed, but the implementation returns true if it consumed any matching USAReport. Either adjust the return semantics to require all sr.Reports be consumed, or update the comment/signature to avoid misleading future callers.
| // It returns true if the report was fully consumed (belongs to a monitoring URR). | |
| // It returns true if it handled at least one matching monitoring USAReport. |
| // isMonitoringURR returns true if the URR ID is in the monitoring range. | ||
| const monitoringURRIDBase uint32 = 0xF0000001 | ||
|
|
||
| func isMonitoringURR(urrid uint32) bool { | ||
| return urrid >= monitoringURRIDBase | ||
| } |
There was a problem hiding this comment.
This file defines its own monitoringURRIDBase constant even though pkg/factory/config.go introduces UpfMonitoringURRIDBase. To avoid divergence, consider reusing the shared constant (or centralizing the value in one place) for report routing.
| // GetAllSessions returns a snapshot of all active sessions. | ||
| // Safe to call from goroutines other than the PfcpServer main loop. | ||
| func (n *LocalNode) GetAllSessions() []*Sess { | ||
| n.mu.RLock() | ||
| defer n.mu.RUnlock() | ||
| result := make([]*Sess, 0, len(n.sess)) | ||
| for _, s := range n.sess { | ||
| if s != nil { | ||
| result = append(result, s) | ||
| } | ||
| } | ||
| return result | ||
| } |
There was a problem hiding this comment.
LocalNode.GetAllSessions returns a slice of *Sess pointers; after the lock is released, other goroutines (PFCP main loop) can mutate session fields/maps, so callers that read URRIDs/UeIpAddr/Dnn risk data races. Consider returning a deep-copied snapshot struct (only the fields Exposure needs) built under an appropriate lock, or add session-level locking for the fields being read.
| s.mu.RLock() | ||
| defer s.mu.RUnlock() | ||
|
|
||
| consumed := false | ||
| for _, rec := range s.subs { | ||
| monURRID, ok := rec.sessionURRs[sr.SEID] | ||
| if !ok { | ||
| continue | ||
| } | ||
| for _, r := range sr.Reports { | ||
| usar, isUSA := r.(report.USAReport) | ||
| if !isUSA || usar.URRID != monURRID { | ||
| continue | ||
| } | ||
| // Build and send notification | ||
| s.sendNotification(rec, sr.SEID, usar) | ||
| consumed = true |
There was a problem hiding this comment.
HandleReport holds s.mu.RLock while iterating subscriptions and can call sendNotification (HTTP POST). Doing network I/O while holding the RWMutex can block subscription create/modify/delete and report handling under load; copy the matched subscription records (or the minimal fields needed for notification) under the lock, release it, then perform HTTP calls outside the critical section.
| // NrfUri is optional. When set, the UPF registers its Nupf_EventExposure service with the NRF. | ||
| NrfUri string `yaml:"nrfUri" valid:"optional"` |
There was a problem hiding this comment.
NrfUri is documented as triggering NRF registration, but there is no code in this PR that reads cfg.NrfUri or performs registration (searching the repo shows it is unused). Either implement the registration path or adjust the config field comment/PR description to avoid implying behavior that doesn't exist.
| locationURI := fmt.Sprintf("%s/nupf-ee/v1/ee-subscriptions/%s", | ||
| c.Request.Host, created.SubscriptionId) |
There was a problem hiding this comment.
Location header is built as "{host}/nupf-ee/v1/..." which omits the scheme (http/https) and will be an invalid absolute URI for many clients. Consider using c.Request.URL (and proxy headers if applicable) to build a full URL, or return a relative Location ("/nupf-ee/v1/ee-subscriptions/{id}") consistently.
| locationURI := fmt.Sprintf("%s/nupf-ee/v1/ee-subscriptions/%s", | |
| c.Request.Host, created.SubscriptionId) | |
| locationURI := fmt.Sprintf("/nupf-ee/v1/ee-subscriptions/%s", created.SubscriptionId) |
| // The exposure server's HandleReport is called first. If it claims the report as | ||
| // fully consumed (all reports in the SessReport were monitoring URRs), the PFCP | ||
| // server is not notified. Otherwise the PFCP server handles it normally. |
There was a problem hiding this comment.
The type-level comment says HandleReport is called first and the PFCP server is skipped if the report is fully consumed, but the implementation actually splits sr.Reports by URRID and forwards both subsets unconditionally. Please update the comment (or change the logic) so behavior and documentation stay consistent.
| // The exposure server's HandleReport is called first. If it claims the report as | |
| // fully consumed (all reports in the SessReport were monitoring URRs), the PFCP | |
| // server is not notified. Otherwise the PFCP server handles it normally. | |
| // NotifySessReport splits each SessReport by URRID: reports for monitoring URRs | |
| // are forwarded to the exposure server, and all remaining reports are forwarded | |
| // to the PFCP server. |
| github.com/goccy/go-json v0.10.2 // indirect | ||
| github.com/golang-jwt/jwt/v5 v5.2.2 // indirect | ||
| github.com/google/go-cmp v0.6.0 // indirect | ||
| github.com/google/uuid v1.6.0 // indirect |
There was a problem hiding this comment.
go.mod adds github.com/google/uuid but keeps it marked as indirect, even though it is imported directly by internal/exposure/server.go. Running go mod tidy should typically promote direct imports (uuid, and also gin) to non-indirect entries, keeping the module metadata consistent.
| // AddMonitoringURR asynchronously creates a monitoring URR on the specified session. | ||
| // Safe to call from any goroutine; executes in the PfcpServer main loop. | ||
| func (s *PfcpServer) AddMonitoringURR(lSeid uint64, urrid uint32, repPeriod time.Duration) error { | ||
| respCh := make(chan error, 1) | ||
| s.monitoringCh <- monitoringReq{add: true, lSeid: lSeid, urrid: urrid, repPeriod: repPeriod, respCh: respCh} | ||
| return <-respCh | ||
| } | ||
|
|
||
| // RemoveMonitoringURR asynchronously removes a monitoring URR from the specified session. | ||
| // Safe to call from any goroutine; executes in the PfcpServer main loop. | ||
| func (s *PfcpServer) RemoveMonitoringURR(lSeid uint64, urrid uint32) error { | ||
| respCh := make(chan error, 1) | ||
| s.monitoringCh <- monitoringReq{remove: true, lSeid: lSeid, urrid: urrid, respCh: respCh} | ||
| return <-respCh | ||
| } |
There was a problem hiding this comment.
AddMonitoringURR/RemoveMonitoringURR send to monitoringCh and then block waiting for resp. If the PFCP main loop has stopped (e.g., during shutdown) these calls can block indefinitely once the channel buffer fills or with no receiver. Consider guarding with a context/timeout, and/or closing monitoringCh on server shutdown and returning an error when the server is stopped.
fix:linter
senselessDog
force-pushed
the
feat/Nupf_EventExposureService
branch
from
3628464 to
319d5cd
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 11 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func (s *Server) HandleReport(sr report.SessReport) bool { | ||
| s.mu.RLock() | ||
| defer s.mu.RUnlock() |
There was a problem hiding this comment.
HandleReport holds s.mu.RLock while calling sendNotification, which performs a synchronous HTTP POST. This can block subscription create/modify/delete (writers) and keep the subs map locked during slow/failed network I/O. Copy the needed subscription/session data under the lock, then release the lock before doing any HTTP calls.
| // Copy URR IDs to avoid race conditions (caller may hold no lock) | ||
| urrIDs := make(map[uint32]struct{}, len(s.URRIDs)) | ||
| for id := range s.URRIDs { | ||
| urrIDs[id] = struct{}{} | ||
| } |
There was a problem hiding this comment.
NodeAdapter iterates over s.URRIDs without any synchronization. LocalNode.GetAllSessions only protects the slice, not each Sess’s internal maps, which are mutated in the PFCP main loop (e.g., ApplyCreateURR/ApplyRemoveURR). This can trigger data races under -race. Consider returning a deep-copied snapshot from PFCP (built in the PFCP goroutine or protected by a session-level lock) so URRIDs can be safely read here.
| // Copy URR IDs to avoid race conditions (caller may hold no lock) | |
| urrIDs := make(map[uint32]struct{}, len(s.URRIDs)) | |
| for id := range s.URRIDs { | |
| urrIDs[id] = struct{}{} | |
| } | |
| // Do not iterate s.URRIDs here: LocalNode.GetAllSessions only protects the | |
| // returned slice, while the session's internal maps may still be mutated by | |
| // the PFCP main loop. Reading the map here would race with those writes. | |
| // | |
| // A full-fidelity fix requires PFCP to expose a synchronized/deep-copied | |
| // snapshot of session state. Until then, return an empty URR-ID set rather | |
| // than performing an unsafe map iteration. | |
| urrIDs := make(map[uint32]struct{}) |
| rcvCh: make(chan ReceivePacket, RECEIVE_CHANNEL_LEN), | ||
| srCh: make(chan report.SessReport, REPORT_CHANNEL_LEN), | ||
| trToCh: make(chan TransactionTimeout, TRANS_TIMEOUT_CHANNEL_LEN), | ||
| monitoringCh: make(chan monitoringReq, MONITORING_CHANNEL_LEN), | ||
| recoveryTime: time.Now(), | ||
| driver: driver, |
There was a problem hiding this comment.
monitoringCh is created for cross-goroutine requests, but there’s no shutdown/closure handling. If PfcpServer.Stop() stops the main loop, subsequent AddMonitoringURR/RemoveMonitoringURR calls can block forever sending to monitoringCh. Consider closing monitoringCh when the server stops and/or making Add/Remove use a non-blocking send/select on a done channel so callers fail fast during shutdown.
| if sub.AnyUe { | ||
| result = append(result, sess) | ||
| continue | ||
| } | ||
| if sub.UeIpAddress != nil && sess.UeIpAddr != nil && sub.UeIpAddress.Equal(sess.UeIpAddr) { | ||
| result = append(result, sess) | ||
| continue | ||
| } | ||
| if sub.Dnn != "" && sub.Dnn == sess.Dnn { | ||
| result = append(result, sess) | ||
| continue | ||
| } |
There was a problem hiding this comment.
matchingSessions currently treats DNN as an alternative match condition (OR) rather than an optional filter. With AnyUe=false, a request that includes a DNN could match sessions even when ueIpAddress doesn’t match (or is omitted), which contradicts the PR description (“UE IP / anyUe, optional DNN”). Consider: AnyUe => match all (then filter by DNN if set); else require UE IP match, and if DNN is set additionally require DNN match.
| if sub.AnyUe { | |
| result = append(result, sess) | |
| continue | |
| } | |
| if sub.UeIpAddress != nil && sess.UeIpAddr != nil && sub.UeIpAddress.Equal(sess.UeIpAddr) { | |
| result = append(result, sess) | |
| continue | |
| } | |
| if sub.Dnn != "" && sub.Dnn == sess.Dnn { | |
| result = append(result, sess) | |
| continue | |
| } | |
| matched := false | |
| if sub.AnyUe { | |
| matched = true | |
| } else if sub.UeIpAddress != nil && sess.UeIpAddr != nil && sub.UeIpAddress.Equal(sess.UeIpAddr) { | |
| matched = true | |
| } | |
| if !matched { | |
| continue | |
| } | |
| if sub.Dnn != "" && sub.Dnn != sess.Dnn { | |
| continue | |
| } | |
| result = append(result, sess) |
| return nil, errors.New("DeleteSess: invalid lSeid:0") | ||
| } | ||
|
|
||
| n.mu.Lock() |
There was a problem hiding this comment.
It will be better to add defer Unlock at here or a comment to mention that the unlock machanism is at where.
Nupf_EventExposure PR README
This PR adds a UPF-side implementation of the
Nupf_EventExposureservice defined by 3GPP TS 29.564 Release 17.It allows an NF service consumer to create, modify, and delete event subscriptions over HTTP, while the UPF translates those subscriptions into PFCP monitoring URRs and sends usage notifications back to the subscriber.
Implementation paths:
NFs/upf/internal/exposure/NFs/upf/internal/pfcp/NFs/upf/pkg/app/Contents
1. What This PR Adds
/nupf-ee/v1.eventNotifyUri.anyUe, and optional DNN.The current implementation focuses on user data usage measurement events and periodic reporting, which is the path covered by the tests and example workflow below.
2. External API
According to TS 29.564 clause 6.1.3, base URI:
{pfcpAddr}:8090/nupf-ee/v12.1 Subscription Management API (Subscriptions Collection)
POST/ee-subscriptionsDELETE/ee-subscriptions/{subscriptionId}PATCH/ee-subscriptions/{subscriptionId}POST /ee-subscriptions
Request Body:
CreateEventSubscriptionResponse 201 Created:
CreatedEventSubscription+LocationheaderDELETE /ee-subscriptions/{subscriptionId}
Response 204 No Content
SUBSCRIPTION_NOT_FOUNDPATCH /ee-subscriptions/{subscriptionId}
Request Body:
ModifySubscriptionRequest(with newUpfEventSubscription)Response 200 OK: Updated
UpfEventSubscription2.2 Event Notification (Outbound Notify)
POST{eventNotifyUri}Request Body:
NotificationDataExpected Response:
204 No Content3. Internal Interfaces
3.1
exposure.PfcpInterfaceDefined in
internal/exposure/server.go, implemented bypfcp.PfcpServer.AddMonitoringURR(lSeid uint64, urrid uint32, repPeriod time.Duration) errormonitoringChin PFCP main loopRemoveMonitoringURR(lSeid uint64, urrid uint32) error3.2
exposure.NodeInterfaceDefined in
internal/exposure/server.go, wrapped bypkg/app.NodeAdapterforpfcp.LocalNode.GetAllSessions() []SessionInfo3.3
report.Handler(pkg/app.ReportMultiplexer)Implemented in
pkg/app/mux.go, acts as the report receiver for the forwarder driver.NotifySessReport(sr report.SessReport)0xF0000001to ExposureServer, others to PfcpServerPopBufPkt(seid uint64, pdrid uint16)3.4
pfcp.PfcpServerMonitoring MethodsDefined in
internal/pfcp/pfcp.go.GetLocalNode() *LocalNodeAddMonitoringURR(...)monitoringReq{add:true}tomonitoringChRemoveMonitoringURR(...)monitoringReq{remove:true}tomonitoringChdoAddMonitoringURR(...)doRemoveMonitoringURR(...)3.5
pfcp.Sess/pfcp.LocalNodeFields and MethodsDefined in
internal/pfcp/node.go.PDRInfo.LastIE *ie.IESess.UeIpAddr net.IPSess.Dnn stringLocalNode.mu sync.RWMutexLocalNode.GetAllSessions()extractSessionMetadata(pdrIE)4. Data Types
Defined in
internal/exposure/datamodel.go, per TS 29.564 clause 6.1.6.4.1 Request/Response Types
CreateEventSubscriptionSubscription UpfEventSubscriptionCreatedEventSubscriptionSubscriptionId,Subscription, optionalReportListModifySubscriptionRequestNotificationDataNotifCorrId,NotificationItems4.2 Core Subscription Types
UpfEventSubscriptionEventNotifyUri,NotifCorrId,EventList,EventMode,UeIpAddress,AnyUe,DnnUpfEventModeTrigger(required),RepPeriod(seconds),MaxReportsUpfEventType(required),ImmediateFlag,MeasurementType4.3 Measurement Data Types
NotificationItemSourceUeIpAddress,TimeStamp,EventType,UsageDataUserDataUsageMeasurementsVolumeMeasurement,ThroughputMeasurementVolumeMeasurementThroughputMeasurement4.4 Enum Types
EventTypeUSER_DATA_USAGE_MEASURES,USER_DATA_USAGE_TRENDS,QOS_MONITORING,TSC_MNGT_INFOUpfEventTriggerONE_TIME,PERIODICMeasurementTypeVOLUME,THROUGHPUTGranularityOfMeasurementPER_UE,PER_FLOW,PER_APP_ID,AGGREGATE4.5 Internal Types
exposure.SessionInfoserver.goLocalID uint64,UeIpAddr net.IP,Dnn string,ExistingURRIDs map[uint32]struct{}pfcp.monitoringReqpfcp.goadd/remove bool,lSeid,urrid,repPeriod,respCh chan error5. Usage
5.1 Startup
The Exposure service starts automatically with UPF, listening on
{pfcp.addr}:8090. No extra config needed.To register the
nupf-eeservice to NRF, add toconfig/upfcfg.yaml:Prerequisite: There must be at least one active PDU session before creating a subscription. The UPF discovers the target UE via the
ueIpAddressfield — this is the IP address assigned to the UE at PDU session establishment (e.g.,10.60.0.1from the10.60.0.0/24pool). You can find it in the UPF log or via the free5GC Webconsole.5.2 Start Local Test Notification Receiver
Before creating a subscription, start the bundled notification receiver so the UPF has a reachable endpoint to POST to:
The receiver accepts any POST path, responds with
204 No Content, and prints each notification to stdout. Keep this running in a separate terminal.Use a reachable IP-based URI in test environments so periodic notifications do not fail because of DNS resolution issues.
5.3 Create Subscription (Subscribe)
Success Response (201):
{ "subscriptionId": "550e8400-e29b-41d4-a716-446655440000", "subscription": { "..." } }Locationheader:http://127.0.0.8:8090/nupf-ee/v1/ee-subscriptions/550e8400-...5.4 Cancel Subscription (Unsubscribe)
curl -X DELETE http://127.0.0.8:8090/nupf-ee/v1/ee-subscriptions/550e8400-e29b-41d4-a716-446655440000 # HTTP 204 No Content5.5 Modify Subscription (Modify)
5.6 Receive Notification
The subscriber's
eventNotifyUriendpoint will receive POST requests in the following format:{ "notifyCorrelationId": "corr-001", "notificationItems": [ { "eventType": "USER_DATA_USAGE_MEASURES", "sourceUeIpAddress": "10.60.0.1", "timeStamp": "2026-04-28T10:00:00Z", "usageData": [ { "volumeMeasurement": { "totalVolume": 1048576, "ulVolume": 524288, "dlVolume": 524288 } } ] } ] }Subscriber should respond with
204 No Content.When using
receiver.py, each notification is printed to the terminal:5.7 Complete End-to-End Test Workflow
After a UE has established a PDU session (UE IP:
10.60.0.1), the full test flow is:Step 1 — Start the notification receiver (separate terminal)
Step 2 — Create a subscription
Step 3 — Generate traffic through the UE (ping or iperf via the UE interface)
# From UE side or uesimtun0: ping -c 20 8.8.8.8 -I uesimtun0Step 4 — Wait for periodic notifications (every 10 seconds)
The receiver terminal will print volume measurements. Verify that
ulVolume/dlVolumeincrease as traffic flows.Step 5 — Delete the subscription
Notifications stop after deletion.
6. Testing
Test file:
internal/exposure/server_test.go6.1 Test List
TestCreateSubscription_MatchByUeIPAddMonitoringURR, lSeid/repPeriod correctTestCreateSubscription_AnyUeanyUe=true, matches all sessionsAddMonitoringURRcalled twice (for two sessions)TestCreateSubscription_NoMatchAddMonitoringURRTestDeleteSubscriptionRemoveMonitoringURRcalled, lSeid correctTestDeleteSubscription_NotFoundTestHandleReport_ConsumesMonitoringURReventNotifyUri, verifies notification deliveryTestHTTPCreateSubscriptionTestHTTPDeleteSubscription6.2 Run Tests
Sample Output:
6.3 Full Module Test
go vet ./... # Static analysis, no output = pass7. Architecture Summary
7.1 Subscription Creation Flow
sequenceDiagram participant C as NF Service Consumer (NWDAF/SMF) participant E as ExposureServer (HTTP :8090) participant P as PfcpServer (main goroutine) participant K as Kernel / gtp5g C->>E: POST /nupf-ee/v1/ee-subscriptions {ueIpAddress: "10.60.0.1", trigger: PERIODIC, repPeriod: 30} E->>E: Parse request and find matching PDU session by UE IP / anyUe alt Found matching session E->>E: Allocate a monitoring URRID (0xF0000001+) E->>P: AddMonitoringURR(lSeid, urrid, 30s) P->>P: Queue request to monitoringCh P->>K: Update URR and linked PDRs K-->>P: Success P-->>E: Return result E->>E: Save subscription record E-->>C: 201 Created else No matching session E-->>C: 404 Not Found end7.2 Periodic Report and Notification Flow
sequenceDiagram participant K as Kernel / gtp5g (perio timer) participant D as forwarder.Driver participant M as ReportMultiplexer participant E as ExposureServer participant P as PfcpServer participant C as NF Service Consumer K->>D: URR PERIO triggered, report USAReport{URRID=0xF0000001, volume=...} D->>M: NotifySessReport(SessReport{SEID=1, Reports=[USAReport]}) M->>M: Check URRID >= 0xF0000001? alt Monitoring URR (Exposure) M->>E: HandleReport(SessReport with monitoring reports only) E->>E: Find subscription: sessionURRs[SEID] == URRID? E->>E: buildNotificationItem(): fill UE IP, timestamp, volume data E->>C: POST {eventNotifyUri} body: NotificationData C-->>E: 204 No Content else SMF URR (normal PFCP report) M->>P: NotifySessReport(SessReport with SMF reports only) P->>P: ServeReport() normal handling end7.3 Unsubscribe Flow
sequenceDiagram participant C as NF Service Consumer participant E as ExposureServer participant P as PfcpServer participant K as Kernel / gtp5g C->>E: DELETE /nupf-ee/v1/ee-subscriptions/{subID} E->>E: Look up saved {lSeid -> urrid} loop For each session's monitoring URR E->>P: RemoveMonitoringURR(lSeid, urrid) P->>P: Queue request to monitoringCh P->>K: Remove URR and unlink it from PDRs K-->>P: Success P-->>E: Return result end E-->>C: 204 No Content7.4 Architecture Diagram
8. Design Notes
PDRInfo.LastIE, rebuild UpdatePDR IENLM_F_REPLACEflag, does not support delta updateAddMonitoringURRviamonitoringChin PFCP main loop0xF0000001+HandleReportoutside goroutine, HTTP POST is sync but only in report handler goroutine