Skip to content

Commit b6cb5da

Browse files
Oxel40Oxel40
authored
feat(faro.receiver): Add sourcemap fetching from remote locations (#4614)
#### PR Description This adds the ability to fetch sourcemaps from remote locations over HTTP to the `faro.receiver`. Allowing `sourcemaps > location` blocks to point to an HTTP URL as the path, e.g., `path = "https://foo.com/blob/sourcemaps/"`. The motivation behind this feature is to support the use case where your frontend is served by an external CDN and Alloy is running on internal infrastructure, where external resources are not readily accessible, and managing and attaching volumes to the Alloy container is complex or undesired. This feature allows you to host your sourcemaps internally, e.g., on a blob storage service. It also fits the use case where you don't want to expose sourcemaps to customers but still want to be able to fetch them internally for `faro.receiver`'s stack-trace transformations. If multiple location blocks are defined, blocks pointing to on-disk paths will be checked first before attempting to fetch the sourcemaps over HTTP. #### Which issue(s) this PR fixes None. #### Notes to the Reviewer None. #### PR Checklist - [X] CHANGELOG.md updated - [X] Documentation added - [X] Tests updated
1 parent d777ed1 commit b6cb5da

File tree

3 files changed

+315
-7
lines changed

3 files changed

+315
-7
lines changed

‎docs/sources/reference/components/faro/faro.receiver.md‎

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ You can use the following blocks with `faro.receiver`:
6464
| `server` > [`rate_limiting`][rate_limiting] | Configures rate limiting for the HTTP server. | no |
6565
| [`sourcemaps`][sourcemaps] | Configures sourcemap retrieval. | no |
6666
| `sourcemaps` > [`cache`][cache] | Configures sourcemap caching behavior. | no |
67-
| `sourcemaps` > [`location`][location] | Configures on-disk location for sourcemap retrieval. | no |
67+
| `sourcemaps` > [`location`][location] | Configures the location for sourcemap retrieval. | no |
6868

6969
[cache]: #cache
7070
[location]: #location
@@ -168,7 +168,7 @@ The `*` character indicates a wildcard.
168168
By default, sourcemap downloads are subject to a timeout of `"1s"`, specified by the `download_timeout` argument.
169169
Setting `download_timeout` to `"0s"` disables timeouts.
170170

171-
To retrieve sourcemaps from disk instead of the network, specify one or more [`location` blocks][location].
171+
To retrieve sourcemaps from disk or another network location, specify one or more [`location` blocks][location].
172172
When `location` blocks are provided, they're checked first for sourcemaps before falling back to downloading.
173173

174174
#### `cache`
@@ -195,10 +195,10 @@ Set `cleanup_check_interval` to adjust this frequency.
195195
The `location` block declares a location where sourcemaps are stored on the filesystem.
196196
You can specify the `location` block multiple times to declare multiple locations where sourcemaps are stored.
197197

198-
| Name | Type | Description | Default | Required |
199-
|------------------------|----------|-----------------------------------------------------|---------|----------|
200-
| `minified_path_prefix` | `string` | The prefix of the minified path sent from browsers. | | yes |
201-
| `path` | `string` | The path on disk where sourcemaps are stored. | | yes |
198+
| Name | Type | Description | Default | Required |
199+
|------------------------|----------|-----------------------------------------------------------|---------|----------|
200+
| `minified_path_prefix` | `string` | The prefix of the minified path sent from browsers. | | yes |
201+
| `path` | `string` | The path on disk or base URL where sourcemaps are stored. | | yes |
202202

203203
The `minified_path_prefix` argument determines the prefix of paths to JavaScript files, such as `http://example.com/`.
204204
The `path` argument then determines where to find the sourcemap for the file.
@@ -220,6 +220,35 @@ To look up the sourcemaps for a file hosted at `http://example.com/example.js`,
220220
Optionally, the value for the `path` argument may contain `{{ .Release }}` as a template value, such as `/var/my-app/{{ .Release }}/build`.
221221
The template value is replaced with the release value provided by the [Faro Web App SDK][faro-sdk].
222222

223+
When you specify a remote location, the procedure for retrieving the sourcemaps is the same as for a location block with a local path, except that the component retrieves the sourcemap from a remote HTTP server.
224+
225+
In the following example, the `faro.receiver` sends a GET request to `http://storage.example.com/blob/sourcemaps/example.js.map` and retrieves the sourcemap for a file hosted at
226+
`http://example.com/example.js`.
227+
228+
You can specify multiple location blocks. For example:
229+
230+
```alloy
231+
location {
232+
path = "http://storage.example.com/blob/sourcemaps/"
233+
minified_path_prefix = "http://example.com/"
234+
}
235+
236+
```alloy
237+
location {
238+
path = "/var/my-app/build"
239+
minified_path_prefix = "http://example.com/"
240+
}
241+
location {
242+
path = "http://storage.example.com/blob/sourcemaps/"
243+
minified_path_prefix = "http://example.com/"
244+
}
245+
```
246+
247+
The `faro.receiver` component searches through all locations for the sourcemap files.
248+
Local on-disk paths take precedence over remote paths.
249+
For a file hosted at `http://example.com/example.js`, the `faro.receiver` first checks
250+
the path `/var/my-app/build/example.js.map`, and then tries to retrieve `http://storage.example.com/blob/sourcemaps/example.js.map`.
251+
223252
## Exported fields
224253

225254
`faro.receiver` doesn't export any fields.

‎internal/component/faro/receiver/sourcemaps.go‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -312,14 +312,30 @@ func (store *sourceMapsStoreImpl) Stop() {
312312
func (store *sourceMapsStoreImpl) getSourceMapContent(sourceURL string, release string) (content []byte, sourceMapURL string, err error) {
313313
// Attempt to find the source map in the filesystem first.
314314
for _, loc := range store.locs {
315+
if hasHttpPrefix(loc.Path) {
316+
continue
317+
}
318+
315319
content, sourceMapURL, err = store.getSourceMapFromFileSystem(sourceURL, release, loc)
316320
if content != nil || err != nil {
317321
return content, sourceMapURL, err
318322
}
319323
}
320324

325+
// Attempt to find the source map in the remote locations.
326+
for _, loc := range store.locs {
327+
if !(hasHttpPrefix(loc.Path)) {
328+
continue
329+
}
330+
331+
content, sourceMapURL, err = store.getSourceMapFromRemote(sourceURL, release, loc)
332+
if content != nil || err != nil {
333+
return content, sourceMapURL, err
334+
}
335+
}
336+
321337
// Attempt to download the sourcemap if enabled.
322-
if strings.HasPrefix(sourceURL, "http") && urlMatchesOrigins(sourceURL, store.args.DownloadFromOrigins) && store.args.Download {
338+
if store.args.Download && hasHttpPrefix(sourceURL) && urlMatchesOrigins(sourceURL, store.args.DownloadFromOrigins) {
323339
return store.downloadSourceMapContent(sourceURL)
324340
}
325341
return nil, "", nil
@@ -369,6 +385,34 @@ func (store *sourceMapsStoreImpl) getSourceMapFromFileSystem(sourceURL string, r
369385
return content, sourceURL, err
370386
}
371387

388+
func (store *sourceMapsStoreImpl) getSourceMapFromRemote(sourceURL string, release string, loc *sourcemapFileLocation) (content []byte, sourceMapURL string, err error) {
389+
if len(sourceURL) == 0 || !strings.HasPrefix(sourceURL, loc.MinifiedPathPrefix) || strings.HasSuffix(sourceURL, "/") {
390+
return nil, "", nil
391+
}
392+
393+
var rootPath bytes.Buffer
394+
395+
err = loc.pathTemplate.Execute(&rootPath, struct{ Release string }{Release: cleanFilePathPart(release)})
396+
if err != nil {
397+
return nil, "", err
398+
}
399+
400+
subPath := strings.TrimPrefix(strings.Split(sourceURL, "?")[0], loc.MinifiedPathPrefix) + ".map"
401+
mapURL, err := url.JoinPath(rootPath.String(), subPath)
402+
if err != nil {
403+
level.Debug(store.log).Log("msg", "failed to construct sourcemap url for remote location", "base_path", rootPath, "sub_path", subPath, "err", err)
404+
return nil, "", err
405+
}
406+
407+
content, err = store.downloadFileContents(mapURL)
408+
if err != nil {
409+
level.Debug(store.log).Log("msg", "failed to download sourcemap file from remote location", "url", mapURL, "err", err)
410+
return nil, "", err
411+
}
412+
413+
return content, sourceURL, err
414+
}
415+
372416
func (store *sourceMapsStoreImpl) downloadSourceMapContent(sourceURL string) (content []byte, resolvedSourceMapURL string, err error) {
373417
level.Debug(store.log).Log("msg", "attempting to download source file", "url", sourceURL)
374418

@@ -468,6 +512,10 @@ func urlMatchesOrigins(URL string, origins []string) bool {
468512
return false
469513
}
470514

515+
func hasHttpPrefix(URL string) bool {
516+
return strings.HasPrefix(URL, "http://") || strings.HasPrefix(URL, "https://")
517+
}
518+
471519
func cleanFilePathPart(x string) string {
472520
return strings.TrimLeft(strings.ReplaceAll(strings.ReplaceAll(x, "\\", ""), "/", ""), ".")
473521
}

‎internal/component/faro/receiver/sourcemaps_test.go‎

Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -469,6 +469,237 @@ func Test_sourceMapsStoreImpl_ReadFromFileSystemAndNotDownloadIfDisabled(t *test
469469
require.Equal(t, expect, actual)
470470
}
471471

472+
func Test_sourceMapsStoreImpl_ReadFromRemoteLocation(t *testing.T) {
473+
var (
474+
logger = alloyutil.TestLogger(t)
475+
476+
httpClient = &mockHTTPClient{
477+
responses: []struct {
478+
*http.Response
479+
error
480+
}{
481+
{newResponseFromTestData(t, "foo.js.map"), nil},
482+
},
483+
}
484+
485+
fileService = newTestFileService()
486+
)
487+
488+
var store = newSourceMapsStore(
489+
logger,
490+
SourceMapsArguments{
491+
Download: false,
492+
DownloadFromOrigins: []string{"*"},
493+
Locations: []LocationArguments{
494+
{
495+
MinifiedPathPrefix: "http://foo.com/",
496+
Path: "http://baz.com/baz",
497+
},
498+
},
499+
},
500+
newSourceMapMetrics(prometheus.NewRegistry()),
501+
httpClient,
502+
fileService,
503+
)
504+
505+
expect := &payload.Exception{
506+
Stacktrace: &payload.Stacktrace{
507+
Frames: []payload.Frame{
508+
{
509+
Colno: 37,
510+
Filename: "/__parcel_source_root/demo/src/actions.ts",
511+
Function: "?",
512+
Lineno: 6,
513+
},
514+
{
515+
Colno: 5,
516+
Filename: "http://bar.com/foo.js",
517+
Function: "callUndefined",
518+
Lineno: 6,
519+
},
520+
},
521+
},
522+
}
523+
524+
actual := transformException(logger, store, &payload.Exception{
525+
Stacktrace: &payload.Stacktrace{
526+
Frames: []payload.Frame{
527+
{
528+
Colno: 6,
529+
Filename: "http://foo.com/foo.js",
530+
Function: "eval",
531+
Lineno: 5,
532+
},
533+
{
534+
Colno: 5,
535+
Filename: "http://bar.com/foo.js",
536+
Function: "callUndefined",
537+
Lineno: 6,
538+
},
539+
},
540+
},
541+
}, "123")
542+
543+
require.Equal(t, []string{}, fileService.stats)
544+
require.Equal(t, []string{}, fileService.reads)
545+
require.Equal(t, []string{"http://baz.com/baz/foo.js.map"}, httpClient.requests)
546+
require.Equal(t, expect, actual)
547+
}
548+
549+
func Test_sourceMapsStoreImpl_ReadFromFileSystemIfBothLocalAndRemoteLocation(t *testing.T) {
550+
var (
551+
logger = alloyutil.TestLogger(t)
552+
553+
httpClient = &mockHTTPClient{}
554+
555+
fileService = newTestFileService()
556+
)
557+
fileService.files = map[string][]byte{
558+
filepath.FromSlash("/var/build/latest/foo.js.map"): loadTestData(t, "foo.js.map"),
559+
}
560+
561+
var store = newSourceMapsStore(
562+
logger,
563+
SourceMapsArguments{
564+
Download: false,
565+
DownloadFromOrigins: []string{"*"},
566+
Locations: []LocationArguments{
567+
{
568+
MinifiedPathPrefix: "http://foo.com/",
569+
Path: "http://baz.com/baz/",
570+
},
571+
{
572+
MinifiedPathPrefix: "http://foo.com/",
573+
Path: filepath.FromSlash("/var/build/latest/"),
574+
},
575+
},
576+
},
577+
newSourceMapMetrics(prometheus.NewRegistry()),
578+
httpClient,
579+
fileService,
580+
)
581+
582+
expect := &payload.Exception{
583+
Stacktrace: &payload.Stacktrace{
584+
Frames: []payload.Frame{
585+
{
586+
Colno: 37,
587+
Filename: "/__parcel_source_root/demo/src/actions.ts",
588+
Function: "?",
589+
Lineno: 6,
590+
},
591+
{
592+
Colno: 5,
593+
Filename: "http://bar.com/foo.js",
594+
Function: "callUndefined",
595+
Lineno: 6,
596+
},
597+
},
598+
},
599+
}
600+
601+
actual := transformException(logger, store, &payload.Exception{
602+
Stacktrace: &payload.Stacktrace{
603+
Frames: []payload.Frame{
604+
{
605+
Colno: 6,
606+
Filename: "http://foo.com/foo.js",
607+
Function: "eval",
608+
Lineno: 5,
609+
},
610+
{
611+
Colno: 5,
612+
Filename: "http://bar.com/foo.js",
613+
Function: "callUndefined",
614+
Lineno: 6,
615+
},
616+
},
617+
},
618+
}, "123")
619+
620+
require.Equal(t, []string{"/var/build/latest/foo.js.map"}, fileService.stats)
621+
require.Equal(t, []string{"/var/build/latest/foo.js.map"}, fileService.reads)
622+
require.Nil(t, httpClient.requests)
623+
require.Equal(t, expect, actual)
624+
}
625+
626+
func Test_sourceMapsStoreImpl_ReadFromRemoteLocationIfBothDownloadAndLocationIsSet(t *testing.T) {
627+
var (
628+
logger = alloyutil.TestLogger(t)
629+
630+
httpClient = &mockHTTPClient{
631+
responses: []struct {
632+
*http.Response
633+
error
634+
}{
635+
{newResponseFromTestData(t, "foo.js.map"), nil},
636+
},
637+
}
638+
639+
fileService = newTestFileService()
640+
)
641+
642+
var store = newSourceMapsStore(
643+
logger,
644+
SourceMapsArguments{
645+
Download: true,
646+
DownloadFromOrigins: []string{"*"},
647+
Locations: []LocationArguments{
648+
{
649+
MinifiedPathPrefix: "http://foo.com/",
650+
Path: "http://baz.com/baz/",
651+
},
652+
},
653+
},
654+
newSourceMapMetrics(prometheus.NewRegistry()),
655+
httpClient,
656+
fileService,
657+
)
658+
659+
expect := &payload.Exception{
660+
Stacktrace: &payload.Stacktrace{
661+
Frames: []payload.Frame{
662+
{
663+
Colno: 37,
664+
Filename: "/__parcel_source_root/demo/src/actions.ts",
665+
Function: "?",
666+
Lineno: 6,
667+
},
668+
{
669+
Colno: 5,
670+
Filename: "http://bar.com/foo.js",
671+
Function: "callUndefined",
672+
Lineno: 6,
673+
},
674+
},
675+
},
676+
}
677+
678+
actual := transformException(logger, store, &payload.Exception{
679+
Stacktrace: &payload.Stacktrace{
680+
Frames: []payload.Frame{
681+
{
682+
Colno: 6,
683+
Filename: "http://foo.com/foo.js",
684+
Function: "eval",
685+
Lineno: 5,
686+
},
687+
{
688+
Colno: 5,
689+
Filename: "http://bar.com/foo.js",
690+
Function: "callUndefined",
691+
Lineno: 6,
692+
},
693+
},
694+
},
695+
}, "123")
696+
697+
require.Equal(t, []string{}, fileService.stats)
698+
require.Equal(t, []string{}, fileService.reads)
699+
require.Equal(t, []string{"http://baz.com/baz/foo.js.map"}, httpClient.requests)
700+
require.Equal(t, expect, actual)
701+
}
702+
472703
func Test_sourceMapsStoreImpl_FilepathSanitized(t *testing.T) {
473704
var (
474705
logger = alloyutil.TestLogger(t)

0 commit comments

Comments
 (0)