Scan Error: Out of Memory #431
Description
When trying to scan my current git repo, I get the following error
Reproduction not possible, because coding is confidential.
I expected the scan to run without issues, as it also runs in other repos in which I develop.
OS: Windows 10 Enterprise
From my perspective the repo is relatively large and there are also jupyter notebooks included.
Full Error Message:
C:\Users\USER\PycharmProjects\Project>talisman --scan
Talisman Fetch Blobs: 254 / 254 <---------------------------------------------------------------------------> 100.00%
d8888b. db db d8b db d8b db d888888b d8b db d888b .d8888. .o88b. .d8b. d8b db
88 8D 88 88 888o 88 888o 88 88' 888o 88 88' Y8b 88' YP d8P Y8 d8' 8b 888o 88 88oobY' 88 88 88V8o 88 88V8o 88 88 88V8o 88 88 8bo. 8P 88ooo88 88V8o 88
888b 88 88 88 V8o88 88 V8o88 88 88 V8o88 88 ooo Y8b. 8b 88~~~88 88 V8o88
88 88. 88b d88 88 V888 88 V888 .88. 88 V888 88. ~8~ db 8D Y8b d8 88 88 88 V888 db db 88 YD ~Y8888P' VP V8P VP V8P Y888888P VP V8P Y888P 8888Y' `Y88P' YP YP VP V8P VP VP
Talisman Scan: 1361 / 2133 <---------------------------------------------------↘.............................> 63.81% runtime: out of memory: cannot allocate 104857600-byte block (1192984576 in use)
fatal error: out of memory
goroutine 949 [running]:
runtime.throw({0xfe3851, 0xd})
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/panic.go:992 +0x64 fp=0x22c9ae6c sp=0x22c9ae58 pc=0xcf6f94
runtime.(*mcache).allocLarge(0x1cd0c18, 0x62b8000, 0x1)
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/mcache.go:215 +0x1e9 fp=0x22c9ae94 sp=0x22c9ae6c pc=0xcd4589
runtime.mallocgc(0x62b8000, 0x0, 0x0)
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/malloc.go:1096 +0x446 fp=0x22c9aedc sp=0x22c9ae94 pc=0xccbc56
runtime.rawbyteslice(0x62b627a)
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/string.go:286 +0xdb fp=0x22c9aef8 sp=0x22c9aedc pc=0xd0fe3b
runtime.stringtoslicebyte(0x0, {0x27f9e000, 0x62b627a})
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/string.go:178 +0x4c fp=0x22c9af18 sp=0x22c9aef8 pc=0xd0f8dc
talisman/detector/filecontent.(*FileContentDetector).Test.func1({{0x12e8c4dd, 0x2c}, {0x12e8c4f2, 0x17}, {0x124061d0, 0x1, 0x1}, {0x14fe0000, 0x62b627a, 0x62b627a}})
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:122 +0x210 fp=0x22c9afc4 sp=0x22c9af18 pc=0xf5d4f0
talisman/detector/filecontent.(*FileContentDetector).Test.func3()
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:132 +0x2e fp=0x22c9aff0 sp=0x22c9afc4 pc=0xf5d2be
runtime.goexit()
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/asm_386.s:1326 +0x1 fp=0x22c9aff4 sp=0x22c9aff0 pc=0xd23791
created by talisman/detector/filecontent.(*FileContentDetector).Test
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:109 +0x717
goroutine 1 [runnable]:
talisman/detector/filecontent.processContent({{0x12d680ff, 0xc}, {0x12d680ff, 0xc}, 0x1, {0x5d667800, 0x668, 0x700}, 0x3}, 0x0, ...)
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:170 +0xa6
talisman/detector/filecontent.(*FileContentDetector).Test(0x125ae240, {{0x12bc2a0, 0x124a93e0}, {0x12bc644, 0x14efcdc}, 0x124cc0a0}, {0x12ba2000, 0x2c7, 0x400}, 0x124cc0a0, ...)
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:154 +0x527
talisman/detector.(*Chain).Test(0x125ae210, {0x12ba2000, 0x2c7, 0x400}, 0x124cc0a0, 0x12cec6c0)
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/chain.go:62 +0x46f
main.(*ScannerCmd).Run(0x125ae030, 0x124cc0a0)
/Users/javiermatias-cabrera/workspace/external-repos/talisman/cmd/scanner_cmd.go:32 +0x102
main.run({0x0, {0x12bbea0, 0x14efcdc}})
/Users/javiermatias-cabrera/workspace/external-repos/talisman/cmd/talisman.go:152 +0x753
main.main()
/Users/javiermatias-cabrera/workspace/external-repos/talisman/cmd/talisman.go:124 +0x2c1
...
goroutine 1396 [semacquire]:
sync.runtime_Semacquire(0x128d87f8)
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/runtime/sema.go:56 +0x36
sync.(*WaitGroup).Wait(0x128d87f0)
/Users/javiermatias-cabrera/.goenv/versions/1.18.10/src/sync/waitgroup.go:136 +0x87
talisman/detector/filecontent.(*FileContentDetector).Test.func2()
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:135 +0x2e
created by talisman/detector/filecontent.(*FileContentDetector).Test
/Users/javiermatias-cabrera/workspace/external-repos/talisman/detector/filecontent/filecontent_detector.go:134 +0x290