Fix tokenizer lifecycle close races #251

Job	Run time
go-security / Go Security Checks	24s
Rust Tests	2m 19s
Go Tests (ubuntu-latest, 1.24)	2m 12s
Go Tests (macos-latest, 1.24)	3m 40s
Go Tests (windows-latest, 1.24)	5m 17s
Integration Tests	-1s
	13m 51s

Provide feedback