Latest Next.js Issue: tiktoken doesn't work in App Router.

[alfonsusac]

Not sure why but tiktoken doesn't work in the App Router despite already following all the instructions from the Readme.

My workaround is to move it to pages dir and it all works.

When used in AppDir it works during dev and build. But when deployed to Vercel, it produce the following error:

⨯ Error: Missing tiktoken_bg.wasm
    at 1941 (.next/server/app/api/get-context-length/route.js:1:2533)
    at t (.next/server/webpack-runtime.js:1:204)
    at 8566 (.next/server/app/api/get-context-length/route.js:1:9929)
    at t (.next/server/webpack-runtime.js:1:204)
    at n (.next/server/app/api/get-context-length/route.js:1:11581)
    at <unknown> (.next/server/app/api/get-context-length/route.js:1:11612)
    at t.X (.next/server/webpack-runtime.js:1:1332)
    at <unknown> (.next/server/app/api/get-context-length/route.js:1:11594)
    at Object.<anonymous> (.next/server/app/api/get-context-length/route.js:1:11639)
    at Module.<anonymous> (../../opt/rust/bytecode.js:2:1435) {
  page: '/api/get-context-length'
}

And crashes the whole function (not even try catch is catching the error).