Can't run the tiktokenizer from the app folder in next.js

[gablabelle]

Hello,

Trying to run tiktokenizer from the app folder and I'm getting the following error:

Error: Element type is invalid. Received a promise that resolves to: [object Promise]. Lazy element type must resolve to a class or function.

Here is the branch to reproduce the error.

[dqbd]

Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC) and NextJS implementation which does not allow importing WASM modules / the WASM package of tiktoken.

One possible solution would be to replace tiktoken with js-tiktoken, if that's needed, but I'm still keen on experimenting with WASM and RSC 🙂

[gablabelle]

Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC) and NextJS implementation which does not allow importing WASM modules / the WASM package of tiktoken.

One possible solution would be to replace tiktoken with js-tiktoken, if that's needed, but I'm still keen on experimenting with WASM and RSC 🙂

I've since migrated to js-tiktoken and it works! 👍🏽