Fix repeated token due to wrong text preprocessing

[JulienVig]

Addresses items listed in #654 #656 #657

Fixes #656:

• [x] Fixes text preprocessing, now generation doesn't repeat the last input token and memory leak is fixed
• [x] Add test cases for gpt-tfjs
• [ ] Create test cases for text preprocessing

Addresses parts of #654

• [x] Add explanation comments to gpt-tfjs code
• [x] Refactor classes and code organization
• [x] Add a compile method

Additionally

• Add support for choosing model parameters at initialization (rather than hard coded in the task or the class definition)
• Add support for model config serialization

[JulienVig]

@tharvik do you have an opinion on putting the gpt-tfjs test spec file, which relies on tfjs-node, in the discojs-core/.../gpt? Or better to put it in discojs-node?

[tharvik]

@tharvik do you have an opinion on putting the gpt-tfjs test spec file, which relies on tfjs-node, in the discojs-core/.../gpt? Or better to put it in discojs-node?

for now, there isn't a good place to put theses tests, it's de facto in server/tests. with default tasks splitted from discojs-core (tracked in #647), it'll fit naturally in this "tasks" package

[JulienVig]

for now, there isn't a good place to put theses tests, it's de facto in server/tests. with default tasks splitted from discojs-core (tracked in https://github.com/epfml/disco/issues/647), it'll fit naturally in this "tasks" package

The thing is that this test is independent of the task, I tried to keep it as unitary as possible. It only tests gpt-tfjs implementation and capabilities

[tharvik]

with default tasks splitted from discojs-core (tracked in #647), it'll fit naturally in this "tasks" package

The thing is that this test is independent of the task, I tried to keep it as unitary as possible. It only tests gpt-tfjs implementation and capabilities

ho, I misread, I though it was requiring discojs-node, not tfjs-node; well done then! so yeah, having it in discojs-core makes clearly sense. you can add tfjs-node to dev deps, so that's available for building/testing but not exposed from discojs-core (sadly not enforced).