Does codeprep works on JavaScript source code preprocessing?

[Kelin-hao]

It seems that it dosenot work when delt with javascript language. And is there any solution to remove end of a token'\t' in the token sequence.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label question to this issue, with a confidence of 0.94. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[Kelin-hao]

I try to use corpus to preprocess my JavaScript source code as following: import codeprep.api.corpus as cps cps.basic('D:/NpmData/forTest/npm_1/@angular-cdk',no_com=True, no_str= True,extensions="js") But an error ocured: `RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.`

[hlibbabii]

Hi @Wang-JLU! Thanks for creating the issue. It should work for a JS corpus as well. From the error message you are getting, it does not look like it has something to do with the language you want to pre-process. It rather seems to be a multi-processing issue on Windows. Unfortunately, I couldn't reproduce the error. If you still haven't figured out what the problem is, could you please give more details (Windows and python version you are running)?

Regarding token, currently, we are always returning since without it's impossible to reconstruct original tokens. Therefore we don't have any switch/parameter at the moment to change this behavior. So, unfortunately, you have to do this yourself. Could you please tell us your use case, why exactly you need it, so we can consider adding such a switch.