code2vec Source code parsing pipeline

Hi @sonoisa,

I wasn't able to understand how you arrived at the dataset you provide in your code2vec/data directory. Could you clarify your source code parsing pipeline? If I understand correctly, you seem to have started with the parsed tokens serialized as JSON from http://groups.inf.ed.ac.uk/cup/codeattention/ and you have converted into *.txt in code2vec/data. Am I right?

Would you be able to add the code for doing this into the repo? I need to parse sources written in C which is why I'm seeking a clearer picture of parsing.

Thanks

Jun 08 '20 05:06 dhas

I have the same problem.

Aug 26 '20 02:08 wangyu1997

@wangyu1997 - You may want to take a look at https://github.com/JetBrains-Research/astminer. They have a great parsing pipeline and a small implementation of Code2Vec, which can get you started

Aug 26 '20 06:08 dhas

@dhas Thank you for reply, after i reviewing you code, I notice that the all variables in you terminal_idxs.txt are represented like "@var_xx", could you tell more about the detail? thanks!

Aug 27 '20 05:08 wangyu1997

Hi @dhas, @wangyu1997, I just uploaded a Jupyter Notebook (Scala script) to run the preprocessing.

preprocessing to create Path-Contexts: https://github.com/sonoisa/code2vec/blob/master/create_path_contexts.ipynb

Aug 27 '20 10:08 sonoisa