flucoma-sc icon indicating copy to clipboard operation
flucoma-sc copied to clipboard

change the JSON parser to the native one

Open tremblap opened this issue 3 years ago • 8 comments

as disucssed with @weefuzzy we had a pre 3.6 home-made parser. we could implement it as a fall back with try like in FluidSound, or just remove it and replace by the following code which is much faster for large DS to dump to dict for instance (via the tempfile)

(
~parse = {
    var str = File.readAllString("zepath");
    var d = str.parseJSON; 
    d
};
~dic = ~parse.value; 
~dic.postln; 
)

tremblap avatar Oct 05 '22 08:10 tremblap

Looks like it was changed to the handrolled thing in c1f7725d71748d9c3807f0d8ed607adf78f4f31b because the native parsing was enforcing a change of types somewhere undesirable.

weefuzzy avatar Oct 05 '22 08:10 weefuzzy

@g-roma do you happen to remember what parseYAMLFie was doing that was undesirable?

weefuzzy avatar Oct 05 '22 09:10 weefuzzy

I can't remember, but there was some problem.

g-roma avatar Oct 05 '22 09:10 g-roma

The problem seems to be that parseYAML and friends make everything a String, e.g.

d = "{\"foo\":[1.0,2.0,3,4,5]}".parseJSON
d["foo"].do{|x| x.class.postln}
-> Dictionary[ (foo -> [ 1.0, 2.0, 3, 4, 5 ]) ]
String
String
String
String
String

🤦

So, I guess the alternative to what we're doing now is to walk the whole Stringified Dictionary after faster parsing and coerce values back to float where appropriate. That may or may not be faster than the current implementation.

weefuzzy avatar Oct 05 '22 11:10 weefuzzy

see https://github.com/supercollider/supercollider/issues/1154 which has been open since 2014

looks like yaml-cpp doesn't provide a nice way to determine whether a scalar is a string, int or float (see https://stackoverflow.com/questions/19994312/obtain-type-of-value-stored-in-yamlnode-for-yaml-cpp)

weefuzzy avatar Oct 05 '22 11:10 weefuzzy

Indeed @weefuzzy at the moment I can read faster but I cannot use any of it without converting to float the strings... not super useful :) Slow it stays then, until we find a way around this.

tremblap avatar Oct 20 '22 16:10 tremblap

actually, running

i["cols"] = i["cols"].asInteger;

)

(
x = Main.elapsedTime;
i["data"].keysValuesDo{|a,b|i["data"][a] = b.asFloat};
(Main.elapsedTime - x).postln;
)

takes 100% cpu for quite some time. So I don't know what the answer to

That may or may not be faster than the current implementation.

is yet...

tremblap avatar Oct 20 '22 16:10 tremblap

ok our version is faster than parsing the dict to itself - although it suddenly occured to me that maybe I'm making an infinite loop by parsing over itself - another test is needed.

tremblap avatar Oct 20 '22 17:10 tremblap