rascal
rascal copied to clipboard
usethesource
Rascal build has gotten 2x slower
Build:
- https://github.com/usethesource/rascal/actions/runs/1578764861 is still roughly 30 minutes
- https://github.com/usethesource/rascal/actions/runs/1587747510 (switching to rascal-maven-plugin 0.6.1) takes 3h
- Then there is some work to stop some build phases from happening twice
- after https://github.com/usethesource/rascal/actions/runs/1677146649 (switches to rascal-maven-plugin 0.7.1) takes 1h/1:15h
Observations:
- this is all after the move to java 11
- the test are also a bit slower (4min before, now 7m), I've locally run the test between these two commits => not that big of a difference in speed
Hypothesis:
- build machines of github are slower (will validate this by running both builds locally between these two points in time)
- rascal-core has gotten slower since 0.6.0 release of rascal-maven-plugin
build 1578764861 was failing something, now looking at 19c6af6dd2fc7157ebb17b6b999e4e70925daad9 as candidate for last time before it got slow.
Very curious to see! It could also be a change in the interpreter (final hypothesis)
I think we can rule out the github machines getting slower hypothesis:
Running mvn compile on 19c6af6dd2fc7157ebb17b6b999e4e70925daad9 takes 15min, on 5e4a2689ce7d9ca9a2879fe1d7129916150f934b (current main) takes 26min.
I'll now run the old rascal-maven-plugin on main, to see how that performs.
This would evaluate the difference between rascal-maven-plugin version 0.4.5-RC5 and version 0.7.4.
yup, running the 0.4.5-RC4 on current main gives the same speed as before, so somewhere in these commits there has been a performance degradation
https://github.com/usethesource/rascal-maven-plugin/compare/v0.4.5-RC5...v0.7.6
some things I see:
- we went from rascal 0.19.4-RC2 to 0.22.1 https://github.com/usethesource/rascal/compare/v0.19.4-RC2...v0.22.1
- typepal 0.4.11 to 0.7.0 https://github.com/usethesource/typepal/compare/v0.4.11...v0.7.0
- rascal-core 0.4.18-RC2 to 0.7.5 https://github.com/usethesource/rascal-core/compare/v0.4.18-RC2...v0.7.5
there are also some small changes to the build path setup, but those seem small.
Comparing the output, they have the same files as "stale items".
nice. looks like you are close to the culprit!
- vallang also changed quite a bit in this range of rascal versions
- typepal and rascal-core were refactored significantly to avoid cyclic extends/imports, as far as I remember, which was also necessary because changes in the interpreter were triggering latent issues in typepal and rascal-core
I suspect that comparing two type-checker runs on a big module with some imports with the sampling profiler of the interpreter on would give the most insight at this time.
It's actually a bit worse,
I disabled the parallel mode:
- 0.4.5: 21min
- 0.7.6: 53min
What's the best way to run it on a big module? load a REPL with the specific jars of rascal core and typepal and rascal on the classpath?
Another observation: we are in the middle of a bootstrap and TypePal now has to convert all tpl files every time.
I just ran with 0.7.3 version, which is 2 months old, and that one is also slow. so that's good news, smaller diff.
- https://github.com/usethesource/rascal-maven-plugin/compare/v0.4.5-RC5...v0.7.3
- https://github.com/usethesource/rascal/compare/v0.19.4-RC2...v0.22.0 (0.22.0 instead of 0.22.1)
- https://github.com/usethesource/typepal/compare/v0.4.11...v0.7.0
- https://github.com/usethesource/rascal-core/compare/v0.4.18-RC2...v0.7.0 (so 0.7.0 instead of 0.7.5)
The bootstrap is the last release of Rascal-core and typepal the current head of main on both projects.
The best way is to start a repl in the Rascal-core project using mvn rascal:console with the right pom.xml. if you have to go back beyond the existence of that feature, than start a Rascal.jar with classpath printed using the maven dependencies plugin and RascalShell as the main class.
I ran the rascal-core checker on prelude & parser generator.
Timings for the rascal-core v0.7.5 (aka HEAD): (1096s)
FRAMES PROFILE: 6436 data points, 1096896 ticks, tick = 1 milliSecs
Scope Ticks % Source
newSolver 438127 39.9% |lib://typepal/src/analysis/typepal/Solver.rsc|(454,71160,<25,0>,<1813,1>)
isSameFile 154417 14.1% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
checkOverloading 118968 10.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(12791,3901,<300,0>,<368,1>)
isStrictlyContainedIn 67521 6.2% |lib://rascal/Location.rsc|(1675,790,<65,0>,<89,1>)
newScopeGraph 57986 5.3% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(6708,13747,<178,0>,<499,1>)
isOverloadedFunction 40646 3.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9112,364,<224,0>,<232,1>)
addGrammar 38465 3.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(4891,5533,<108,0>,<226,1>)
newCollector 32040 2.9% |lib://typepal/src/analysis/typepal/Collector.rsc|(8275,36570,<190,0>,<1043,1>)
rascalIsAcceptableSimple 14389 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(1916,1276,<64,0>,<87,1>)
saveModule 14006 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(12491,6048,<269,0>,<382,1>)
ASTS PROFILE: 6436 data points, 1096896 ticks, tick = 1 milliSecs
Scope Ticks % Source
getType 222930 20.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(22117,7,<640,41>,<640,48>)
getType 80779 7.4% |lib://typepal/src/analysis/typepal/Solver.rsc|(21694,1,<634,19>,<634,20>)
isSameFile 67097 6.1% |lib://rascal/Location.rsc|(1068,1,<39,15>,<39,16>)
isSameFile 52741 4.8% |lib://rascal/Location.rsc|(1001,1,<37,15>,<37,16>)
checkOverloading 27273 2.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16071,2,<356,116>,<356,118>)
checkOverloading 27158 2.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16061,2,<356,106>,<356,108>)
getType 25330 2.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(22125,3,<640,49>,<640,52>)
_addTModel 22294 2.0% |lib://typepal/src/analysis/typepal/Collector.rsc|(39859,2,<915,19>,<915,21>)
isOverloadedFunction 21944 2.0% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9346,1,<228,66>,<228,67>)
checkOverloading 20127 1.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16056,2,<356,101>,<356,103>)
checkOverloading 19361 1.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16035,2,<356,80>,<356,82>)
getType 19228 1.8% |lib://typepal/src/analysis/typepal/Solver.rsc|(22004,5,<639,68>,<639,73>)
isStrictlyContainedIn 18449 1.7% |lib://rascal/Location.rsc|(1675,790,<65,0>,<89,1>)
getType 16816 1.5% |lib://typepal/src/analysis/typepal/Solver.rsc|(21500,1232,<624,4>,<652,5>)
isSameFile 15185 1.4% |lib://rascal/Location.rsc|(1082,1,<39,29>,<39,30>)
lookupPathsWide 14633 1.3% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(14688,5,<362,49>,<362,54>)
isSameFile 14074 1.3% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
_run 11026 1.0% |lib://typepal/src/analysis/typepal/Solver.rsc|(68117,7,<1738,61>,<1738,68>)
rascal-core 0.7.0: (955s)
FRAMES PROFILE: 6030 data points, 955304 ticks, tick = 1 milliSecs
Scope Ticks % Source
newSolver 303723 31.8% |lib://typepal/src/analysis/typepal/Solver.rsc|(454,71035,<25,0>,<1813,1>)
isSameFile 173380 18.1% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
checkOverloading 118763 12.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(12791,3901,<300,0>,<368,1>)
isStrictlyContainedIn 116239 12.2% |lib://rascal/Location.rsc|(1675,790,<65,0>,<89,1>)
newScopeGraph 49841 5.2% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(6706,13747,<178,0>,<499,1>)
addGrammar 42300 4.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(4891,5533,<108,0>,<226,1>)
isOverloadedFunction 33442 3.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9112,364,<224,0>,<232,1>)
saveModule 11057 1.2% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(12479,6041,<269,0>,<382,1>)
rascalIsAcceptableSimple 10278 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(1916,1276,<64,0>,<87,1>)
ASTS PROFILE: 6030 data points, 955304 ticks, tick = 1 milliSecs
Scope Ticks % Source
getType 146040 15.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(22069,7,<640,41>,<640,48>)
isSameFile 65459 6.9% |lib://rascal/Location.rsc|(1001,1,<37,15>,<37,16>)
isSameFile 52147 5.5% |lib://rascal/Location.rsc|(1068,1,<39,15>,<39,16>)
getType 44696 4.7% |lib://typepal/src/analysis/typepal/Solver.rsc|(21646,1,<634,19>,<634,20>)
checkOverloading 30153 3.2% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16061,2,<356,106>,<356,108>)
checkOverloading 28941 3.0% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16071,2,<356,116>,<356,118>)
isStrictlyContainedIn 24851 2.6% |lib://rascal/Location.rsc|(1675,790,<65,0>,<89,1>)
isSameFile 24359 2.5% |lib://rascal/Location.rsc|(1082,1,<39,29>,<39,30>)
isStrictlyContainedIn 21800 2.3% |lib://rascal/Location.rsc|(2250,5,<82,54>,<82,59>)
isSameFile 21579 2.3% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
getType 20238 2.1% |lib://typepal/src/analysis/typepal/Solver.rsc|(22077,3,<640,49>,<640,52>)
addGrammar 19046 2.0% |lib://rascal/Location.rsc|(2312,5,<82,116>,<82,121>)
checkOverloading 18334 1.9% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16056,2,<356,101>,<356,103>)
checkOverloading 18171 1.9% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16035,2,<356,80>,<356,82>)
isOverloadedFunction 16918 1.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9346,1,<228,66>,<228,67>)
addGrammar 15995 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(5503,1,<118,92>,<118,93>)
getType 15881 1.7% |lib://typepal/src/analysis/typepal/Solver.rsc|(21956,5,<639,68>,<639,73>)
lookupPathsWide 14822 1.6% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(14686,5,<362,49>,<362,54>)
getType 13694 1.4% |lib://typepal/src/analysis/typepal/Solver.rsc|(21452,1232,<624,4>,<652,5>)
_run 10141 1.1% |lib://typepal/src/analysis/typepal/Solver.rsc|(67992,7,<1738,61>,<1738,68>)
getType 9643 1.0% |lib://typepal/src/analysis/typepal/Solver.rsc|(21946,1,<639,58>,<639,59>)
list[ModuleMessages]: [
and for the v0.4.18-RC2 version or rascal-core: (252s)
FRAMES PROFILE: 5670 data points, 252585 ticks, tick = 1 milliSecs
Scope Ticks % Source
newScopeGraph 48101 19.0% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(6703,13747,<178,0>,<499,1>)
newSolver 47488 18.8% |lib://typepal/src/analysis/typepal/Solver.rsc|(454,69652,<25,0>,<1782,1>)
isOverloadedFunction 37211 14.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9947,364,<243,0>,<251,1>)
saveModule 13499 5.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(10879,5458,<236,0>,<335,1>)
rascalIsAcceptableSimple 10025 4.0% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(2174,1276,<70,0>,<93,1>)
addADTs 7898 3.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(518,187,<24,0>,<28,1>)
newCollector 7498 3.0% |lib://typepal/src/analysis/typepal/Collector.rsc|(8291,35339,<188,0>,<1014,1>)
rascalReportUnused 6174 2.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(10313,2862,<253,0>,<309,1>)
isContainedIn 2778 1.1% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
ASTS PROFILE: 5670 data points, 252585 ticks, tick = 1 milliSecs
Scope Ticks % Source
isOverloadedFunction 18402 7.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(10181,1,<247,66>,<247,67>)
lookupPathsWide 14723 5.8% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(14683,5,<362,49>,<362,54>)
isOverloadedFunction 9552 3.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(10195,5,<247,80>,<247,85>)
saveModule 8355 3.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(14949,2,<307,19>,<307,21>)
_run 7782 3.1% |lib://typepal/src/analysis/typepal/Solver.rsc|(66609,7,<1707,61>,<1707,68>)
addADTs 4177 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(585,1,<26,17>,<26,18>)
addADTs 3687 1.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(666,1,<26,98>,<26,99>)
bindWide 3666 1.5% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(13720,2,<340,19>,<340,21>)
rascalIsAcceptableSimple 3186 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(3432,15,<92,12>,<92,27>)
isOverloadedFunction 3110 1.2% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(10150,3,<247,35>,<247,38>)
instantiate 2997 1.2% |lib://typepal/src/analysis/typepal/Solver.rsc|(47845,5,<1258,16>,<1258,21>)
isOverloadedFunction 2863 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(10185,3,<247,70>,<247,73>)
saveModule 2781 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(14979,1,<307,49>,<307,50>)
rascalIsAcceptableSimple 2745 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(3324,17,<89,19>,<89,36>)
Observations
- the rascal-core in v0.4.18-RC2 is between 3x and 4x as fast as the latest (v0.4.18 is from March 2021)
-
TypePal::newSolverof typepal takes relatively more time in the newer versions. -
Location::isSameFile&Location::isStrictlyContainedInpops up a lot in the new trace -
RascalCore::checkOverloadingis a new top-scoring function for the rascal-core versions -
TypePal::getTypeis a new top-scoring function
I need the insights of @PaulKlint to clarify these.
Current hypothesis
- [ ] H1: The changes to postSolve phase have had some unexpected side-effects (these two commits: https://github.com/usethesource/typepal/commit/c0ed363e740330450825876cff42e90f98efb915 & https://github.com/usethesource/typepal/commit/bd7a81fae77341f227dbb70f37f3b6e80489d351 )
- [ ] H2: the new rascal-core misses some opportunities for using existing tpls?
- [ ] H3: some change in rascal-core has slowed down the solver.
I just ran the latest rascal-core with the old typepal & rascal & vallang
rascal-core 0.7.0 with dependencies the same as 0.4.18-RC2: (1143s)
FRAMES PROFILE: 6046 data points, 1134633 ticks, tick = 1 milliSecs
Scope Ticks % Source
newSolver 355000 31.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(454,69652,<25,0>,<1782,1>)
isSameFile 184378 16.3% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
checkOverloading 122056 10.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(12791,3901,<300,0>,<368,1>)
isStrictlyContainedIn 105634 9.3% |lib://rascal/Location.rsc|(1675,790,<65,0>,<89,1>)
newScopeGraph 48506 4.3% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(6703,13747,<178,0>,<499,1>)
collect 46366 4.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/CollectSyntaxDeclaration.rsc|(5443,2427,<128,0>,<168,1>)
addGrammar 40430 3.6% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(4891,5533,<108,0>,<226,1>)
isOverloadedFunction 38139 3.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9112,364,<224,0>,<232,1>)
isConcreteSyntaxRole 14971 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/AType.rsc|(4694,121,<166,0>,<166,121>)
saveModule 13219 1.2% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(12479,6041,<269,0>,<382,1>)
ASTS PROFILE: 6046 data points, 1134633 ticks, tick = 1 milliSecs
Scope Ticks % Source
getType 138530 12.2% |lib://typepal/src/analysis/typepal/Solver.rsc|(22069,7,<640,41>,<640,48>)
isSameFile 72121 6.4% |lib://rascal/Location.rsc|(1001,1,<37,15>,<37,16>)
isSameFile 62907 5.5% |lib://rascal/Location.rsc|(1068,1,<39,15>,<39,16>)
getType 47444 4.2% |lib://typepal/src/analysis/typepal/Solver.rsc|(21646,1,<634,19>,<634,20>)
checkOverloading 31006 2.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16061,2,<356,106>,<356,108>)
checkOverloading 28525 2.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16071,2,<356,116>,<356,118>)
isStrictlyContainedIn 22739 2.0% |lib://rascal/Location.rsc|(1675,790,<65,0>,<89,1>)
getType 22231 2.0% |lib://typepal/src/analysis/typepal/Solver.rsc|(22077,3,<640,49>,<640,52>)
isSameFile 21302 1.9% |lib://rascal/Location.rsc|(1082,1,<39,29>,<39,30>)
checkOverloading 19277 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16056,2,<356,101>,<356,103>)
isSameFile 19180 1.7% |lib://rascal/Location.rsc|(885,230,<32,0>,<40,5>)
checkOverloading 19005 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16035,2,<356,80>,<356,82>)
isOverloadedFunction 18973 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9346,1,<228,66>,<228,67>)
instantiate 18817 1.7% |lib://typepal/src/analysis/typepal/Solver.rsc|(47845,5,<1258,16>,<1258,21>)
isStrictlyContainedIn 18532 1.6% |lib://rascal/Location.rsc|(2250,5,<82,54>,<82,59>)
isStrictlyContainedIn 17416 1.5% |lib://rascal/Location.rsc|(2312,5,<82,116>,<82,121>)
getType 15730 1.4% |lib://typepal/src/analysis/typepal/Solver.rsc|(21452,1232,<624,4>,<652,5>)
addGrammar 14868 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(5503,1,<118,92>,<118,93>)
getType 14655 1.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(21956,5,<639,68>,<639,73>)
lookupPathsWide 14334 1.3% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(14683,5,<362,49>,<362,54>)
I think that rules out the influence of changes to vallang/rascal/typepal. So the reason for the loss of performance is inside the last year of commits in rascal-core.
Rascal-core has gotten a lot better, so that might be the price of that, but at least we can rule out any unforseen infrastructure changes.
@PaulKlint if I were a betting person, I would say that this commit: https://github.com/usethesource/rascal-core/commit/3cba191d66bdd0712ff5789564c37d26cacad174#diff-ca186182fdf269848d2318c2c9ba8ddc3a19cff571c3eb88d735253798e83e91R115 is one if the expensive additions. It's calculating the product of all the productions and checking which is contained in which one.
if I disable that filter (replacing the line with prodLocs2 = prodLocs1;) the speed jumps from 1143s to 845s.
FRAMES PROFILE: 6009 data points, 845492 ticks, tick = 1 milliSecs
Scope Ticks % Source
newSolver 369922 43.8% |lib://typepal/src/analysis/typepal/Solver.rsc|(454,69652,<25,0>,<1782,1>)
checkOverloading 119323 14.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(12791,3901,<300,0>,<368,1>)
newScopeGraph 49586 5.9% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(6703,13747,<178,0>,<499,1>)
collect 48119 5.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/CollectSyntaxDeclaration.rsc|(5443,2427,<128,0>,<168,1>)
isOverloadedFunction 38392 4.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9112,364,<224,0>,<232,1>)
addGrammar 21831 2.6% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(4904,5570,<109,0>,<228,1>)
isConcreteSyntaxRole 14787 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/AType.rsc|(4694,121,<166,0>,<166,121>)
saveModule 12908 1.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(12479,6041,<269,0>,<382,1>)
isNonTerminalType 11373 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ATypeUtils.rsc|(54527,83,<1269,0>,<1269,83>)
rascalIsAcceptableSimple 10024 1.2% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(1916,1276,<64,0>,<87,1>)
getLoc 9777 1.2% |lib://typepal/src/analysis/typepal/GetLoc.rsc|(545,248,<28,0>,<33,19>)
getLastLoc 8600 1.0% |lib://typepal/src/analysis/typepal/GetLoc.rsc|(359,184,<19,0>,<26,1>)
ASTS PROFILE: 6009 data points, 845492 ticks, tick = 1 milliSecs
Scope Ticks % Source
getType 154680 18.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(22069,7,<640,41>,<640,48>)
getType 44331 5.2% |lib://typepal/src/analysis/typepal/Solver.rsc|(21646,1,<634,19>,<634,20>)
checkOverloading 27589 3.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16071,2,<356,116>,<356,118>)
checkOverloading 27578 3.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16061,2,<356,106>,<356,108>)
getType 20799 2.5% |lib://typepal/src/analysis/typepal/Solver.rsc|(22077,3,<640,49>,<640,52>)
checkOverloading 20613 2.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16035,2,<356,80>,<356,82>)
checkOverloading 20126 2.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16056,2,<356,101>,<356,103>)
isOverloadedFunction 19531 2.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9346,1,<228,66>,<228,67>)
instantiate 19348 2.3% |lib://typepal/src/analysis/typepal/Solver.rsc|(47845,5,<1258,16>,<1258,21>)
getType 16447 1.9% |lib://typepal/src/analysis/typepal/Solver.rsc|(21452,1232,<624,4>,<652,5>)
getType 16110 1.9% |lib://typepal/src/analysis/typepal/Solver.rsc|(21956,5,<639,68>,<639,73>)
lookupPathsWide 14907 1.8% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(14683,5,<362,49>,<362,54>)
_run 10403 1.2% |lib://typepal/src/analysis/typepal/Solver.rsc|(66609,7,<1707,61>,<1707,68>)
Anonymous Function 9655 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/CollectSyntaxDeclaration.rsc|(7643,2,<161,46>,<161,48>)
getType 9559 1.1% |lib://typepal/src/analysis/typepal/Solver.rsc|(21946,1,<639,58>,<639,59>)
isOverloadedFunction 8994 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9360,5,<228,80>,<228,85>)
checkOverloading 8991 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16046,1,<356,91>,<356,92>)
not this is obviously not correct anymore, but it has quite an influence, and we now have other stuff in the profiler again. it still not back to the old 252s level, but there might be some more "low-hanging" fruit.
H2 is also very interesting. In the standard output it is easy to see if there was a difference in reuse of the .tpl files in the standard library modules.
It seems to me that the newest .tpl files have not been released yet in the rascal.jar, also if you run with an old rascal.jar and a new rascal-core, the .tpl files will always mismatch.
If the type-checker is simply used more because we do not reuse as many .tpl files, then any optimization we come up with will have a larger effect than before. but also it hides the real issue. The suggestion would be to first eliminate this possibility before hunting for other optimization options.
I've run the checker in a way that initially all the tpl files are invalidated (I pointed it at the rascal source directory, where I ran touch on all the files).
And looking at the output, I don't see any missing files "re-done".
output of the old typechecker:
--- lang::rascal::grammar::ConcreteSyntax is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::Lookahead is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- List is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Symbols is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Priorities is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Literals is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Exception is no longer valid (latest $2022-03-31T11:04:54.647+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- ParseTree is no longer valid (latest $2022-03-31T11:04:54.647+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- String is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Productions is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- util::Monitor is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Type is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Regular is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::ParserGenerator is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::syntax::Rascal is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Modules is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Grammar is no longer valid (latest $2022-03-31T11:04:54.647+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Message is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Set is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Node is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Parameters is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Keywords is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::References is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::format::Grammar is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- util::Maybe is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- IO is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- util::Math is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Map is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Characters is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- ValueIO is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::format::Escape is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- analysis::grammars::Dependency is no longer valid (latest $2022-03-31T11:04:54.647+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Relation is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- analysis::graphs::Graph is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Attributes is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Names is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- lang::rascal::grammar::definition::Layout is no longer valid (latest $2022-03-31T11:04:54.650+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- DateTime is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Boolean is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Traversal is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- ListRelation is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
--- Prelude is no longer valid (latest $2022-03-31T11:04:54.648+00:00$, previous check used $2021-06-15T10:27:06.000+00:00$)
output of the new (and slower typechecker)
pathConfig(
javaCompilerPath=[],
bin=|tmp:///x11/|,
classloaders=[],
srcs=[|file:///export/scratch1/landman/rascal-stuff/rascal/src/org/rascalmpl/library/|])
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/ParserGenerator.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/ParserGenerator.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Priorities.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Priorities.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Set.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Set.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/util/Math.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/util/Math.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Exception.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Exception.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/List.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/List.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/IO.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/IO.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Map.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Map.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Grammar.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Grammar.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/ParseTree.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/ParseTree.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Message.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Message.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Type.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Type.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/References.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/References.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Symbols.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Symbols.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/syntax/Rascal.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/syntax/Rascal.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Literals.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Literals.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/String.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/String.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Characters.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Characters.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/util/Maybe.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/util/Maybe.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/format/Grammar.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/format/Grammar.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/analysis/graphs/Graph.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/analysis/graphs/Graph.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Relation.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Relation.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/format/Escape.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/format/Escape.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/analysis/grammars/Dependency.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/analysis/grammars/Dependency.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/ValueIO.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/ValueIO.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Productions.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Productions.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Names.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Names.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Attributes.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Attributes.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/Lookahead.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/Lookahead.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Regular.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Regular.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Modules.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Modules.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Layout.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Layout.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/ConcreteSyntax.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/ConcreteSyntax.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Node.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Node.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Keywords.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Keywords.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/lang/rascal/grammar/definition/Parameters.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/lang/rascal/grammar/definition/Parameters.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/util/Monitor.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/util/Monitor.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Prelude.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Prelude.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Boolean.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Boolean.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/DateTime.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/DateTime.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/ListRelation.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/ListRelation.tpl|
getTPLReadLoc: DOES NOT EXIST: |tmp:///x11/rascal/Traversal.tpl|
getTPLReadLoc: DOES NOT EXIST: |lib://rascal/rascal/Traversal.tpl|
getTPLWriteLoc: Exception => <false, |tmp:///x11/rascal/Exception.tpl|>
saveModule(Exception) => |tmp:///x11/rascal/Exception.tpl|
Written: |tmp:///x11/rascal/Exception.tpl|
getTPLWriteLoc: IO => <false, |tmp:///x11/rascal/IO.tpl|>
saveModule(IO) => |tmp:///x11/rascal/IO.tpl|
Written: |tmp:///x11/rascal/IO.tpl|
getTPLWriteLoc: Map => <false, |tmp:///x11/rascal/Map.tpl|>
saveModule(Map) => |tmp:///x11/rascal/Map.tpl|
Written: |tmp:///x11/rascal/Map.tpl|
getTPLWriteLoc: List => <false, |tmp:///x11/rascal/List.tpl|>
saveModule(List) => |tmp:///x11/rascal/List.tpl|
Written: |tmp:///x11/rascal/List.tpl|
getTPLWriteLoc: util::Math => <false, |tmp:///x11/rascal/util/Math.tpl|>
saveModule(util::Math) => |tmp:///x11/rascal/util/Math.tpl|
Written: |tmp:///x11/rascal/util/Math.tpl|
getTPLWriteLoc: Set => <false, |tmp:///x11/rascal/Set.tpl|>
saveModule(Set) => |tmp:///x11/rascal/Set.tpl|
Written: |tmp:///x11/rascal/Set.tpl|
getTPLWriteLoc: Message => <false, |tmp:///x11/rascal/Message.tpl|>
saveModule(Message) => |tmp:///x11/rascal/Message.tpl|
Written: |tmp:///x11/rascal/Message.tpl|
getTPLWriteLoc: Type => <false, |tmp:///x11/rascal/Type.tpl|>
saveModule(Type) => |tmp:///x11/rascal/Type.tpl|
Written: |tmp:///x11/rascal/Type.tpl|
getTPLWriteLoc: ParseTree => <false, |tmp:///x11/rascal/ParseTree.tpl|>
saveModule(ParseTree) => |tmp:///x11/rascal/ParseTree.tpl|
Written: |tmp:///x11/rascal/ParseTree.tpl|
getTPLWriteLoc: lang::rascal::syntax::Rascal => <false, |tmp:///x11/rascal/lang/rascal/syntax/Rascal.tpl|>
saveModule(lang::rascal::syntax::Rascal) => |tmp:///x11/rascal/lang/rascal/syntax/Rascal.tpl|
Written: |tmp:///x11/rascal/lang/rascal/syntax/Rascal.tpl|
getTPLWriteLoc: Grammar => <false, |tmp:///x11/rascal/Grammar.tpl|>
saveModule(Grammar) => |tmp:///x11/rascal/Grammar.tpl|
Written: |tmp:///x11/rascal/Grammar.tpl|
getTPLWriteLoc: String => <false, |tmp:///x11/rascal/String.tpl|>
saveModule(String) => |tmp:///x11/rascal/String.tpl|
Written: |tmp:///x11/rascal/String.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Literals => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Literals.tpl|>
saveModule(lang::rascal::grammar::definition::Literals) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Literals.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Literals.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Characters => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Characters.tpl|>
saveModule(lang::rascal::grammar::definition::Characters) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Characters.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Characters.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Symbols => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Symbols.tpl|>
saveModule(lang::rascal::grammar::definition::Symbols) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Symbols.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Symbols.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::References => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/References.tpl|>
saveModule(lang::rascal::grammar::definition::References) => |tmp:///x11/rascal/lang/rascal/grammar/definition/References.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/References.tpl|
getTPLWriteLoc: util::Maybe => <false, |tmp:///x11/rascal/util/Maybe.tpl|>
saveModule(util::Maybe) => |tmp:///x11/rascal/util/Maybe.tpl|
Written: |tmp:///x11/rascal/util/Maybe.tpl|
getTPLWriteLoc: Relation => <false, |tmp:///x11/rascal/Relation.tpl|>
saveModule(Relation) => |tmp:///x11/rascal/Relation.tpl|
Written: |tmp:///x11/rascal/Relation.tpl|
getTPLWriteLoc: analysis::graphs::Graph => <false, |tmp:///x11/rascal/analysis/graphs/Graph.tpl|>
saveModule(analysis::graphs::Graph) => |tmp:///x11/rascal/analysis/graphs/Graph.tpl|
Written: |tmp:///x11/rascal/analysis/graphs/Graph.tpl|
getTPLWriteLoc: lang::rascal::format::Escape => <false, |tmp:///x11/rascal/lang/rascal/format/Escape.tpl|>
saveModule(lang::rascal::format::Escape) => |tmp:///x11/rascal/lang/rascal/format/Escape.tpl|
Written: |tmp:///x11/rascal/lang/rascal/format/Escape.tpl|
getTPLWriteLoc: analysis::grammars::Dependency => <false, |tmp:///x11/rascal/analysis/grammars/Dependency.tpl|>
saveModule(analysis::grammars::Dependency) => |tmp:///x11/rascal/analysis/grammars/Dependency.tpl|
Written: |tmp:///x11/rascal/analysis/grammars/Dependency.tpl|
getTPLWriteLoc: ValueIO => <false, |tmp:///x11/rascal/ValueIO.tpl|>
saveModule(ValueIO) => |tmp:///x11/rascal/ValueIO.tpl|
Written: |tmp:///x11/rascal/ValueIO.tpl|
getTPLWriteLoc: lang::rascal::format::Grammar => <false, |tmp:///x11/rascal/lang/rascal/format/Grammar.tpl|>
saveModule(lang::rascal::format::Grammar) => |tmp:///x11/rascal/lang/rascal/format/Grammar.tpl|
Written: |tmp:///x11/rascal/lang/rascal/format/Grammar.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Names => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Names.tpl|>
saveModule(lang::rascal::grammar::definition::Names) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Names.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Names.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Attributes => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Attributes.tpl|>
saveModule(lang::rascal::grammar::definition::Attributes) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Attributes.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Attributes.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Productions => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Productions.tpl|>
saveModule(lang::rascal::grammar::definition::Productions) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Productions.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Productions.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Priorities => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Priorities.tpl|>
saveModule(lang::rascal::grammar::definition::Priorities) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Priorities.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Priorities.tpl|
getTPLWriteLoc: Boolean => <false, |tmp:///x11/rascal/Boolean.tpl|>
saveModule(Boolean) => |tmp:///x11/rascal/Boolean.tpl|
Written: |tmp:///x11/rascal/Boolean.tpl|
getTPLWriteLoc: DateTime => <false, |tmp:///x11/rascal/DateTime.tpl|>
saveModule(DateTime) => |tmp:///x11/rascal/DateTime.tpl|
Written: |tmp:///x11/rascal/DateTime.tpl|
getTPLWriteLoc: Node => <false, |tmp:///x11/rascal/Node.tpl|>
saveModule(Node) => |tmp:///x11/rascal/Node.tpl|
Written: |tmp:///x11/rascal/Node.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Layout => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Layout.tpl|>
saveModule(lang::rascal::grammar::definition::Layout) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Layout.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Layout.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Modules => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Modules.tpl|>
saveModule(lang::rascal::grammar::definition::Modules) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Modules.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Modules.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Regular => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Regular.tpl|>
saveModule(lang::rascal::grammar::definition::Regular) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Regular.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Regular.tpl|
getTPLWriteLoc: lang::rascal::grammar::Lookahead => <false, |tmp:///x11/rascal/lang/rascal/grammar/Lookahead.tpl|>
saveModule(lang::rascal::grammar::Lookahead) => |tmp:///x11/rascal/lang/rascal/grammar/Lookahead.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/Lookahead.tpl|
getTPLWriteLoc: lang::rascal::grammar::ConcreteSyntax => <false, |tmp:///x11/rascal/lang/rascal/grammar/ConcreteSyntax.tpl|>
saveModule(lang::rascal::grammar::ConcreteSyntax) => |tmp:///x11/rascal/lang/rascal/grammar/ConcreteSyntax.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/ConcreteSyntax.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Keywords => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Keywords.tpl|>
saveModule(lang::rascal::grammar::definition::Keywords) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Keywords.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Keywords.tpl|
getTPLWriteLoc: lang::rascal::grammar::definition::Parameters => <false, |tmp:///x11/rascal/lang/rascal/grammar/definition/Parameters.tpl|>
saveModule(lang::rascal::grammar::definition::Parameters) => |tmp:///x11/rascal/lang/rascal/grammar/definition/Parameters.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/definition/Parameters.tpl|
getTPLWriteLoc: util::Monitor => <false, |tmp:///x11/rascal/util/Monitor.tpl|>
saveModule(util::Monitor) => |tmp:///x11/rascal/util/Monitor.tpl|
Written: |tmp:///x11/rascal/util/Monitor.tpl|
getTPLWriteLoc: lang::rascal::grammar::ParserGenerator => <false, |tmp:///x11/rascal/lang/rascal/grammar/ParserGenerator.tpl|>
saveModule(lang::rascal::grammar::ParserGenerator) => |tmp:///x11/rascal/lang/rascal/grammar/ParserGenerator.tpl|
Written: |tmp:///x11/rascal/lang/rascal/grammar/ParserGenerator.tpl|
getTPLWriteLoc: ListRelation => <false, |tmp:///x11/rascal/ListRelation.tpl|>
saveModule(ListRelation) => |tmp:///x11/rascal/ListRelation.tpl|
Written: |tmp:///x11/rascal/ListRelation.tpl|
getTPLWriteLoc: Traversal => <false, |tmp:///x11/rascal/Traversal.tpl|>
saveModule(Traversal) => |tmp:///x11/rascal/Traversal.tpl|
Written: |tmp:///x11/rascal/Traversal.tpl|
getTPLWriteLoc: Prelude => <false, |tmp:///x11/rascal/Prelude.tpl|>
saveModule(Prelude) => |tmp:///x11/rascal/Prelude.tpl|
Written: |tmp:///x11/rascal/Prelude.tpl|
I have some other things to focus on, I just wanted to finish the analysis.
Last thing to note, a lot of time is spent on the generators in this loop in checkOverloading: https://github.com/usethesource/rascal-core/blob/master/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc#L356-L357=
that loop does a lot of redudant calls to getType if I build a map outside of it, to store the types of all of them. the profile & performance changes for the better again (from 845s to 358s).
FRAMES PROFILE: 5817 data points, 358459 ticks, tick = 1 milliSecs
Scope Ticks % Source
checkOverloading 79838 22.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(12791,3943,<300,0>,<369,1>)
newSolver 49026 13.7% |lib://typepal/src/analysis/typepal/Solver.rsc|(454,69652,<25,0>,<1782,1>)
newScopeGraph 48361 13.5% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(6703,13747,<178,0>,<499,1>)
isOverloadedFunction 36322 10.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9112,364,<224,0>,<232,1>)
addGrammar 21034 5.9% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(4904,5570,<109,0>,<228,1>)
saveModule 12904 3.6% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(12479,6041,<269,0>,<382,1>)
rascalIsAcceptableSimple 10049 2.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(1916,1276,<64,0>,<87,1>)
newCollector 7556 2.1% |lib://typepal/src/analysis/typepal/Collector.rsc|(8291,35339,<188,0>,<1014,1>)
addADTs 6964 1.9% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(562,352,<27,0>,<33,1>)
rascalReportUnused 6144 1.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9478,2876,<234,0>,<290,1>)
ASTS PROFILE: 5817 data points, 358459 ticks, tick = 1 milliSecs
Scope Ticks % Source
checkOverloading 24615 6.9% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16113,2,<357,110>,<357,112>)
checkOverloading 24013 6.7% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16103,2,<357,100>,<357,102>)
isOverloadedFunction 18542 5.2% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9346,1,<228,66>,<228,67>)
lookupPathsWide 14382 4.0% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(14683,5,<362,49>,<362,54>)
checkOverloading 13458 3.8% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16080,2,<357,77>,<357,79>)
checkOverloading 12434 3.5% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(16098,2,<357,95>,<357,97>)
_run 10353 2.9% |lib://typepal/src/analysis/typepal/Solver.rsc|(66609,7,<1707,61>,<1707,68>)
isOverloadedFunction 8762 2.4% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/RascalConfig.rsc|(9360,5,<228,80>,<228,85>)
addGrammar 8292 2.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(8136,7,<182,31>,<182,38>)
saveModule 7258 2.0% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/Import.rsc|(17092,2,<353,19>,<353,21>)
addGrammar 4589 1.3% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(5719,1,<123,66>,<123,67>)
addADTs 4038 1.1% |file:///export/scratch1/landman/rascal-stuff/rascal-core/src/org/rascalmpl/core/library/lang/rascalcore/check/ADTandGrammar.rsc|(789,1,<31,17>,<31,18>)
bindWide 3938 1.1% |lib://typepal/src/analysis/typepal/ConfigurableScopeGraph.rsc|(13720,2,<340,19>,<340,21>)
list[ModuleMessages]: [
I'm gonna offer that performance trick as a PR soon, as it's semantics preserving but a lot faster.
Update:
- https://github.com/usethesource/rascal-core/pull/11 was merged, that increased the speed of the checker even more than discussed in the previous comment (from 845s to 290s)
- the priority implemention is still a high cause of performance hit (the 845s is with this function disabled, it used to be 1143s)
-
isOverloadedFunctionwhich is used for reporting unused functions, is called a lot and due to the way we don't have a map entry iterator, this one is slower than it "needs" to be. but without translating the model from a map to binary relation, I have no local change I can think of to improve the speed.
- why do we need to "filter out productions contained in priority/associativity declarations" again? @PaulKlint in
addGrammar. Are they in there twice? I guess that's necessary. It would be easy to collect all productions that are defined below an (abstract or concrete) priority or associativity node first (using a fast deep visit), and then filter the set instead of using source location containment again and again on all nested structures. - the entryIterator vs keyIterator difference in Rascal is not as pronounced as it is in Java; this is due to the other interpretative overhead.
- However, the
ooperator as inrel o mapormap o mapwould filter on a map's values and internally use an entryIterator. So rewriting that code to use "compose" might have an effect.
Regarding entry iterator. In rascal we have to write this:
for (k <- someMap) print(someMap[k]);
Which makes you do traverse the map trie for every k. So it should have been a O(n) operation, but it becomes a O(n * log(n)) operation. (okay to be fair, not exactly log(n) as our tries are never more than 8 deep, but then we get into hash-collision arrays).
I know our maps are not that deep, but still in large maps and calling this code a lot, it starts to matter.
Actually this is not about worst case analysis. The map is on average 3 or 4 levels deep. I did some testing years ago and the variable lookup/declaration was dominating.
Yep. The real worst case for map iteration plus lookup is currently in O(n^2) with hash collisions. Without hash collisions, or with a constant amount it's in O(n) because the depth of the trie maxes out at 7. So the big O notation is not good enough to analyze the problems we have with this situation.
The best thing IMO would be to refactor the code to use the compose operator. This removes both the variable binding overhead as well as the lookup overhead.
sure; after the mvn stuff is working for Paul and I've a demo of the error recovery.