racket
fine-grained control of check syntax library
It would be better to have the ability to control which functions are executed and which are not in the drracket/check-syntax library.
The motivation is that certain functions, such as the function that powers add-docs-menu, are super slow, take several seconds to run on hundreds lines of code. But some other functions are be 10x faster. If an IDE plugin needs to run only a subset of the functions, it can obtain a lot of speed improvements.
Can you share something that illustates this slowness? While it may make sense to push this kind of thing to the client, it may also be that there is a missing opportunity to use one of the many caches in there or somehow otherwise speed things up.
Yes. I find the slowest function is get-index-entry-info in document-variable:
https://github.com/racket/drracket/blob/e9e5656744e4bc9b1fe332fe3cf949339acdaa9b/drracket-tool-text-lib/drracket/private/syncheck/traversals.rkt#L1497
Each document-variable call takes several milliseconds, and it will be called thousand times in a big file.
I have a customized check-syntax library in my op_setup.rkt repo which only made the function document-variable empty. And I use this benchmark script gist.
The lib.rkt in that repo has 900 loc and 10k lines of expanded code.
This is the result with check-syntax in Racket 8.9 CS in that repo:
⬢ [archlinux:latest] ❯ racket bm.rkt lib.rkt
expand
cpu time: 468 real time: 474 gc time: 96
traversal
cpu time: 3858 real time: 3904 gc time: 114
After replace drracket/check-syntax with "check-syntax.rkt", this is the result with the customized check-syntax:
⬢ [archlinux:latest] ❯ racket bm.rkt lib.rkt
expand
cpu time: 475 real time: 481 gc time: 108
traversal
cpu time: 193 real time: 195 gc time: 14
It turns out the most time of get-index-entry-info spent on xref-binding->definition-tag calls which is probably cacheable.
I've pushed something that adds a cache there and see about a 4.5x speedup for the traversal time of the "racket bm.rkt lib.rkt" benchmark on my machine. Not as dramatic as your machine but our machines are probably not directly comparable so I am curious to know what you see if you include this commit.
nice! It reduced from ~3600 ms to ~800 ms for me and 4.5x faster.
My previous benchmark result was with fully disabled document-variable, it makes sense that it's much faster.
thanks. Another bottleneck sometimes I found is current-module-name-resolver:
https://github.com/racket/drracket/blob/2657eafdcfb5e4ccef19405492244f679b9234ef/drracket-tool-text-lib/drracket/private/syncheck/traversals.rkt#L1343
Added a time around the expression ((current-module-name-resolver) datum #f #f #t), I got this result with the latest cached check-syntax:
⬢ [archlinux:latest] ❯ racket bm.rkt copier.rkt
expand
cpu time: 1086 real time: 1100 gc time: 245
traversal
cpu time: 0 real time: 0 gc time: 0
cpu time: 932 real time: 944 gc time: 195
cpu time: 1 real time: 1 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 1162 real time: 1176 gc time: 213
and
⬢ [archlinux:latest] ❯ racket bm.rkt check-syntax.rkt
expand
cpu time: 1035 real time: 1050 gc time: 227
traversal
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 853 real time: 866 gc time: 179
cpu time: 0 real time: 0 gc time: 0
cpu time: 0 real time: 0 gc time: 0
cpu time: 1077 real time: 1093 gc time: 200
It's heavy single call, and looks like there is another expand inside it. I don't know if it can be improved.
Those are the only two bottlenecks I've noticed.
This one is because it has to compile some dependencies. In the case of copier.rkt, it is compiling check-syntax.rkt. If you do "raco make check-syntax.rkt" you should see a speed up there.
It's a good suggestion. The expand time and traversal time is improved a lot, from 2s to 800ms in total.
I made a change (just locally on my machine) to eliminate the extra indirection via a thread (not safe in general but fine for the benchmark) and it reveals that about 43% of the traversal time is being spent in database queries. That doesn't seem optimal so it still seems reasonable to do more work here.
I also made another change to actually do the profile and it runs the traversal step a bunch of times to try to make the results more accurate; these are the results I see.
Total cpu time observed: 2704ms (out of 2716ms)
Number of samples taken: 55 (once every 49ms)
===========================================================================================
Caller
Idx Total Self Name+src Local%
ms(pct) ms(pct) Callee
===========================================================================================
[1] 2704(100.0%) 0(0.0%) body of "/Users/robby/git/6cdh/op_setup.rkt/bm.rkt" ...
profile-thunk [2] 100.0%
-------------------------------------------------------------------------------------------
body of "/Users/robby/git/6cdh/op_setup.rkt/bm.rkt" [1]100.0%
[2] 2704(100.0%) 0(0.0%) profile-thunk ...are/pkgs/profile-lib/main.rkt:9:0
??? [3] 100.0%
-------------------------------------------------------------------------------------------
profile-thunk [2] 100.0%
[3] 2704(100.0%) 0(0.0%) ??? ...acket/share/pkgs/profile-lib/main.rkt:40:10
??? [4] 100.0%
-------------------------------------------------------------------------------------------
??? [3] 100.0%
[4] 2704(100.0%) 0(0.0%) ??? ...acket/share/pkgs/profile-lib/main.rkt:59:31
traversal [5] 100.0%
-------------------------------------------------------------------------------------------
??? [4] 100.0%
[5] 2704(100.0%) 0(0.0%) traversal ...bby/git/6cdh/op_setup.rkt/bm.rkt:25:0
...row-higher-order.rkt:375:33 [6] 100.0%
-------------------------------------------------------------------------------------------
traversal [5] 100.0%
[6] 2704(100.0%) 0(0.0%) ...row-higher-order.rkt:375:33 ...order.rkt:375:33
expanded-expression [7] 100.0%
-------------------------------------------------------------------------------------------
...row-higher-order.rkt:375:33 [6] 100.0%
[7] 2704(100.0%) 0(0.0%) expanded-expression ...ncheck/traversals.rkt:48:10
annotate-variables [8] 83.7%
try-next [9] 14.5%
annotate-contracts [10] 1.8%
-------------------------------------------------------------------------------------------
expanded-expression [7] 100.0%
[8] 2263(83.7%) 0(0.0%) annotate-variables ...yncheck/traversals.rkt:743:0
document-variable [11] 82.6%
connect-identifier [13] 15.3%
initialize-binder-connections [15] 2.2%
-------------------------------------------------------------------------------------------
try-next [9] 11.5%
body-loop [21] 14.6%
level+tail+mod-loop [12] 14.6%
expanded-expression [7] 59.4%
[9] 392(14.5%) 0(0.0%) try-next .../private/syncheck/traversals.rkt:325:6
level+tail+mod-loop [12] 78.1%
try-next [9] 11.5%
body-loop [21] 10.4%
-------------------------------------------------------------------------------------------
expanded-expression [7] 100.0%
[10] 50(1.8%) 0(0.0%) annotate-contracts .../contract-traversal.rkt:11:0
loop [14] 100.0%
-------------------------------------------------------------------------------------------
annotate-variables [8] 100.0%
[11] 1868(69.1%) 0(0.0%) document-variable ...yncheck/traversals.rkt:1488:0
hash-ref! [16] 94.8%
build-docs-label [19] 5.2%
-------------------------------------------------------------------------------------------
body-loop [21] 1.2%
level+tail+mod-loop [12] 16.0%
try-next [9] 82.7%
[12] 392(14.5%) 196(7.2%) level+tail+mod-loop ...ncheck/traversals.rkt:219:2
try-next [9] 16.7%
level+tail+mod-loop [12] 16.0%
collect-general-info [20] 15.6%
add-binders [22] 12.5%
stx-pair? [18] 12.5%
body-loop [21] 2.5%
-------------------------------------------------------------------------------------------
annotate-variables [8] 100.0%
[13] 346(12.8%) 49(1.8%) connect-identifier ...yncheck/traversals.rkt:917:0
connect-identifier-to-a-require [17] 85.8%
-------------------------------------------------------------------------------------------
annotate-contracts [10] 5.3%
loop [14] 94.7%
[14] 50(1.8%) 0(0.0%) loop ...ivate/syncheck/contract-traversal.rkt:22:2
loop [14] 94.7%
stx-pair? [18] 5.3%
-------------------------------------------------------------------------------------------
annotate-variables [8] 100.0%
[15] 49(1.8%) 49(1.8%) initialize-binder-connections ...ersals.rkt:1181:0
-------------------------------------------------------------------------------------------
document-variable [11] 100.0%
[16] 1770(65.5%) 48(1.8%) hash-ref! .../racket/private/more-scheme.rkt:377:2
??? [23] 97.3%
-------------------------------------------------------------------------------------------
connect-identifier [13] 100.0%
[17] 296(11.0%) 100(3.7%) connect-identifier-to-a-require ...rsals.rkt:959:0
connect-syntaxes [24] 66.4%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 49.7%
loop [14] 50.3%
[18] 98(3.6%) 98(3.6%) stx-pair? .../collects/racket/private/stx.rkt:40:4
-------------------------------------------------------------------------------------------
document-variable [11] 100.0%
[19] 98(3.6%) 0(0.0%) build-docs-label ...syncheck/traversals.rkt:1515:0
loop [25] 100.0%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 100.0%
[20] 98(3.6%) 0(0.0%) collect-general-info ...heck/traversals.rkt:266:12
loop [27] 50.0%
add-disappeared-uses [26] 50.0%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 33.3%
try-next [9] 66.7%
[21] 98(3.6%) 0(0.0%) body-loop ...rivate/syncheck/traversals.rkt:293:10
try-next [9] 83.3%
level+tail+mod-loop [12] 16.7%
-------------------------------------------------------------------------------------------
level+tail+mod-loop [12] 100.0%
[22] 49(1.8%) 0(0.0%) add-binders ...vate/syncheck/traversals.rkt:1410:0
add-id [30] 100.0%
-------------------------------------------------------------------------------------------
hash-ref! [16] 100.0%
[23] 1722(63.7%) 0(0.0%) ??? ...lib/drracket/private/syncheck/xref.rkt:47:4
xref-binding-tag [28] 74.2%
xref-tag->path+anchor [29] 25.8%
-------------------------------------------------------------------------------------------
connect-identifier-to-a-require [17] 100.0%
[24] 197(7.3%) 197(7.3%) connect-syntaxes ...syncheck/traversals.rkt:1198:0
-------------------------------------------------------------------------------------------
loop [25] 25.3%
build-docs-label [19] 74.7%
[25] 98(3.6%) 98(3.6%) loop .../private/map.rkt:40:19
loop [25] 25.3%
-------------------------------------------------------------------------------------------
collect-general-info [20] 100.0%
[26] 49(1.8%) 49(1.8%) add-disappeared-uses ...check/traversals.rkt:723:0
-------------------------------------------------------------------------------------------
loop [27] 50.0%
collect-general-info [20] 50.0%
[27] 49(1.8%) 0(0.0%) loop ...ket/private/syncheck/traversals.rkt:1372:2
loop [27] 50.0%
add-id [30] 50.0%
-------------------------------------------------------------------------------------------
??? [23] 100.0%
[28] 1278(47.3%) 0(0.0%) xref-binding-tag ...le-lib/scribble/xref.rkt:122:2
loop [31] 100.0%
-------------------------------------------------------------------------------------------
??? [23] 100.0%
[29] 444(16.4%) 0(0.0%) xref-tag->path+anchor ...b/scribble/xref.rkt:160:0
compose-class [32] 100.0%
-------------------------------------------------------------------------------------------
loop [27] 50.0%
add-binders [22] 50.0%
[30] 98(3.6%) 49(1.8%) add-id ...t/private/syncheck/traversals.rkt:1565:0
do-ref [33] 50.0%
-------------------------------------------------------------------------------------------
loop [31] 39.7%
xref-binding-tag [28] 60.3%
[31] 1278(47.3%) 0(0.0%) loop ...bble/scribble-lib/scribble/search.rkt:75:5
resolve-search [34] 53.7%
loop [31] 39.7%
resolve-get/tentative [36] 5.4%
??? [38] 1.3%
-------------------------------------------------------------------------------------------
xref-tag->path+anchor [29] 100.0%
[32] 444(16.4%) 99(3.7%) compose-class ...private/class-internal.rkt:2142:0
gen-for-each [35] 55.5%
make-field-info [37] 11.2%
??? [39] 11.0%
-------------------------------------------------------------------------------------------
add-id [30] 100.0%
[33] 49(1.8%) 49(1.8%) do-ref ...ollects/syntax/private/id-table.rkt:77:2
-------------------------------------------------------------------------------------------
loop [31] 100.0%
[34] 1084(40.1%) 0(0.0%) resolve-search ...ibble-lib/scribble/core.rkt:73:0
resolve-get/ext-id* [40] 100.0%
-------------------------------------------------------------------------------------------
compose-class [32] 100.0%
[35] 246(9.1%) 0(0.0%) gen-for-each ...lects/racket/private/map.rkt:267:2
??? [41] 100.0%
-------------------------------------------------------------------------------------------
loop [31] 100.0%
[36] 145(5.4%) 0(0.0%) resolve-get/tentative ...ib/scribble/core.rkt:69:0
resolve-get/where [43] 100.0%
-------------------------------------------------------------------------------------------
compose-class [32] 100.0%
[37] 50(1.8%) 50(1.8%) make-field-info ...rivate/class-internal.rkt:295:0
-------------------------------------------------------------------------------------------
loop [31] 100.0%
[38] 50(1.8%) 0(0.0%) ??? ...contract/private/arrow-val-first.rkt:486:18
module-path-index->taglet [42] 100.0%
-------------------------------------------------------------------------------------------
compose-class [32] 100.0%
[39] 49(1.8%) 49(1.8%) ??? ...scribble-lib/scribble/html-render.rkt:263:2
-------------------------------------------------------------------------------------------
resolve-search [34] 100.0%
[40] 1084(40.1%) 0(0.0%) resolve-get/ext-id* ...-lib/scribble/core.rkt:55:0
resolve-get/where [43] 100.0%
-------------------------------------------------------------------------------------------
gen-for-each [35] 100.0%
[41] 246(9.1%) 0(0.0%) ??? ...s/racket/private/class-internal.rkt:2664:32
loop [44] 100.0%
-------------------------------------------------------------------------------------------
??? [38] 100.0%
[42] 50(1.8%) 0(0.0%) module-path-index->taglet ...scribble/tag.rkt:98:0
do-module-path-index->taglet [45] 100.0%
-------------------------------------------------------------------------------------------
resolve-get/tentative [36] 11.8%
resolve-get/ext-id* [40] 88.2%
[43] 1229(45.4%) 50(1.8%) resolve-get/where ...le-lib/scribble/core.rkt:24:0
??? [46] 96.0%
-------------------------------------------------------------------------------------------
??? [41] 100.0%
[44] 246(9.1%) 246(9.1%) loop .../racket/private/class-internal.rkt:2678:49
-------------------------------------------------------------------------------------------
module-path-index->taglet [42] 100.0%
[45] 50(1.8%) 0(0.0%) do-module-path-index->taglet ...ibble/tag.rkt:64:0
combine-relative-elements [47] 100.0%
-------------------------------------------------------------------------------------------
resolve-get/where [43] 100.0%
[46] 1180(43.6%) 0(0.0%) ??? ...ribble-lib/scribble/base-render.rkt:1218:28
send-arg69 [48] 100.0%
-------------------------------------------------------------------------------------------
do-module-path-index->taglet [45] 100.0%
[47] 50(1.8%) 50(1.8%) combine-relative-elements ...llapse-noctc.rkt:75:2
-------------------------------------------------------------------------------------------
??? [46] 100.0%
[48] 1180(43.6%) 0(0.0%) send-arg69 ...scribble-lib/scribble/xref.rkt:68:19
??? [49] 100.0%
-------------------------------------------------------------------------------------------
send-arg69 [48] 100.0%
[49] 1180(43.6%) 0(0.0%) ??? .../plt/pkgs/racket-index/setup/xref.rkt:122:2
try [50] 100.0%
-------------------------------------------------------------------------------------------
??? [49] 100.0%
[50] 1180(43.6%) 0(0.0%) try .../plt/pkgs/racket-index/setup/xref.rkt:126:6
call-with-database [51] 100.0%
-------------------------------------------------------------------------------------------
try [50] 100.0%
[51] 1180(43.6%) 0(0.0%) call-with-database ...llects/setup/doc-db.rkt:71:0
loop [52] 100.0%
-------------------------------------------------------------------------------------------
call-with-database [51] 100.0%
[52] 1180(43.6%) 50(1.8%) loop ...plt/racket/collects/setup/doc-db.rkt:507:2
??? [53] 66.5%
start-transaction method in transactions% [54] 29.3%
-------------------------------------------------------------------------------------------
loop [52] 100.0%
[53] 784(29.0%) 0(0.0%) ??? ...plt/racket/collects/setup/doc-db.rkt:523:11
call-with-lock* method in locking% [58] 62.3%
??? [55] 37.7%
-------------------------------------------------------------------------------------------
loop [52] 100.0%
[54] 346(12.8%) 0(0.0%) start-transaction method in transactions% ...338:4
call-with-lock* method in locking% [58] 100.0%
-------------------------------------------------------------------------------------------
??? [53] 100.0%
[55] 296(10.9%) 49(1.8%) ??? ...t/plt/racket/collects/setup/doc-db.rkt:94:3
~s [56] 50.0%
query-rows0 [57] 33.4%
-------------------------------------------------------------------------------------------
??? [55] 100.0%
[56] 148(5.5%) 148(5.5%) ~s ...plt/racket/collects/racket/format.rkt:150:16
-------------------------------------------------------------------------------------------
??? [55] 100.0%
[57] 99(3.7%) 0(0.0%) query-rows0 ...private/generic/functions.rkt:119:0
compose-statement [59] 50.0%
call-with-lock* method in locking% [58] 50.0%
-------------------------------------------------------------------------------------------
query-rows0 [57] 5.6%
start-transaction method in transactions% [54] 39.1%
??? [53] 55.3%
[58] 884(32.7%) 50(1.8%) call-with-lock* method in locking% ...on.rkt:209:4
??? [60] 88.8%
call-with-lock [61] 5.6%
-------------------------------------------------------------------------------------------
query-rows0 [57] 100.0%
[59] 50(1.8%) 0(0.0%) compose-statement ...te/generic/functions.rkt:90:0
find-method/who [70] 100.0%
-------------------------------------------------------------------------------------------
call-with-lock* method in locking% [58] 100.0%
[60] 784(29.0%) 50(1.8%) ??? ...ollects/db/private/generic/common.rkt:180:8
??? [62] 93.7%
-------------------------------------------------------------------------------------------
call-with-lock* method in locking% [58] 100.0%
[61] 50(1.8%) 0(0.0%) call-with-lock ...private/generic/common.rkt:172:2
lock-acquire/start-atomic [63] 100.0%
-------------------------------------------------------------------------------------------
??? [60] 100.0%
[62] 735(27.2%) 0(0.0%) ??? ...llects/db/private/generic/common.rkt:215:18
??? [64] 59.7%
??? [65] 40.3%
-------------------------------------------------------------------------------------------
call-with-lock [61] 100.0%
[63] 50(1.8%) 50(1.8%) lock-acquire/start-atomic ...eric/common.rkt:128:2
-------------------------------------------------------------------------------------------
??? [62] 100.0%
[64] 439(16.2%) 0(0.0%) ??? ...ollects/db/private/generic/common.rkt:360:8
end-transaction* method in connection% [66] 100.0%
-------------------------------------------------------------------------------------------
??? [62] 100.0%
[65] 296(10.9%) 0(0.0%) ??? ...ollects/db/private/generic/common.rkt:340:8
start-transaction* method in connection% [67] 100.0%
-------------------------------------------------------------------------------------------
??? [64] 100.0%
[66] 439(16.2%) 0(0.0%) end-transaction* method in connection% ...kt:384:4
query1 method in connection% [68] 100.0%
-------------------------------------------------------------------------------------------
??? [65] 100.0%
[67] 296(10.9%) 0(0.0%) start-transaction* method in connection% ...:364:4
query1 method in connection% [68] 100.0%
-------------------------------------------------------------------------------------------
start-transaction* method in connection% [67] 40.3%
end-transaction* method in connection% [66] 59.7%
[68] 735(27.2%) 47(1.7%) query1 method in connection% ...onnection.rkt:89:4
prepare1 method in statement-cache% [69] 46.7%
prepare1* method in connection% [73] 26.8%
finalize method in prepared-statement% [71] 6.7%
find-method/who [70] 6.7%
??? [72] 6.7%
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 100.0%
[69] 343(12.7%) 0(0.0%) prepare1 method in statement-cache% ...n.rkt:560:4
make-sql-classifier [74] 43.3%
prepare1* method in connection% [73] 42.3%
use-cache? method in statement-cache% [75] 14.4%
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 50.0%
compose-statement [59] 50.0%
[70] 99(3.7%) 99(3.7%) find-method/who ...ivate/class-internal.rkt:3969:0
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 100.0%
[71] 50(1.8%) 50(1.8%) finalize method in prepared-statement% ...rkt:73:4
-------------------------------------------------------------------------------------------
query1 method in connection% [68] 100.0%
[72] 49(1.8%) 49(1.8%) ??? ...s/racket/private/class-internal.rkt:2619:34
-------------------------------------------------------------------------------------------
prepare1 method in statement-cache% [69] 42.4%
query1 method in connection% [68] 57.6%
[73] 342(12.6%) 0(0.0%) prepare1* method in connection% ...ction.rkt:273:4
??? [76] 56.6%
do-make-object/real-class [77] 43.4%
-------------------------------------------------------------------------------------------
prepare1 method in statement-cache% [69] 100.0%
[74] 148(5.5%) 50(1.8%) make-sql-classifier ...te/generic/common.rkt:609:2
sql-skip-comments [78] 66.7%
-------------------------------------------------------------------------------------------
prepare1 method in statement-cache% [69] 100.0%
[75] 50(1.8%) 50(1.8%) use-cache? method in statement-cache% ...rkt:477:4
-------------------------------------------------------------------------------------------
prepare1* method in connection% [73] 100.0%
[76] 194(7.2%) 194(7.2%) ??? ...ts/db/private/sqlite3/connection.rkt:470:22
-------------------------------------------------------------------------------------------
prepare1* method in connection% [73] 100.0%
[77] 148(5.5%) 50(1.8%) do-make-object/real-class ...s-internal.rkt:3629:0
continue-make-object [79] 66.7%
-------------------------------------------------------------------------------------------
make-sql-classifier [74] 100.0%
[78] 99(3.7%) 99(3.7%) sql-skip-comments ...vate/generic/common.rkt:616:0
-------------------------------------------------------------------------------------------
do-make-object/real-class [77] 100.0%
[79] 99(3.7%) 0(0.0%) continue-make-object .../class-internal.rkt:3655:0
??? [80] 100.0%
-------------------------------------------------------------------------------------------
continue-make-object [79] 100.0%
[80] 99(3.7%) 0(0.0%) ??? ...llects/db/private/generic/prepared.rkt:14:2
extract-arg [81] 100.0%
-------------------------------------------------------------------------------------------
??? [80] 100.0%
[81] 99(3.7%) 0(0.0%) extract-arg ...t/private/class-internal.rkt:3767:0
assq [82] 100.0%
-------------------------------------------------------------------------------------------
extract-arg [81] 100.0%
[82] 99(3.7%) 99(3.7%) assq ...et/collects/racket/private/list.rkt:187:13
-------------------------------------------------------------------------------------------
I saw the usage of the database in [50] pkgs/racket-index/setup/xref.rkt is query only. Is it possible to preprocess the database, eliminate the need for it, and use a hashtable instead?
Just doing a quick check, it looks like we get to that line 1970 times and in 1516 of them, the p or the key arguments are different.
I mean the hash table is faster than database query. This is a test:
I exported data from /usr/share/doc/racket/docindex.sqlite
$ cp /usr/share/doc/racket/docindex.sqlite a.sqlite
$ echo 'select * from documented' | sqlite3 a.sqlite > data.txt
then use this script
#lang racket
(define ht (make-hash))
(define stags '())
(for ([line (file->lines "data.txt")])
(match (string-split line "|")
[(list stag pathid)
(hash-set! ht stag (string->number pathid))
(set! stags (cons stag stags))]
[_ (error "unknown item:" line)]))
(define (hash-query keys)
(for/list ([key keys])
(hash-ref ht key)))
(require db)
(define (db-query conn keys)
(for/list ([key keys])
(define rows (query-rows conn
(virtual-statement "SELECT pathid FROM documented WHERE stag=$1")
key))
(define pathid (and (pair? rows) (vector-ref (car rows) 0)))
pathid))
(define keys (take (shuffle stags) 10000))
(length keys)
(time (void (hash-query keys)))
(define conn (sqlite3-connect #:database "a.sqlite"
#:mode 'read-only
#:busy-retry-limit 0))
(time (void (db-query conn keys)))
(equal? (hash-query keys) (db-query conn keys))
result:
10000
cpu time: 8 real time: 8 gc time: 0
cpu time: 897 real time: 906 gc time: 73
#t
The function call-with-retry/transaction is slow, but we will not need it if we preprocess the database and use hash table.
I think we don't want to stop using a database for the documentation. Unless I misunderstand your meaning?
Robby
On Wed, Aug 16, 2023 at 11:12 PM 6cdh @.***> wrote:
The function call-with-retry/transaction is slow, but we will not need it if we preprocess the database and use hash table.
— Reply to this email directly, view it on GitHub https://github.com/racket/drracket/issues/633#issuecomment-1681583823, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADBNMA7FV4DS7WZTU6O4ZTXVWK3TANCNFSM6AAAAAA3MLSIF4 . You are receiving this because you commented.Message ID: @.***>
No. I didn't mean stop using a database.
I mean only stop using the databse at the bottleneck part.
My idea is that only change the code at [49] pkgs/racket-index/setup/xref.rkt:122 from
(lambda (key use-id)
;; connect database
;; call-with-retry/transaction
;; database query
;; disconnect database
)
to
(let ([hash-table (preprocess-database main-db user-db)])
(lambda (key use-id)
;; hash table access
))
I am not following how this can be achieved. How do you know what to put into the hash table?
On Thu, Aug 17, 2023 at 6:39 AM 6cdh @.***> wrote:
No. I didn't mean stop using a database.
I mean only stop using the databse at the bottleneck part. My idea is that only change the code at [49] pkgs/racket-index/setup/xref.rkt:122 from
(lambda (key use-id) ;; connect database ;; call-with-retry/transaction ;; database query ;; disconnect database )
to
(let ([hash-table (preprocess-database main-db user-db)]) (lambda (key use-id) ;; hash table access ))
— Reply to this email directly, view it on GitHub https://github.com/racket/drracket/issues/633#issuecomment-1682134270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADBNMGTVYLG2D3Y6IHUCATXVX7FTANCNFSM6AAAAAA3MLSIF4 . You are receiving this because you commented.Message ID: @.***>
All rows in the table documented of the database as the function doc-db-key->path only cares that table.
The schema is
sqlite> .schema documented
CREATE TABLE documented (stag VARCHAR(256), pathid SMALLINT, UNIQUE (stag, pathid));
CREATE INDEX documentedStags on documented (stag);
Currently the number of rows is
sqlite> select count(*) from documented;
83935
It's not too large.