jitdb icon indicating copy to clipboard operation
jitdb copied to clipboard

Consider default pageSize value in toPullStream

Open arj03 opened this issue 4 years ago • 3 comments

I was looking at some numbers today and noticed that there was a significant overhead of using toPullStream in some cases. The following query takes 2.4 seconds:

pull(
      ssb.db.query(
        where(
          and(
            author('@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519')
          )
        ),
        toPullStream(),
        pull.collect((err, results) => {
            ....

while:

    ssb.db.query(
      where(
        and(
          author('@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519')
        )
      ),
      toCallback((err, results) => {
            ....

takes 1.2s.

It turns out that toPullStream actually does pagination internally in all cases. Tweaking this number made quite a difference. So I'm wondering how this number came about? And if it should be something like maybe 1000 instead?

arj03 avatar Jan 13 '22 14:01 arj03

Hmm, yeah 20 might be too low. I think I came up with the number arbitrarily.

staltz avatar Jan 13 '22 15:01 staltz

Remember this one? :)

arj03 avatar Nov 08 '22 09:11 arj03

Hmmm

staltz avatar Nov 08 '22 10:11 staltz