[Feature Request] Support multiple auth for the event of 429

[Tarrgon]

An easy "fix" to this rate limiting issue is to just allow switching between multiple accounts in the event of a 429. So when doing scraper.login() you can provide an array of username/password/email/etc combos to switch between if a 429 occurs.

Alternatively, allow scraper to emit a "rate limit" event that can be used to change the active auth. This would probably be easier to develop and gives the ability to handle it to the user rather than having to code it all in yourself.

I think this is better than waiting for a long time, since I was a little confused as to why my app was hanging until I dug into the code and saw that it can wait for up to 13 minutes without warning.

[karashiiro]

Released support for customizing the rate-limiting strategy in v0.15.0 - this doesn't yet include a built-in way to cycle auth, but I put a bit on how the new features can be used by library consumers to do this in the meantime in the PR description:

On pooling (tl;dr not yet but now you can DIY it)

One of the primary desired use cases for this is pooling auth to automatically rotate between users whenever one user gets rate-limited. Implementing this within the Scraper class will require a more significant refactor of the scraper and so is not in scope at this time. As a workaround, library consumers can leverage ErrorRateLimitStrategy to throw immediately when being rate-limited, and use that to pool Scraper instances themselves (pseudocode):

async function createScraper(args) {
  const scraper = new Scraper({ rateLimitStrategy: new ErrorRateLimitStrategy() });
  await scraper.login(...args);
}

const scrapers = new Map<UserId, Scraper>([
  ['user-1', await createScraper(authInfo1)],
  ['user-2', await createScraper(authInfo2)]
]);

// later...
function getTweets(args) {
  for (const [userId, scraper] of scrapers.entries()) {
    try {
      return scraper.getTweets(...args);
    } catch (err) {
      console.warn(`Scraper "${userId}" is currently rate-limited`)
    }
  }

  throw new Error('All scrapers are currently rate-limited.');
}

[pkdev08]

@karashiiro Hi, any idea on what the rate limit is before the 429? How many requests in a certain time? Thanks

[karashiiro]

@karashiiro Hi, any idea on what the rate limit is before the 429? How many requests in a certain time? Thanks

I don't think it's a fixed number, I'm not sure what it's based on exactly. When the 429 occurs, the response headers tell you how long until the current period resets, and possibly what the maximum number of requests in the most recent period was, but I don't think any of those numbers are consistent (see code here).

When I tried to hit the rate limit once just now to check, it had a limit of 187 requests (x-rate-limit-limit) with a window reset of about 2 minutes, but when I hit it a second time after that it had a window reset of 15 minutes instead.