kane icon indicating copy to clipboard operation
kane copied to clipboard

List all topics using pagination

Open Arkenan opened this issue 3 years ago • 1 comments

Currently there's a limit of about 100 topics when calling Kane.Topic.all function, as the pubsub API paginates the results. There are several ways of removing this 100 topic limit, but the easiest one is, by default, traversing the next pages.

When listing topics, we get a map of the following kind:

%{
  "topics" => list(%Kane.Topic{}),
  "nextPageToken" => str
}

We can GET projects/#{project}/topics?pageToken=#{page_token} to get the next page. If we use this recursively we can get all of the tokens until the map no longer has the nextPageToken present and we conclude that's the last page.

Arkenan avatar Mar 15 '22 16:03 Arkenan

Here's an example PR of how we can do this: https://github.com/peburrows/kane/pull/40

We can proceed on that idea in that same PR or we can move to a different one, I'm open to discuss ideas! That PR was tested with a pubsub with more than 100 topics and it worked without any issues.

Arkenan avatar Mar 15 '22 16:03 Arkenan