Kippt-for-Python
Kippt-for-Python copied to clipboard
thomasbiddle
Provide iterators for certain endpoints
What about returning iterators instead of the raw JSON for endpoints that return paginated results? E.g., instead of returning the JSON result
{
"meta": {
"limit": 20,
"next": "/api/clips/?limit=20&offset=20",
"offset": 0,
"previous": null,
"total_count": 33
},
"objects": [
{
"id": 15,
...
},
...
]
}
as described here, return an iterator that returns each of the objects on iteration:
{
"id": 15,
...
}
Additionally, the user would not have to worry about pagination -- the iterator would automatically request the next page as needed.
Similar to how the Ruby wrapper is setup? https://github.com/vesan/kippt/blob/master/lib/kippt/collection.rb
Sounds like a great idea!
I don't know Ruby, but maybe something like this:
class Depaginator:
def __init__(self, kippt, endpoint):
self.kippt = kippt
self.queue = queue.Queue()
self.next = endpoint
def __iter__(self):
return self
def __next__(self):
if not self.queue.empty():
return self.queue.get()
elif self.next:
r = requests.get(self.kippt.base + self.next, headers=self.kippt.header).json()
for obj in r["objects"]:
self.queue.put(obj)
self.next = r["meta"]["next"]
return next(self)
else:
raise StopIteration()
Each endpoint that returns multiple results could return a new Depaginator.