[Feature Request] Api scrapper field

[tedr56]

First, thanks for this niece piece of software. I just installed it and I have a few idea for my usecase that others may need.

As mentionned in #721, a data scrapper is really important for big collections fields with public data. I was thinking a Api field with sub Url, Header, Query subfields, that could use previous data fields. For example, I'm working on pokemon cards, but could be anything:

CardNumber: Number
Set: String
ApiData:
  Url: https://api.pokemontcg.io/v2/cards
  Method: Get
  Headers:
    x-api-key: 0123456789
  Query: q=number:{{CardNumber}} set:{{Set}}
Illustration: ApiData.images.large

I know this a big piece of code. I tried to look into it but I'm not that good of a developper. What do you think?

Thanks.

[benjaminjonard]

Hello,

That's something I'd like to implements, but I have yet to find a good way to do that.

As Koillection aims to handle any kind of collections, I don't want to couple it too much with one particular use case. Your example works for that API, that returns json. But I know people wants to scrap HTML too. That's why I added a REST API as a first solution, people can make their own custom scrapping scripts to import items into Koillection.

I'll see if adding a scrapper builder directly into Koillection is doable

[tedr56]

Ho. I didn't know that. I've seen api commits but did't know the actual use.

I was thinking this kind of big setting could go near template and choice list menu. But then, it would be harder to link it with dynamic fields.

Also, we could imagine a rest scraper field, and an html scraper fields?