benjaminjonard
[Feature Request] Import collection
I have been tracking an extensive video game collection with CLZ. I have exported the collection to a CSV, and I'd like to be able to import it into Koillection, at least the fields that are present. It doesn't have to be anything fancy. I'd rename the columns in the spreadsheet to match my fields manually, if I had to.
Great addition indeed, It should be a must-thing for importing big collections
You can archive that with the API. I've build myself a import logic with a small C# app that fits my needs, but I also requested additional API functions in #1375 which can currently only be changed externally with SQL queries.
My goal was like yours, to export a CSV (from Excel for example) and to import that data into Koillection. For now, till a import feature gets added to the webinterface, the API is your only friend to archive that.
I see a few challenges with importing from a CSV file:
-
How to handle images? We could use an HTTP link, but that would require the image to be publicly accessible. Alternatively, we could use a separate ZIP file containing the images, with the CSV referencing the image path.
-
How to handle additional data? Text data is relatively straightforward, but other types require proper formatting. For example, the "Country" type requires a two-letter ISO code to work correctly. So for each data field, we’d need something like (assuming a JSON format):
{
"type": "country",
"label": "Country",
"value": "FR"
}
We'd also need to validate each field to ensure correct formatting and values.
Wouldn’t that be too difficult to use for most users (I guess it would be easier than using the API ) ? Do you have any other ideas on how to implement this?
Mass image imports would make the process itself way more complicated for the beginning I think. As the OP mentioned only a CSV I would say images could be added afterwards when the basic import systems already exists.
Regarding the additional data, this needs to be formated into a processable format you're right on that. In my import logic for example I read all the lines and generate a collection list first which checks for dupes + additional data field informations. After that I do the same for all items + data fields in another list. In the end I merge them into a final list I can run the creation logic with.
The final list looks something like this, keep in my I wrote it in C# with custom classes:
Input CSV example:
string;string;string;string;string;string;string
string;string;string;string;string;string;string
string;string;string;string;string;string;string
Output example:
- collection A
- string A
- string B
- string C
- string D
- string E
- List<> items
- item A
- string A
- string B
- string C
- string D
- item B
- string A
- string B
- string C
- string D
- item C
- string A
- string B
- string C
- string D
- collection B
- [.....]
Regarding field validation I'm not sure how its handeld from the webinterface, but one thing I found out through the API is that for example a checkbox, which according to the documentation uses integer values of 0 and 1, accepts strings with 0 and 1 instead of integers just fine. So maybe the whole system is not that strict about a few of those fields.
Talking about complexity for end users. To make it easier there could be a UI which reads the data temorarly first where users could match the fields from the CSV to existing templates/fields inside Koillection. There could also be a validation step that checks the import data to the selected field inside Koillection and only after all those validations pass then the import job which creates all of that could be run.
But my guess is that implementing all of this to the webinterface will take a longer time. Even if not suited for non-advanced users the API already works and with a little script or programm you can archive the same already.
I would display a field matching interface, where each field detected in the CSV would be matched with one of those belonging to the collection. This could also be done with the extra metadata. This element matching mechanism is performed by the Komga app, for managing books and comics, which reads a comic reading list and matches them with what you tell it to. It would be something like that, but matching fields with fields. I've done this in my collection app and it works quite well. Images should be managed as paths to them on the server, not as OLE or embedded objects as some databases do.
I started working and experimenting on this. I opted for the mapping interface suggested by @lopiv2. Right now it looks like this (very early WIP — please disregard missing translations and broken CSS 😅):
- The user uploads a CSV file, for example:
name,image,Japanese title,Author
"Frieren #1","https://www.manga-news.com/public/images/series/.Frieren_1_ki-oon_large.webp","葬送のフリーレン","Abe Tsukasa, Test new tag"
- The user can then map each field from the file to the new item’s properties (name, image, and data fields). Currently, the image field only accepts external URLs. I’ll try to add support for local paths later, but since only the admin can use it, I’ve put it aside for now.
-
After submitting the mapping, the user is taken to a preview page showing the items that will be created.
-
On submit, items are finally created
There is still a lot of work to do, the item data only work for simple text fields at the moment.
What I plan to add :
- Allow mapping of dedicated fields (like name and image) for tags, visibility, quantity...
- On the preview, allow checking/unchecking items to import
- Basic duplicate detection in the preview (probably by name)
- Add data validation and find a clear way to display errors
- Maybe handle IDs? For example: if an ID already exists in the database, override the item data with the CSV data (probably for a v2)
This is a small PoC for now, feedback is very welcome