concat dataset

[yueyinqiu]

• add support for torch.utils.data.ConcatDataset
  • a new interface IDataset<out T> is added
    • now Dataset<T> implements IDataset<T>
  • more overloads of DataLoader() has been added, to accept IDataset datasets
    • those overloads directly return a DataLoader<T, S>, rather than its subclasses
  • datasets supported by DataLoader<T, S> have been relaxed to IDataset<T>
• parameter of collate functions in DataLoader<T, S> has been relaxed to IReadOnlyList

[yueyinqiu]

related to https://github.com/dotnet/TorchSharp/discussions/1348#discussioncomment-10035168 #1354 #1358

[yueyinqiu]

As mentioned here https://github.com/dotnet/TorchSharp/pull/1357#issuecomment-2478038515, the current version has no API changes (except the type of dataset and collate_fn in DataLoader<T, S>).

However we don't actually need Dataset<T>, Dataset and IterableDataset. Do we have any plan to remove them later? They may also occupy the position of some other classes in PyTorch, like #1353.

(But I find it hard to use [Obsolete] on them since some methods are using them as the return type.)

[K1T00]

What is the status on this one? I waiting for this to be merged.

[yueyinqiu]

Perhaps you could implement one yourself... This PR is one year old, and I'm not even sure if it conflicts with existing code...