[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split

[EdwardSJ151]

Solution/Feature

When creating a custom task, the few-shot samples can be specified via a split from the dataset. However, it would be convenient if there were a way to make it so I can specify samples from the split that will be used as the examples, similar to what LM Evaluation Harness does here in its yaml config:

test_split: train
fewshot_split: train
fewshot_config:
  sampler: id_sampler
  sampler_config:
    id_list: ["2022_21", "2022_88", "2022_143"]
    id_column: id
    exclude_from_task: true

Something along the lines of:

@dataclass
class LightevalTaskConfig:
    """Stored configuration of a given [`LightevalTask`].

    Arguments:
    ...
        hf_subset (str): Subset used for the current task, will be default if none is selected.
        hf_avail_splits (list[str]): All the available splits in the evaluation dataset
        evaluation_splits (list[str]): List of the splits actually used for this evaluation
        few_shots_split (str): Name of the split from which to sample few-shot examples
        few_shots_id_column (str): Column name in the dataset to use for the few_shot_id_list
        few_shots_id_list (list[str]): List of unique identifiers corresponding to the samples in the dataset, based on matching these IDs with the values in few_shots_id_column.
        few_shots_select (str): Method with which to sample few-shot examples
    ...
    """

The new arguments would be few_shots_id_column and few_shots_id_list, and wrote a rough description of what they would look like.