[BUG] custom config.yaml and frameworks.yaml NOT uploaded on `aws` mode

[lujiaying]

Hi amazing automlbenchmark team,

Following the instruction, I have put my custom yaml files under PROJ_ROOT/user_dir,

user_dir
|-- config.yaml
|-- frameworks.yaml

And in user_dir/config.yaml, I have set up aws-resource_files according to HOWTO

---
frameworks:
  definition_file:  # this allows to add custom framework definitions (in {user}/frameworks.yaml) on top of the default ones.
    - '{root}/resources/frameworks.yaml'
    - '{user}/frameworks.yaml'

project_repository: https://github.com/lujiaying/automlbenchmark#ag_cascade_benchmark

versions:
  python: 3.9

aws:
  region: 'us-west-1'
  resource_files:
    - '{user}/config.yaml'
    - '{user}/frameworks.yaml'

 ...

Then I ran a test on aws mode by python runbenchmark.py autogluon_v0.5.1_high_il0.001 example 1h8c -u user_dir -m aws Note autogluon_v0.5.1_high_il0.001 is a custom framework added in user_dir/frameworks.yaml (I tested successfully on local mode).

However, it seems that `'{user}/config.yaml', '{user}/frameworks.yaml' are NOT uploaded. Error message are like:

[INFO] [amlb:07:14:14.127] Running benchmark `autogluon_v0.5.1_high_il0.001` on `example` framework in `local` mode.
[DEBUG] [amlb:07:14:14.127] Script args: Namespace(framework='autogluon_v0.5.1_high_il0.001', benchmark='example', constraint='1h8c', mode='local', task=None, fold=None, indir='/s3bucket/input', outdir='/s3bucket/output', userdir='/s3bucket/user', parallel=1, setup='only', keep_scores=True, exit_on_error=False, logging='console:info,app:debug,root:info', profiling=False, resume=False, session='', extra=['seed=auto']).
[DEBUG] [amlb.utils.config:07:14:14.127] Loading config file `/repo/resources/config.yaml`.
[DEBUG] [amlb.utils.config:07:14:14.164] No config file at `/s3bucket/user/config.yaml`, ignoring it.
...
...
[ERROR] [amlb:07:14:14.233]
ERROR:
Incorrect framework `autogluon_v0.5.1_high_il0.001`: not listed in ['/repo/resources/frameworks.yaml'].

May I know if I mess up anything? Shall I not use user_dir, but just directly modify the default config.yaml and framework.yaml? Thank you for the help!

[lujiaying]

In the log, notice some lines:

 [DEBUG] [amlb.runners.aws:07:10:52.759] Skipping upload of `/home/ec2-user/automlbenchmark/resources/benchmarks/example.yaml` to s3 bucket.
 [DEBUG] [amlb.runners.aws:07:10:52.759] Skipping upload of `/home/ec2-user/automlbenchmark/user_dir/config.yaml` to s3 bucket.
 [DEBUG] [amlb.runners.aws:07:10:52.759] Skipping upload of `/home/ec2-user/automlbenchmark/user_dir/frameworks.yaml` to s3 bucket.

...

[INFO] [amlb.runners.aws:18:21:36.357] Starting new EC2 instance with params: autogluon_v0.5.1_high_il0.001 example 1h8c   -Xseed=auto

1. user_dir is not uploaded to S3, because user_dir is in app_dir, refer to https://github.com/openml/automlbenchmark/blob/7729f0dcaa37b2d5c48ca7c63b66ddd5f729e786/amlb/runners/aws.py#L825-L828 https://github.com/openml/automlbenchmark/blob/7729f0dcaa37b2d5c48ca7c63b66ddd5f729e786/amlb/runners/aws.py#L842-L852

2. script_params== autogluon_v0.5.1_high_il0.001 example 1h8c -Xseed=auto, missing -u user_dir https://github.com/openml/automlbenchmark/blob/7729f0dcaa37b2d5c48ca7c63b66ddd5f729e786/amlb/runners/aws.py#L557-L558

[lujiaying]

Alright, able to get rid of these errors by python runbenchmark.py autogluon_v0.5.1_high_il0.001 example 1h8c -u user_dir -m aws

the price is to directly modify resources/config.yaml and resources/frameworks.yaml

[sebhrusen]

@lujiaying, you found the reason:

user_dir is not uploaded to S3, because user_dir is in app_dir

if you put your user_dir outside the amlb folder structure, it should work. Personally, I use directly or a sub-folder of the default ~/.config/automlbenchmark. If you need to easily switch between many configurations, you could also have a folder next to the app_dir with all your configs: as soon as the user_dir is completely dissociated from app_dir then it will upload files correctly

[lujiaying]

Thanks for the solution, @sebhrusen .

Was wondering the design philosophy behind this: If user_dir is inside app_dir, I totally agree that there's no need to upload user_dir to s3. However, it would be sweet if created EC2 instance can run with -u user_dir. Now it seems that they did not specify -u flag, which lead to my situation.

[Innixma]

I just ran into this issue as well, and it was quite counterintuitive the root cause. It would be very nice if this worked. If I have a branch of AMLB and want to have my own configs separated from the original configs, it is hard to run them on AWS. I first need to copy them outside of AMLB directory and specify the copied location. This isn't ideal and would be confusing to users who are not experts on how AMLB works.