Space characters in the objective name AND specifying a threshold leads to an error message: "AssertionError: Outcome constraint should be of form `metric_name >= x"

[nemozor]

When running the code below, I get the following error: AssertionError: Outcome constraint should be of form `metric_name >= x`, where x is a float bound and comparison operator is >= or <=.

The error message seems unrelated to the cause which I found to be the presence of space characters in the objective name.

The error can be avoided by replacing all spaces by underscores. This error does not show when no threshold is specified.

from ax.modelbridge.generation_strategy import GenerationStrategy,GenerationStep
from ax.modelbridge.registry import Models
from ax.service.utils.instantiation import ObjectiveProperties

Experiment_parameters=[
    {
        "name": "param1",
        "type": "range",
        "bounds": [90, 120],
        "value_type": "int",
    },
    {
        "name": "param2",
        "type": "range",
        "bounds": [38, 70],
        "value_type": "int",
    }]

Experiment_objectives={
    "distance to target": ObjectiveProperties(minimize=True, threshold = 0.05),
    "standard deviation": ObjectiveProperties(minimize=True, threshold = 0.25)}

Experiment_name="test"

gs = GenerationStrategy(steps=[
        GenerationStep(
            model=Models.SOBOL,
            num_trials=5
        ),
        GenerationStep(
                model=Models.MOO, 
                num_trials=10
        )])
ax_client = AxClient(generation_strategy=gs)

exp=ax_client.create_experiment(
    name=Experiment_name,
    parameters=Experiment_parameters,
    objectives=Experiment_objectives,)

Replacing the spaces by underscores fixes the issue:

Experiment_objectives={
    "distance_to_target": ObjectiveProperties(minimize=True, threshold = 0.05),
    "standard_deviation": ObjectiveProperties(minimize=True, threshold = 0.25)}

[mpolson64]

Wow, thank you for spotting this bug! We don't typically use metrics with whitespace in the name so we never saw this -- we should either explicitly disallow this or make sure its compatible here. Let me discuss with the team which direction we want to take this and get back to you.

We'll aim to have a PR up by EOW and included in our next release :)

[mpolson64]

Hi @nemozor, sorry to be getting back to you later than I had hoped. The team went back and forth on this a while, and eventually decided the best thing to do would be barring the use of spaces in parameter (and metric) names, but only when used in AxClient where we using string parsing for constraint definitions. The PR implementing this is here: https://github.com/facebook/Ax/pull/2397

This should be available in our next Ax release, which I expect to go out this week. Thank you for your patience!