[Feature Request] Improve compatibility between PromptFlow and Python modules with relative imports

[pamelafox]

I am trying to use the promptflow-evals SDK in a project where I am using relative imports, which works fine because of how I call the modules (with python -m modulename).

However, PromptFlow tries to import my file for some reason, and then errors:

  File "/Users/pamelafox/ai-rag-chat-evaluator/scripts/evaluate.py", line 142, in run_evaluation
    results = evaluate(
              ^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/evals/evaluate/_evaluate.py", line 252, in evaluate
    input_data_df, target_generated_columns, target_run = _apply_target_to_data(target, data, pf_client,
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/evals/evaluate/_evaluate.py", line 128, in _apply_target_to_data
    run = pf_client.run(
          ^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_pf_client.py", line 301, in run
    return self._run(
           ^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_pf_client.py", line 226, in _run
    return self.runs.create_or_update(run=run, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_telemetry/activity.py", line 265, in wrapper
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/operations/_run_operations.py", line 134, in create_or_update
    created_run = RunSubmitter(client=self._client).submit(run=run, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 42, in submit
    self._run_bulk(run=run, stream=stream, **kwargs)
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 111, in _run_bulk
    with flow_overwrite_context(flow_obj, tuning_node, variant, connections=run.connections) as flow:
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/utils.py", line 264, in flow_overwrite_context
    override_flow_yaml(
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/utils.py", line 219, in override_flow_yaml
    update_signatures(code=flow_dir_path, data=flow_dag)
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_utilities/signature_utils.py", line 148, in update_signatures
    signatures, _, _ = infer_signature_for_flex_flow(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_sdk/_utilities/signature_utils.py", line 72, in infer_signature_for_flex_flow
    flow_meta = inspector_proxy.get_entry_meta(entry=entry, working_dir=code)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_proxy/_python_inspector_proxy.py", line 47, in get_entry_meta
    return _generate_flow_meta(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/pamelafox/ai-rag-chat-evaluator/.venv/lib/python3.11/site-packages/promptflow/_core/entry_meta_generator.py", line 84, in _generate_flow_meta
    raise GenerateFlowMetaJsonError(error_message)
promptflow.core._errors.GenerateFlowMetaJsonError: Generate meta failed, detail error:
["Failed to load python module from file '/Users/pamelafox/ai-rag-chat-evaluator/scripts/evaluate.py': (ImportError) attempted relative import with no known parent package"]

So now I have to restructure my project to avoid relative imports. Please loosen this constraint.

[D-W-]

Hi @pamelafox , in our concept, flow should be self-contained. Which makes it easier to build flow snapshot and share from local to cloud. When flow has relative external imports, we could not build snapshot for it. But we can improve the error message to make it more friendly to customer.

[github-actions[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

[pamelafox]

I think promptflow should be able to work with files that import other files, as most developers use modules in order to improve code reusability. If I can't bring in any other files, then I will have to needlessly repeat common code across parts of my codebase.

[D-W-]

Added a long-term tag for this. I have a proposal to introduce a code field to flow's YAML. If user need to import outside of current working directory. They can set flow's code to base folder.

For example, a project organized like this

src/
    common/
    flow1/
    flow2/

The flow1 and flow2 can set code in flow YAML like this

code: path/to/src

But, you'll need to use absolute import like this

from common import xxx

instead of relative import like this

from .. import xxx

since we used multi-processing to execute flow and relative import may fail to find it's parent package.

[mces89]

blocked on the same issue.

[MingStar]

I'm blocked on this issue as well with 2 flows in the same project, both flows required shared code logic, and the VS Code extension were not able to import shared modules as Python nodes/tools.

Restructuring the project only works when just one flow at the root level.

[D-W-]

@MingStar , since it's currently prompt flow's by design behavior. A workaround would be put both of the flow files in root level like this. Let me know if it helps.

lib/
flow1/
flow2/
flow1.dag.yaml
flow2.dag.yaml

[MingStar]

@MingStar , since it's currently prompt flow's by design behavior. A workaround would be put both of the flow files in root level like this. Let me know if it helps.

lib/
flow1/
flow2/
flow1.dag.yaml
flow2.dag.yaml

but with this workaround the VS Code extension does not recognize the 2 DAG yaml files... it seems to only look for flow.dag.yaml

[D-W-]

@MingStar , since it's currently prompt flow's by design behavior. A workaround would be put both of the flow files in root level like this. Let me know if it helps.

lib/
flow1/
flow2/
flow1.dag.yaml
flow2.dag.yaml

but with this workaround the VS Code extension does not recognize the 2 DAG yaml files... it seems to only look for flow.dag.yaml

The workaround I provided only works with SDK/CLI, currently VS Code extension can only recognize flow.dag.yaml is single folder. If you need to use VS Code extension, maybe you can make the common lib an editable local package and install in your local environment or copy the lib folder to both of the flow folders.

We'll update here once switching working directory in flow is supported.