dora icon indicating copy to clipboard operation
dora copied to clipboard

Does Dora-Dataflow not support multiple Dynamic inputs?

Open chengzi0103 opened this issue 1 year ago • 2 comments

Describe the bug When I use multiple Dynamic nodes in a Dataflow, the Dataflow cannot capture the Dynamic inputs after starting (no response). Also, when I run python dynamic.py, there is no response.

I would like to ask, is this a bug or the current design, or is the Dynamic structure designed this way? Will this issue be fixed in future versions?

chengzi0103 avatar Aug 02 '24 03:08 chengzi0103

Describe the bug

When I use multiple Dynamic nodes in a Dataflow, the Dataflow cannot capture the Dynamic inputs after starting (no response). Also, when I run python dynamic.py, there is no response.

I would like to ask, is this a bug or the current design, or is the Dynamic structure designed this way? Will this issue be fixed in future versions?

Hi! Thanks for the report,

  • what is exactly the version of Dora you are using ? crates.io 0.3.5, main branch ? Fix tag ?

  • can you give us the exact dataflow description (YAML) ?

  • can you give us the python code of your dynamic nodes ?

Hennzau avatar Aug 02 '24 05:08 Hennzau

Describe the bug When I use multiple Dynamic nodes in a Dataflow, the Dataflow cannot capture the Dynamic inputs after starting (no response). Also, when I run python dynamic.py, there is no response. I would like to ask, is this a bug or the current design, or is the Dynamic structure designed this way? Will this issue be fixed in future versions?

Hi! Thanks for the report,

  • what is exactly the version of Dora you are using ? crates.io 0.3.5, main branch ? Fix tag ?
  • can you give us the exact dataflow description (YAML) ?
  • can you give us the python code of your dynamic nodes ?

@Hennzau

  1. We are using version 3.5 of dora-rs, and the version is the main branch.
  2. We used the dynamic method for nodes with node_id names web_search_task and reasoner_task_input.
  3. task_input.py is our Python code.
nodes:
  - id: web_search_task
    path: dynamic
    inputs:
      direction: dora/timer/secs/1
    outputs:
    - task
  - id: web_search_agent
    operator:
      python: ./scripts/web_search_agent.py
      inputs:
        web_search_task: web_search_task/task
      outputs:
      - web_search_results
      - web_search_resource
      - web_search_aggregate_output
  - id: more_question_agent
    operator:
      python: ./scripts/more_question_agent.py
      inputs:
        web_search_aggregate_output: web_search_agent/web_search_aggregate_output
      outputs:
      - more_question_results
      - web_search_aggregate_output
  - id: web_search_output
    operator:
      python: ./scripts/web_search_output.py
      inputs:
        web_search_aggregate_output: more_question_agent/web_search_aggregate_output
      outputs:
      - web_search_output
      - web_search_results
      - web_search_resource
      - more_question_results
      - web_search_task
  - id: reasoner_task_input
    path: dynamic
    inputs:
      direction: dora/timer/secs/1
    outputs:
    - reasoner_task
  - id: reasoner_agent
    operator:
      python: ./scripts/reasoner_agent.py
      inputs:
        reasoner_task: reasoner_task_input/reasoner_task
        more_question_results: more_question_agent/more_question_results
      outputs:
      - reasoner_result
  - id: reasoner_output
    operator:
      python: ./scripts/reasoner_output.py
      inputs:
        reasoner_result: reasoner_agent/reasoner_result
      outputs:
      - reasoner_output

Apart from different node_ids, other results of the dynamic nodes are the same.

import json

from dora import Node
import pyarrow as pa

node = Node("web_search_task")

event = node.next()
task_data = input('Please enter your task:  ',)
node.send_output('task',pa.array([json.dumps(task_data)]),event["metadata"])

chengzi0103 avatar Aug 02 '24 05:08 chengzi0103

Hi, sorry for the long time between my last response. Did you succeed in receiving multiple inputs?

Hennzau avatar Aug 15 '24 12:08 Hennzau

Hi, sorry for the long time between my last response. Did you succeed in receiving multiple inputs?

Thank you so much, we have successfully resolved the issue

chengzi0103 avatar Aug 16 '24 03:08 chengzi0103