Error in Tokens count for Iteration node

[zqcing678]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Create an iteration node with LLM in the workflow and run it, then look at the tokens statistics for the iteration node

✔️ Expected Behavior

Correctly accumulate the tokens spent on each iteration of llm.

❌ Actual Behavior

Wrong and huge tokens stats belike:

[dosubot[bot]]

The issue you're experiencing with incorrect token statistics in the iteration node of Dify version 0.15.1 might be related to how tokens are counted and accumulated during the execution of the iteration node. The IterationNode class is responsible for executing operations over a list of items and tracks the total tokens used during execution. This is done through the graph_engine.graph_runtime_state.total_tokens, which is included in the metadata of events like IterationRunSucceededEvent and RunCompletedEvent ^[1].

If the token count appears excessively high, it could be due to a miscalculation in how tokens are being counted or accumulated across iterations. There are test cases in the codebase that validate the token counting logic, such as test_run and test_run_parallel, which ensure that iteration nodes process inputs correctly and produce expected outputs ^[2].

To address this issue, you might want to:

1. Review the implementation of the IterationNode to ensure that the token counting logic aligns with your expectations.
2. Check if there are any recent changes or known issues related to token counting in the iteration nodes that might affect version 0.15.1.
3. Consider running the existing test cases to verify if they pass in your environment, which could help identify if the issue is specific to your setup.

If the problem persists, further investigation into the specific configuration and execution context of your iteration node might be necessary.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Kevin9703]

I've encountered the same issue. When the number of iterations is greater than 10 and parallel mode is enabled, this problem occurs. After 10 iterations, the number of tokens added in subsequent iterations becomes the cumulative sum of previous iterations, rather than the token count of the current iteration.

        with flask_app.app_context():
            parallel_mode_run_id = uuid.uuid4().hex
            graph_engine_copy = graph_engine.create_copy()
            variable_pool_copy = graph_engine_copy.graph_runtime_state.variable_pool
            variable_pool_copy.add([self.node_id, "index"], index)
            variable_pool_copy.add([self.node_id, "item"], item)
            for event in self._run_single_iter(
                iterator_list_value=iterator_list_value,
                variable_pool=variable_pool_copy,
                inputs=inputs,
                outputs=outputs,
                start_at=start_at,
                graph_engine=graph_engine_copy,
                iteration_graph=iteration_graph,
                iter_run_map=iter_run_map,
                parallel_mode_run_id=parallel_mode_run_id,
            ):
                q.put(event)
           graph_engine.graph_runtime_state.total_tokens += graph_engine_copy.graph_runtime_state.total_tokens

[zqcing678]

Fixed with one line of code:

        with flask_app.app_context():
            parallel_mode_run_id = uuid.uuid4().hex
            graph_engine_copy = graph_engine.create_copy()

            graph_engine_copy.graph_runtime_state.total_tokens = 0

            variable_pool_copy = graph_engine_copy.graph_runtime_state.variable_pool
            variable_pool_copy.add([self.node_id, "index"], index)
            variable_pool_copy.add([self.node_id, "item"], item)

[DavideDelbianco]

Thanks god, i lost 10 years of life today when experiencing this issue with a count of Trillions of tokens

[dosubot[bot]]

Hi, @zqcing678. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

• The issue involved incorrect token statistics in the iteration node of Dify version 0.15.1.
• The problem was identified to occur when iterations exceed 10 in parallel mode, leading to cumulative token counts.
• You resolved the issue by resetting the total_tokens to 0 for each iteration.

Next Steps

• Please confirm if this issue is still relevant to the latest version of the Dify repository. If so, you can keep the discussion open by commenting on the issue.
• Otherwise, the issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!