Task output cutoff in Databricks

[yrwrstnhtmr]

I run into an issue where the output from an agent is cut-off even though it is not reaching any token output limit. For example a long string of markdown is terminated mid sentence without even closing the string. This occurs when using GPT4 and Claude Haiku/Sonnet.

The latest output I have had cut-off was 740 tokens (using https://tokenizer.streamlit.app/)

I am running this on Databricks not sure if that is causing it.