pipelines
pipelines copied to clipboard
Optimize the log updating
Currently when you run a pipeline the UI will poll the logs endpoint until it finishes. This means every time it will fetch all of the logs which is far from optimal.
Solutions:
-
- Send
row-startnumber to the logs endpoint to tell backend to only return rows afterrow-start. UI would then merge the given rows to the previous rows.
- Send
-
- Implement websocket to stream the logs to frontend.
For long term the websocket sounds better, and we can leverage it for other things as well.