OceanDataTools
cache data server to serve updates asynchronously for better responsiveness
** Disclaimer - I do not claim to fully understand the inner workings of OpenRVDAS, what I'm requesting may already implemented or completely impossible to implement
The current cached data server serves websocket-connected clients with updated data from the data streams that the clients have subscribed to. The updated are provided synchronously at a default rate of 1Hz. The updates arrive regardless if the underlying data has changed since the previous update was sent.
This creates two scenarios that I feel should be avoided if possible:
- Connected clients receive "new" data even if the data hasn't changed.
- The minimum refresh interval of the client is controlled by the broadcast interval of the cache data server and not by changes in the underlying data.
Recommend refactoring the cache-data-server to send connected clients with updated data when the underlying data changes vs at a server-defined update rate.
CDS waits until it receives a "ready" message from widget, then sends all the data that have come in since the last time it sent any. Sometimes that's "none". So (1) isn't an issue.
The "interval" governs the maximum rate at which the CDS will respond to "ready" messages. If interval=2, it will wait until 2 seconds have elapsed since its last send, regardless of how quickly the "ready" comes. Because the data need to be gathered prior to being sent, I think triggering whenever any of the requested fields has new data will be computationally challenging for the CDS.
On Wed, Oct 2, 2019 at 7:35 AM Webb Pinner [email protected] wrote:
** Disclaimer - I do not claim to fully understand the inner workings of OpenRVDAS, what I'm requesting may already implemented or completely impossible to implement
The current cached data server serves websocket-connected clients with updated data from the data streams that the clients have subscribed to. The updated are provided synchronously at a default rate of 1Hz. The updates arrive regardless if the underlying data has changed since the previous update was sent.
This creates two scenarios that I feel should be avoided if possible:
- Connected clients receive "new" data even if the data hasn't changed.
- The minimum refresh interval of the client is controlled by the broadcast interval of the cache data server and not by changes in the underlying data.
Recommend refactoring the cache-data-server to send connected clients with updated data when the underlying data changes vs at a server-defined update rate.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/davidpablocohn/openrvdas/issues/164?email_source=notifications&email_token=AFO7V3WEQIIZ6PAG3D22JJDQMSWTXA5CNFSM4I4XGVFKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HPEYBLA, or mute the thread https://github.com/notifications/unsubscribe-auth/AFO7V3XMISJMC47EW6KSYITQMSWTXANCNFSM4I4XGVFA .
Next 2 questions... Why wait for a "ready" message? If the client is subscribing to the feed, why should it need to state that it's ready.
What computation is the CDS doing that would make it challenging? Again... not entirely up-to-speed on the implementation but I was assuming the CDS is updating a table of values which it can instantaneously access when needing to push data to a client.
With the current CDS model you could basically forego the websocket/subscription mechanism and simply have a http end-piont that the client sends a request to via setInterval(callback, interval).
I hope this text doesn't come across as being overly-critical. It's just very different from how I thought the CDS would perform and is different from the how I've used websockets to develop real-time web-apps like the Sealog eventlogging system.
The ready message was implemented because I observed clients getting swamped and browsers becoming unresponsive by being overwhelmed by data updates,
On Fri, Oct 4, 2019 at 5:02 AM Webb Pinner [email protected] wrote:
Next 2 questions... Why wait for a "ready" message? If the client is subscribing to the feed, why should it need to state that it's ready.
What computation is the CDS doing that would make it challenging? Again... not entirely up-to-speed on the implementation but I was assuming the CDS is updating a table of values which it can instantaneously access when needing to push data to a client.
With the current CDS model you could basically forego the websocket/subscription mechanism and simply have a http end-piont that the client sends a request to via setInterval(callback, interval).
I hope this text doesn't come across as being overly-critical. It's just very different from how I thought the CDS would perform and is different from the how I've used websockets to develop real-time web-apps like the Sealog eventlogging system.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/davidpablocohn/openrvdas/issues/164?email_source=notifications&email_token=AFO7V3X4HVAOF4TEMI3NN5DQM4WD7A5CNFSM4I4XGVFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEALNTHI#issuecomment-538368413, or mute the thread https://github.com/notifications/unsubscribe-auth/AFO7V3VT2VAIVJLLOBTEH6LQM4WD7ANCNFSM4I4XGVFA .
So with the original model, were clients being sent everything in the CDS per update or just the changes?
i.e. The logger_runner for the POS/MV has an error... The websocket clients are sent the most recent messages for all logger_runners vs the websocket clients are sent just the new message from the POS/MV logger_runner.
I could see a browser being overwhelmed if it were sent all the data in the CDS for each update, but I'm surprised to hear a browser being overwhelmed by just the updated data.
Just the data since the last send. But the display widgets redraw each time they get an update, and if data are coming in quickly, e.g. at 20 Hz from the winch, they're being asked to redraw all N-thousand points with each new data point. Seemed prudent to let the client control how often it got updates.
I'd like to table this for the time being - really a bit overwhelmed trying to field things that I need to have done/fixed before RVTEC.
On Fri, Oct 4, 2019 at 7:30 AM Webb Pinner [email protected] wrote:
So with the original model, were clients being sent everything in the CDS per update or just the changes?
i.e. The logger_runner for the POS/MV has an error... The websocket clients are sent the most recent messages for all logger_runners vs the websocket clients are sent just the new message from the POS/MV logger_runner.
I could see a browser being overwhelmed if it were sent all the data in the CDS for each update, but I'm surprised to hear a browser being overwhelmed by just the updated data.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/davidpablocohn/openrvdas/issues/164?email_source=notifications&email_token=AFO7V3SKBOCTWGADIXO7TSTQM5HPXA5CNFSM4I4XGVFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAL2RNY#issuecomment-538421431, or mute the thread https://github.com/notifications/unsubscribe-auth/AFO7V3X5C7PLWXP4CHYPQILQM5HPXANCNFSM4I4XGVFA .
No worries... I do want to revisit this given the current implementation degrades the potential real-time server performance for what could be addressed at the client (i.e. set the client-code to save the new data to localstorage but only run the render code every x milliseconds).
I can see where async delivery could be bad, and even saw that happen during the early days on the NBP. It is fairly easy to overwhelm downstream processes. So I like the functionality that currently exists. I have tested pulling data at 20Hz (that's the fastest instrument we have on LMG), and it works OK. Seems like a reasonable mechanism. But I do agree that you should be free to drown yourself in data if you really want to do that, and I do agree that async in general is (from a purist standpoint) a desireable model, so maybe as an enhancement Pablo could add a new verb to the CDS's websocket parser. I suggest "firehose". (as in "drinking from the").
