Improvement of PutTimestampedSegment

[alkhwarizmi]

Dear all,

I propose to enhance the mdsobject function putTimestampedSegments(...).

In order to reduce latency, in my application I upload Segment data in chunks, with each chunk representing samples of several channels taken in a certain period of time (buffered continuous data acquisition). I call this chunk a "slice" (of the segment).

I noticed that if the last putTimestampedSegment operation does not fill the segment perfectly (there is more data in the slice than the whole segment can hold), it returns an error. This means that the user of the library needs to keep track of the filling status of the segment or manage the exception (which I would like to avoid because cause of further delay). Moreover, other functions e.g. putRow(...), do not need this special care because they handle themselves the situation internally.

Below there is a picture clarifying the idea and proposing how should it work. I see two possible way to integrate this into the current MDSPlus branch

• Creation of a new mdsobjects function putRows (note the ending s). This will avoid any regression at the price of a adding a new API function.

• Update of how putTimestampedSegment works. This will break any existing code relying on the reception of the exception in the mentioned case (which I cannot estimate if it could be a serious problem or not) but the API will not be expanded.

I believe the implementation of the new feature would simplify the use of the library from the user point of view. I tried to implement the modification myself but I have found the files TreeSegments.c and mdsdescrip.h too difficult to follow because of the macros.

Sincerely,

[zack-vii]

@alkhwarizmi : We have removed most macros in TreeSegments.c in PR #1202 . If you are willing to give it another chance, please do. :)

[drjdpowell]

Just a note: I use MDSplus (using mdsip called from LabVIEW) and put multiple rows in one node using this TDI script (which is modified from the PutRow FUN to call TreePutRow in a For Loop):

_TimeA=$3; _ValA=$4; _SegSize=$2; _RCErr=0; _nid=getnci($1,"NID_NUMBER"); FOR(_i=0;_i<SIZE(_TimeA);_i++) { _RCErr=TreeShr->TreePutRow(val(_nid),val(_SegSize[0]),quadword(_TimeA[_i]),descr(_ValA[_i])); }; _RCErr;

I can also put rows in multiple different nodes with this modification:

_TimeA=$3; _ValA=$4; _SegSize=$2; _RCErr=0; _nid=[]; _nid=getnci($1,"NID_NUMBER"); FOR(_i=0;_i<SIZE(_nid);_i++) { _RCErr=TreeShr->TreePutRow(val(_nid[_i]),val(_SegSize[0]),quadword(_TimeA[_i]),descr(_ValA[_i])); }; _RCErr;

I don't know how this compares to using putTimestampedSegment, but it eliminates the network latency of multiple PutRow calls.

[zack-vii]

You could simply allocate a multiple of the slice so you would always have a happy ending. If you cannot tell in advance how big your slice will be (why?) maybe it would be better to start a new segment. We could add some code that will finalize the last segment before adding the new one. This is done currently during cleanup- and compressDatafile. Finalizing just trims the data and time vector to rows_filled. Depending on the overhead I would say if at all we only provide the tdi fun that basically does putRow in a loop. This for loop of course could also be done in C.

[WhoBrokeTheBuild]

We are exploring alternatives to timestamped segments, which should address this functionality.