simple-binary-encoding How to know an optional group's entry number

Take CME's MDInstrumentDefinitionFX63 (https://www.cmegroup.com/confluence/display/EPICSANDBOX/CME+Globex+EBS+Market+-+Market+Data+Message+Specification) as an example. When user receives a MDInstrumentDefinitionFX63 message, how to know whether there is an NoInstAttrib group and how many entries?

This is a received msg: {"Name": "MDInstrumentDefinitionFX63", "sbeTemplateId": 63, "MatchEventIndicator": ["endOfEvent"], "TotNumReports": 1, "SecurityUpdateAction": "Add", "LastUpdateTime": 1638843903039872193, "MDSecurityTradingStatus": "ReadyToTrade", "ApplID": 533, "MarketSegmentID": 0, "UnderlyingProduct": 1, "SecurityExchange": "CME", "SecurityGroup": "UT", "Asset": "FX", "Symbol": "USD/SGD", "SecurityID": 12345, "SecurityType": "6", "CFICode": "CFI001", "Currency": "USD", "SettlCurrency": "USD", "PriceQuoteCurrency": "USD", "MatchAlgorithm": "F", "MinTradeVol": 1000, "MaxTradeVol": 10000, "MinPriceIncrement": {"mantissa": 1000000000}, "DisplayFactor": {"mantissa": 4}, "PricePrecision": 4, "UnitOfMeasure": "200000", "UnitOfMeasureQty": {"mantissa": 100000}, "HighLimitPrice": {"mantissa": 20500000000}, "LowLimitPrice": {"mantissa": 10500000000}, "MaxPriceVariation": {"mantissa": 22000000000}, "UserDefinedInstrument": "N", "FinancialInstrumentFullName": "FinancialInstrumentFullName", "FXCurrencySymbol": "USD/SGD", "SettlType": "0", "InterveningDays": 2, "FXBenchmarkRateFix": "BenchmarkRate Desc", "RateSource": "EBS", "FixRateLocalTime": "12:00:00", "FixRateLocalTimeZone": "UTC", "MinQuoteLife": 600000000, "MaxPriceDiscretionOffset": {"mantissa": 0}, "InstrumentGUID": 10001, "MaturityMonthYear": {"year": 2021, "month": 12, "day": 9, "week": 0}, "SettlementLocale": "LONDON", "NoEvents": [{"EventType": "Activation", "EventTime": 0}], "NoMDFeedTypes": [{"MDFeedType": "FUT", "MarketDepth": 5}], "NoInstAttrib": [{"InstAttribValue": ["electronicMatchEligible","oTCEligible","eFIXInstrument"]}], "NoLotTypeRules": [{"LotType": 2, "MinLotSize": 1000}, {"LotType": 5, "MinLotSize": 100}], "NoTradingSessions": [{"TradeDate": 26119, "SettlDate": 26121, "MaturityDate": 35018, "SecurityAltID": "XS123456789"}]}

There is one NoInstAttrib entry . My code to access it to this effect:

auto& noInstAttrib = instdef.noInstAttrib(); auto noInstAttribCnt = noInstAttrib.count(); if (1 == noInstAttribCnt) { auto& noInstAttrib1 = noInstAttrib.next(); publishTagVal(m_staticDataFlids.m_InstAttribTypeFlid, noInstAttrib1.instAttribType()); publishTagVal(m_staticDataFlids.m_InstAttribValueFlid, noInstAttrib1.instAttribValue().rawValue()); } else { Ft_WARN << "MDInstrumentDefinitionFX63 repeating group NoInstAttrib count is not 1 but " << noInstAttribCnt << endl; }

However my system complains: MDInstrumentDefinitionFX63 repeating group NoInstAttrib count is not 1 but 0

Dec 07 '21 02:12 chengm349

Do you skip/process all repeating groups prior to NoInstAttrib?

...
instdef.noEvents().skip(); 
// or
auto group = instdef.noEvents();
while (group.hasNext()) {
   group.next();
   ...
   // process group
}

// now your code for noInstAttrib

https://github.com/real-logic/simple-binary-encoding/wiki/Design-Principles

Dec 07 '21 09:12 ksergey

I tried both (skip or while loop):

//instdef.noEvents().skip();
auto& noEvents = instdef.noEvents();
auto noEventsCnt = noEvents.count();
if (0 == noEventsCnt) Ft_TRACE << "Correct" << endl;
while (noEvents.hasNext()) { noEvents.next();}
//instdef.noMDFeedTypes().skip();
auto& noMDFeedTypes = instdef.noMDFeedTypes();
auto noMDFeedTypesCnt = noMDFeedTypes.count();
if (0 == noMDFeedTypesCnt) Ft_TRACE << "Not Correct" << endl;
while (noMDFeedTypes.hasNext()) { noMDFeedTypes.next(); }
auto& noInstAttrib = instdef.noInstAttrib();
auto noInstAttribCnt = noInstAttrib.count();
if (1 == noInstAttribCnt) {
	auto& noInstAttrib1 = noInstAttrib.next();
	publishTagVal(m_staticDataFlids.m_InstAttribTypeFlid, noInstAttrib1.instAttribType());
	publishTagVal(m_staticDataFlids.m_InstAttribValueFlid, 
    noInstAttrib1.instAttribValue().rawValue());
} else {
	Ft_WARN << "MDInstrumentDefinitionFX63 repeating group NoInstAttrib count is not 1 but " << noInstAttribCnt << endl;
	while (noInstAttrib.hasNext()) {
		Ft_TRACE << "Hi" << endl;
		noInstAttrib.next();
	}
}

auto& noLotTypeRules = instdef.noLotTypeRules();
while (noLotTypeRules.hasNext()) {
	Ft_TRACE << "noLotTypeRules" << endl;
	noLotTypeRules.next();
}

auto& noTradingSessions = instdef.noTradingSessions();
while (noTradingSessions.hasNext()) {
	Ft_TRACE << "noTradingSessions" << endl;
	noTradingSessions.next();
}

Some log content: [17:58:26 423790] [105882] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 691: Correct [17:58:26 423802] [105882] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 696: Not Correct [17:58:26 423812] [105882] [warning] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 705: MDInstrumentDefinitionFX63 repeating group NoInstAttrib count is not 1 but 0

Dec 08 '21 10:12 chengm349

What confused me is that within the same function

void CMESubscriptionManager::publishStaticData(...) { ..... auto& instrumentData = const_cast<InstrumentCIdTuple&>(getInstrumentData(sym)); auto& instdef = std::get<0>(instrumentData); Ft_TRACE << sbeFixToString(instdef) << endl; // please refer to previous whole message printing which is correct ... //instdef.noEvents().skip(); auto& noEvents = instdef.noEvents(); while (noEvents.hasNext()) { Ft_TRACE << "noEvents" << endl; noEvents.next(); } //instdef.noMDFeedTypes().skip(); auto& noMDFeedTypes = instdef.noMDFeedTypes(); while (noMDFeedTypes.hasNext()) { Ft_TRACE << "noMDFeedTypes" << endl; noMDFeedTypes.next(); } std::ostringstream ss; instdef.noInstAttrib().forEach( [&](auto& noInstAttrib) { ss << noInstAttrib; }); Ft_TRACE << ss.str() << endl; // in my log file : [20:23:31 300600] [58538] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 708: // So it's empty /*auto& noInstAttrib = instdef.noInstAttrib(); auto noInstAttribCnt = noInstAttrib.count(); if (1 == noInstAttribCnt) { auto& noInstAttrib1 = noInstAttrib.next(); publishTagVal(m_staticDataFlids.m_InstAttribTypeFlid, noInstAttrib1.instAttribType()); publishTagVal(m_staticDataFlids.m_InstAttribValueFlid, noInstAttrib1.instAttribValue().rawValue()); } else { Ft_WARN << "MDInstrumentDefinitionFX63 repeating group NoInstAttrib count is not 1 but " << noInstAttribCnt << endl; }*/ }

really couldn't figure out where could be wrong.

Dec 09 '21 12:12 chengm349

Today's last try:

void CMESubscriptionManager::publishStaticData(...)
{
    ......
    std::ostringstream ss;
    Ft_TRACE << sbeFixToString(instdef) << endl;
    instdef.noEvents().forEach(
    			[&](auto& noEvents)
    			{
    				ss << noEvents;
    			});
    Ft_TRACE << ss.str() << endl;
    ss.str("");
    instdef.noMDFeedTypes().forEach(
				[&](auto& noMDFeedTypes)
				{
					ss << noMDFeedTypes;
				});
	Ft_TRACE << ss.str() << endl;
	ss.str("");
	instdef.noInstAttrib().forEach(
			[&](auto& noInstAttrib)
			{
				ss << noInstAttrib;
			});
	Ft_TRACE << ss.str() << endl;
	ss.str("");
	instdef.noLotTypeRules().forEach(
			[&](auto& noLotTypeRules)
			{
				ss << noLotTypeRules;
			});
	Ft_TRACE << ss.str() << endl;
	ss.str("");
	instdef.noTradingSessions().forEach(
			[&](auto& noTradingSessions)
			{
				ss << noTradingSessions;
			});
	Ft_TRACE << ss.str() << endl;
	ss.str("");
       ......
}

My log file content:

[21:13:46 276763] [64724] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 691: {"Name": "MDInstrumentDefinitionFX63", "sbeTemplateId": 63, "MatchEventIndicator": ["endOfEvent"], "TotNumReports": 1, "SecurityUpdateAction": "Add", "LastUpdateTime": 1639055626275869447, "MDSecurityTradingStatus": "ReadyToTrade", "ApplID": 533, "MarketSegmentID": 0, "UnderlyingProduct": 1, "SecurityExchange": "CME", "SecurityGroup": "UT", "Asset": "FX", "Symbol": "USD/SGD", "SecurityID": 12345, "SecurityType": "6", "CFICode": "CFI001", "Currency": "USD", "SettlCurrency": "USD", "PriceQuoteCurrency": "USD", "MatchAlgorithm": "F", "MinTradeVol": 1000, "MaxTradeVol": 10000, "MinPriceIncrement": {"mantissa": 1000000000}, "DisplayFactor": {"mantissa": 4}, "PricePrecision": 4, "UnitOfMeasure": "200000", "UnitOfMeasureQty": {"mantissa": 100000}, "HighLimitPrice": {"mantissa": 20500000000}, "LowLimitPrice": {"mantissa": 10500000000}, "MaxPriceVariation": {"mantissa": 22000000000}, "UserDefinedInstrument": "N", "FinancialInstrumentFullName": "FinancialInstrumentFullName", "FXCurrencySymbol": "USD/SGD", "SettlType": "0", "InterveningDays": 2, "FXBenchmarkRateFix": "BenchmarkRate Desc", "RateSource": "EBS", "FixRateLocalTime": "12:00:00", "FixRateLocalTimeZone": "UTC", "MinQuoteLife": 600000000, "MaxPriceDiscretionOffset": {"mantissa": 0}, "InstrumentGUID": 10001, "MaturityMonthYear": {"year": 2021, "month": 12, "day": 11, "week": 0}, "SettlementLocale": "LONDON", "NoEvents": [{"EventType": "Activation", "EventTime": 0}], "NoMDFeedTypes": [{"MDFeedType": "FUT", "MarketDepth": 5}], "NoInstAttrib": [{"InstAttribValue": ["electronicMatchEligible","oTCEligible","eFIXInstrument"]}], "NoLotTypeRules": [{"LotType": 2, "MinLotSize": 1000}, {"LotType": 5, "MinLotSize": 100}], "NoTradingSessions": [{"TradeDate": 26121, "SettlDate": 26123, "MaturityDate": 35020, "SecurityAltID": "XS123456789"}]} [21:13:46 276795] [64724] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 697: [21:13:46 276813] [64724] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 704: [21:13:46 276824] [64724] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 711: [21:13:46 276833] [64724] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 718: [21:13:46 276843] [64724] [trace] void CMESubscriptionManager::publishStaticData(const FtSymbol&, SymSession&) - 725:

Dec 09 '21 13:12 chengm349

Btw,my code can get non-group tag value properly.

Dec 09 '21 14:12 chengm349

auto& instrumentData = const_cast<InstrumentCIdTuple&>(getInstrumentData(sym));
auto& instdef = std::get<0>(instrumentData);

Looks like you try to reread a sbe object. If you once processed a message (repeating groups and variable length fields) you need to recreate sbe object. You can copy a sbe object for processing or you can call sbeRewind() before process message again.

https://github.com/real-logic/simple-binary-encoding/blob/079999db64d9f4c4b7348b2872394c5602da81c7/sbe-tool/src/test/cpp/CodeGenTest.cpp#L1070-L1072

Dec 09 '21 15:12 ksergey

Really appreciate your effort to guide. I am doing UT so I create a message and later to simulate I receive the message from network. I downloaded the tool in Jul and that version of 23.1 does not have sbeRewind(). Now I am using 25.2 and I can see sbeRewind(). So it's better. Thanks to achieve the side-effect.

Some part of my previous test code had some shortcut and some did not. So sbeRewind() does work for the shortcut message (created first and directly apply sbeRewind() to it to the effect) but sbeRewind() does not work for the rest which are very close to actual situation:

I created a MDInstrumentDefinitionFX63 message with some optional groups shown in previous pasted in the beginning of this ticket and copied to a buffer for later network receiving simulation. The created message.encodeLength() returned 385 while MDInstrumentDefinitionFX63 sbeBlockLength is only 309 which seems ok to me (?).
Copied memcpy(target, source, 385) ;
build MDInstrumentDefinitionFX63 message from target again by doing:

mktdata::MDInstrumentDefinitionFX63 message{};
message.wrapForDecode(target, 0, 385, sbeheader.getVersion(), 8184); // I used 385 as actingBlockLength so-called in lib
app.onMessage(message, target, 385, ...);

my app class onMessage() printed the message as its first ocde line: Ft_TRACE << sbeFixToString(message) << endl;
sbeFixToString is just a wrapper to call output streamer .
8184 is the target buffer size (linux BUF_SIZ minus a message's header part) which is the effective space for a message itself.
we can see why sbeRewind() does not work is bcos there is no any optional group at first beginning.

{"Name": "MDInstrumentDefinitionFX63", "sbeTemplateId": 63, "MatchEventIndicator": ["endOfEvent"], "TotNumReports": 1, "SecurityUpdateAction": "Add", "LastUpdateTime": 1639129594753495548, "MDSecurityTradingStatus": "ReadyToTrade", "ApplID": 533, "MarketSegmentID": 0, "UnderlyingProduct": 1, "SecurityExchange": "CME", "SecurityGroup": "UT", "Asset": "FX", "Symbol": "USD/SGD", "SecurityID": 12345, "SecurityType": "6", "CFICode": "CFI001", "Currency": "USD", "SettlCurrency": "USD", "PriceQuoteCurrency": "USD", "MatchAlgorithm": "F", "MinTradeVol": 1000, "MaxTradeVol": 10000, "MinPriceIncrement": {"mantissa": 1000000000}, "DisplayFactor": {"mantissa": 4}, "PricePrecision": 4, "UnitOfMeasure": "200000", "UnitOfMeasureQty": {"mantissa": 100000}, "HighLimitPrice": {"mantissa": 20500000000}, "LowLimitPrice": {"mantissa": 10500000000}, "MaxPriceVariation": {"mantissa": 22000000000}, "UserDefinedInstrument": "N", "FinancialInstrumentFullName": "FinancialInstrumentFullName", "FXCurrencySymbol": "USD/SGD", "SettlType": "0", "InterveningDays": 2, "FXBenchmarkRateFix": "BenchmarkRate Desc", "RateSource": "EBS", "FixRateLocalTime": "12:00:00", "FixRateLocalTimeZone": "UTC", "MinQuoteLife": 600000000, "MaxPriceDiscretionOffset": {"mantissa": 0}, "InstrumentGUID": 10001, "MaturityMonthYear": {"year": 2021, "month": 12, "day": 12, "week": 0}, "SettlementLocale": "LONDON", "NoEvents": [], "NoMDFeedTypes": [], "NoInstAttrib": [], "NoLotTypeRules": [], "NoTradingSessions": []}

Dec 10 '21 09:12 chengm349

I did further test.

got a 2K buffer.
built a MDInstrumentDefinitionFX63 message on it by calling:

messageOriginal.wrapAndApplyHeader(buffer, 0, bufsize);
a lot of function calls on messageOriginal to built the message content including optional groups.
Ft_TRACE << sbeFixToString(messageOriginal) << endl; and I could see correct content.

if I called message.wrapAndApplyHeader(buffer, 0, bufsize) again and Ft_TRACE << sbeFixToString(message) << endl; Everything was perfect and message content was exactly the same as messageOriginal.
However if I called message.wrapForDecode(messageOriginal.buffer(), 0, messageOriginal.encodedLength() , messageOriginal.actingVersion(), messageOriginal.bufferLength()); Then I got error std::runtime_error: unknown value for enum SecurityTradingStatus [E103]
If I called message.wrapForDecode(buffer, sizeof(SbeHeader), messageOriginal.encodedLength() , messageOriginal.actingVersion(), messageOriginal.bufferLength()); everything was ok also.

Dec 12 '21 09:12 chengm349

The lib should be ok in general. I solved my problem. But I leave the ticket for some other people to close.

Step 4 simulate my production code.

message.wrapForDecode(messageOriginal.buffer(), 0, messageOriginal.encodedLength() , messageOriginal.actingVersion(), messageOriginal.bufferLength());

My previous production code to this effect.

Read incoming message starting from Simple Binary Encoding Header into a pBuffer.
pBuffer += sizeof(SbeHeader)
- mktdata::MDInstrumentDefinitionFX63 message{};
- message.wrapForDecode(pBuffer, 0, 377, sbeheader.getVersion(), 8184); //8184 does not matter I believe since the buffer is big enough
- app.onMessage(message, pBuffer, 377, ...);
The above code does not work. Now I changed to:

no more pBuffer += sizeof(SbeHeader). So still point to the beginning
mktdata::MDInstrumentDefinitionFX63 message{};
- message.wrapAndApplyHeader(pBuffer, 0, 8184);
- app.onMessage(message, pBuffer, 377, ...);

Now I got a happy result.

Dec 13 '21 02:12 chengm349