Unable to use OpenTelemetry.propagation.fields

[YanivD]

Description of the bug

I'm implementing context propagation for aws_sdk instrumentation (for sqs / sns services). For that, I need to check how many fields are defined for the propagators (the propagated fields should be saved in the message attributes, which limited up to 10 attributes).

I found the method OpenTelemetry.propagation.fields that looks like what I'm searching for. When I use it I get the following error:

The issue

💥 opentelemetry-api-1.0.0/lib/opentelemetry/context/propagation/composite_text_map_propagator.rb:100:in `fields': stack level too deep (SystemStackError)

• Link to fields method
• I couldn't find any existing usage for OpenTelemetry.propagation.fields
• I'm new in Ruby language, so I might missing something trivial 🤷

---

Share details about your runtime

Operating system details: MacOS RUBY_ENGINE: "ruby" RUBY_VERSION: "3.0.2" RUBY_DESCRIPTION: "ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-darwin19]"

Share a simplified reproduction if possible

require 'bundler/inline'

gemfile(true) do
  source 'https://rubygems.org'

  gem 'opentelemetry-instrumentation-net_http'
  gem 'opentelemetry-api'
  gem 'opentelemetry-sdk'
end

require 'opentelemetry-instrumentation-net_http'
require 'opentelemetry-api'
require 'opentelemetry-sdk'


OpenTelemetry::SDK.configure do |c|
  c.use 'OpenTelemetry::Instrumentation::Net::HTTP'
end

puts OpenTelemetry.propagation.fields

[YanivD]

This is fields method. If I understand it correctly, it looks like it is calling itself recursively.

        # Returns the union of the propagation fields returned by the composed injectors
        # or propagators. If your carrier is reused, you should delete the fields returned
        # by this method before calling +inject+.
        #
        # @return [Array<String>] a list of fields that will be used by this propagator.
        def fields
          injectors = @injectors || @propagators
          injectors.flat_map(&fields).uniq
        end

[arielvalentin]

@YanivD This is indeed confusing.

I'll have to take a closer look next week to see why it seems to be in a recursive loop that causes the stack overflow.

The method that you shared looks like it is from the Composite propagator, which is delegating to the injectors/propagators that it wraps. It should not include itself in that list of propagators. If that we're the case then we would see this same exact error in production at GitHub since we use composite propagators to interop between OpenTracing and OpenTelemtry context propagation.

Other instrumentation uses the composite propagator inject/extract method. If I understand your constraint, it's that SQS message headers are limited to 10 fields.

Is that correct? What happens if it exceeds the 10 fields?

That may be a problem for interoperability between systems that use OTTrace that create a separate key/value pair for Baggage attributes. Those fields are also dynamically generated based on the contents of the Context field, so you likely will not see them as "fields" if someone is using that propagator. 🤔

Looks like I'll have to take a look at the other implementations to see how they handle them.

[YanivD]

Is that correct? What happens if it exceeds the 10 fields?

Correct. When setting more than 10 message attributes,sqs.send_message method raise the following error: Number of message attributes [12] exceeds the allowed maximum [10]. (Aws::SQS::Errors::InvalidParameterValue)

The sns.publish method finish without any error but the message never reach SNS service.

[YanivD]

Those fields are also dynamically generated based on the contents of the Context field, so you likely will not see them as "fields" if someone is using that propagator

As I can see in TextMapPropagator, fields are predefined and cannot be changed dynamically. What am I missing? How can I reproduce environment with dynamic Baggage attributes?

[arielvalentin]

@YanivD this is the OTTrace text map propagator:

https://github.com/open-telemetry/opentelemetry-ruby/blob/main/propagator/ottrace/test/opentelemetry/propagator/ottrace/text_map_propagator_test.rb

It adds the baggage fields dynamically and does not expose the keys using the fields method.

Given the constraint of the headers I wonder if the AWS SQS context propagation should be transmitted through the message body like we do with ActiveJob. That would impact any other auto instrumentation but I figured I'd mention it as an alternative.

I imagine there is probably a constraint here to make it work with XRay as well but I'm not at all familiar with it.

This feels too much like a weird edge case where folks who use OTTrace and baggage with SQS would have run into this issue already. I don't think we should worry too much about supporting it other than writing some docs saying "sorry not supported" 😬

I do think the use case im concerned about though is if end users aren't able to add their own headers because tracing headers are taking up slots.

Has this conversation not come up before in other Language SDKs?

[github-actions[bot]]

👋 This issue has been marked as stale because it has been open with no activity. You can: comment on the issue or remove the stale label to hold stale off for a while, add the keep label to hold stale off permanently, or do nothing. If you do nothing this issue will be closed eventually by the stale bot.