fix(milvus): Enhanced Milvus VectorDB Instrumentation for Improved search Monitoring

[divyapathak24]

Fixes #2808

• [x] I have added tests that cover my changes.
• [x] If adding a new instrumentation or changing an existing one, I've added screenshots from some observability platform showing the change.
• [x] PR name follows conventional commits format: feat(instrumentation): ... or fix(instrumentation): ....
• [ ] (If applicable) I have updated the documentation accordingly.

New search attributes for single-vector searches

New search attributes for multi-vector searches

Search Events to log results:

---

[!IMPORTANT] Enhance Milvus VectorDB instrumentation with detailed search monitoring attributes and events, and add tests for single and multi-vector searches.

• Instrumentation Enhancements:
  • Add search duration and result status attributes in _wrap() in wrapper.py.
  • Log search result events in _add_search_result_events() in wrapper.py.
  • Add new search attributes like MILVUS_SEARCH_RADIUS, MILVUS_SEARCH_METRIC_TYPE, and MILVUS_SEARCH_INDEX_TYPE in semconv_ai/__init__.py.
• Testing:
  • Add test_search.py to test single and multi-vector searches, verifying span attributes and events.
  • Test different radius values to ensure varying result counts.
• Misc:
  • Import time module in wrapper.py for measuring search duration.

^{This description was created by for 3fe4f70605773bf0e3a583b587db3d0982f384db. It will automatically update as commits are pushed.}

[CLAassistant]

All committers have signed the CLA.

[divyapathak24]

Thanks @divyapathak24! Can you fix the broken CI?

Sure. I will fix them @nirga

[divyapathak24]

@nirga I think the correct approach is to first add the new semantic conventions-ai and publish a new version of the package (say 0.4.4). Only after that can I safely use those attributes in Milvus—otherwise, the tests will fail because the new attributes won’t be available to milvus yet right?

[divyapathak24]

Hey @divyapathak24,

Thanks for submitting this one! please look at my comments - if you agree with them, please adjust (all occurrences of the things i mentioned, i didn't comment on all of them), otherwise let's discuss it here :)

Hi @galkleinman The explicit attributes like MILVUS_SEARCH_DURATION_IN_MS, MILVUS_SEARCH_RESULT_STATUS, RADIUS, and METRIC_TYPE make Milvus search insights directly accessible, avoiding post-processing and assumptions about span internals.

[galkleinman]

Hey @divyapathak24, Thanks for submitting this one! please look at my comments - if you agree with them, please adjust (all occurrences of the things i mentioned, i didn't comment on all of them), otherwise let's discuss it here :)

Hi @galkleinman The explicit attributes like MILVUS_SEARCH_DURATION_IN_MS, MILVUS_SEARCH_RESULT_STATUS, RADIUS, and METRIC_TYPE make Milvus search insights directly accessible, avoiding post-processing and assumptions about span internals.

Hey again @divyapathak24,

It isn't span internal, an no additional post-processing is needed, it's just the way OTEL works afaik... RADIUS and METRIC_TYPE are indeed unique to Milvus. But the duration of the action (in this case search) implies for all of the spans, and therefore implemented OOTB by start_time, end_time and duration... So it feels (to me at least) like a data duplication and deviation from the standard/protocol.

@nirga wdyt?

[divyapathak24]

Hey @divyapathak24, Thanks for submitting this one! please look at my comments - if you agree with them, please adjust (all occurrences of the things i mentioned, i didn't comment on all of them), otherwise let's discuss it here :)

Hi @galkleinman The explicit attributes like MILVUS_SEARCH_DURATION_IN_MS, MILVUS_SEARCH_RESULT_STATUS, RADIUS, and METRIC_TYPE make Milvus search insights directly accessible, avoiding post-processing and assumptions about span internals.

Hey again @divyapathak24,

It isn't span internal, an no additional post-processing is needed, it's just the way OTEL works afaik... RADIUS and METRIC_TYPE are indeed unique to Milvus. But the duration of the action (in this case search) implies for all of the spans, and therefore implemented OOTB by start_time, end_time and duration... So it feels (to me at least) like a data duplication and deviation from the standard/protocol.

@nirga wdyt?

Hi @galkleinman @nirga,

Thanks for the feedback! I will remove the redundant attributes. Just wanted to confirm if everything else looks good to you both. Let me know if you have any other suggestions for other attributes.

[divyapathak24]

@nirga @galkleinman Can you please review the changes? I have removed redundant attributes and pushed changes and updated test cases accordingly. cc: @hk-bmi

[nirga]

Thanks @divyapathak24 - can you sign the CLA? - https://github.com/traceloop/openllmetry/pull/2815#issuecomment-2802645987

[divyapathak24]

@nirga done. Can you re-trigger the build?

[divyapathak24]

@nirga done. Can you re-trigger the build?

@nirga @galkleinman It seems my test cases are failing as it is not able to find the newly added attributes in semconv ai. How do I resolve this? By changing semconv-ai package version from 0.4.3 to 0.4.4 ?

[nirga]

Yes @divyapathak24 :)

[divyapathak24]

@nirga @galkleinman I have opened another PR https://github.com/traceloop/openllmetry/pull/2883 for semconv version update which has updated search attributes.