The idea is to improve the CTF traces management. First, we update the test inputs (are old, contains old trace types which that do not exist anymore in the manifest file…) and reactivate 2 tests. Currently, the CtfTraceEventSource is file-based (zip file containing the CTF traces). We would like the CtfTraceEventSource to have a live mode (like ETWTraceEventSource). To achieve this, we introduce interfaces (contracts) that must be implemented by component(s) (internal, zip-based, or external) to have their CTF events decoded and translated to TraceEvent object.

Aug 03 '17 14:08 gleocadie

Codecov Report

Merging #340 into master will increase coverage by 1.54%. The diff coverage is 92.73%.

@@            Coverage Diff             @@
##           master     #340      +/-   ##
==========================================
+ Coverage   17.52%   19.06%   +1.54%     
==========================================
  Files         221      229       +8     
  Lines      126188   126592     +404     
  Branches    12173    12192      +19     
==========================================
+ Hits        22116    24140    +2024     
+ Misses     103202   101493    -1709     
- Partials      870      959      +89

Flag	Coverage Δ
#2017	`19.06% <92.73%> (+1.54%)`	:arrow_up:
#Debug	`19.06% <92.73%> (+1.54%)`	:arrow_up:

Impacted Files	Coverage Δ
src/TraceEvent/Ctf/CtfMetadataLegacyParser.cs	`83.82% <0%> (+83.82%)`	:arrow_up:
src/TraceEvent/Ctf/CtfMetadata.cs	`82.6% <0%> (+82.6%)`	:arrow_up:
.../TraceEvent/Ctf/ZippedEvent/ZippedCtfEventTrace.cs	`100% <100%> (ø)`
...fTracing.Tests/CtfMetadataTypes/CtfIntegerTests.cs	`100% <100%> (ø)`
...c/TraceEvent/Ctf/CtfTracing.Tests/CtfTraceTests.cs	`100% <100%> (+100%)`	:arrow_up:
...nt/Ctf/CtfMetadataTypes/CtfIntegerReaderFactory.cs	`100% <100%> (ø)`
src/TraceEvent/Ctf/CtfReader.cs	`70.31% <66.66%> (+70.31%)`	:arrow_up:
src/TraceEvent/Ctf/CtfTraceEventSource.cs	`73.23% <70.9%> (+73.23%)`	:arrow_up:
...raceEvent/Ctf/CtfMetadataTypes/CtfMetadataTypes.cs	`64.28% <75%> (ø)`
src/TraceEvent/Ctf/CtfMetadataTypes/CtfInteger.cs	`80.64% <80.64%> (ø)`
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update dcece49...29431ab. Read the comment docs.

Aug 03 '17 14:08 codecov-io

@leculver FYI

Sep 19 '17 18:09 vancem

@gleocadie, thank you very much for your patience on this PR. I appreciate you submitting this change.

I've had some time to review it, and wanted to make sure I understand the goal. My understanding of your ultimate goal (which I believe we share) is to support live LTTng sessions that are created via HTTP from the machine that is producing the events. Is this right?

In terms of this PR, it looks like this is a first cut at the factoring that will be required to have TraceEvent support live sessions. Namely, instead of taking ZipEntry objects, the APIs take objects that implement the ICtf* interfaces. From what I can see the only implementation of these ICtf* interfaces is by ZippedCtf* objects, so I suspect that this is mostly about laying groundwork for future changes where we actually support listening to an HTTP stream from LTTng. Have I read your code changes correctly, or are there functionality changes as well that I missed?

I suspect that the way you're currently using this is to produce short traces and then immediately consume them while a new trace is being collected. Is that right?

Oct 17 '17 23:10 brianrob

Hello @brianrob , sorry for the long time with no response. We add pile of stuff to do before getting back to LTTng & TraceEvent. As discussed at the Tracing summit, I will send you the code (github link) on how we use the LTTng live session in our code. I'm currently working on moving the MVP into an production-ready application. I also tackled the performance issues present in this PR. I expect getting something by the end of the month. Greg

Feb 10 '18 13:02 gleocadie

Question for you all (@brianrob, @gleocadie , @vancem) is there a known solution or example on how I would capture LTTng in a live session using NetCore?

For my scenario I have one NetCore application running on Linux writing Lttng using EventSource. And I'm hoping to have another NetCore application listening to those events. It's the same setup I have running in production using ETW on Windows.

Any suggestions?

Feb 27 '18 20:02 justinamiller

Adding @leculver, who did the initial implementation of CTFTraceEventSource.

I know that there is some support in LTTng for live sessions. Making sure that worked in CTFTraceEventSource was a low priority. so it is probably within 'striking' distance, but will require some work (which is not currently planned, but you might consider doing yourself).

Note that are current thinking we recognize a major problem with LTTng, namely that the value of the standard is really only high if you ALREADY have tools that consume LTTng that ALREADY exist. Unfortunately we wan our tools to work on Windows (which does not have LTTng) and Linux (which does not have ETW). Now TraceEvent sort of papers over some of the differences, but our current strategy is to provide a OS INDEPDENDENT mechanism (that we call EventPipe), which allow people to control and receive events from the .NET Runtime on both windows or Linux. What is nice about this is that you don't have fidelity losses at the OS level (you have to pick only features common to LTTng and ETW), and so things tend to 'just work' (but you DO lose OS specific events).

We DO have plans for EventPipe to support live sessions, and TraceEvent already has support for EventPipe (see EventPipeEventSource). That is probably ideal for you, but it does not exist yet.

It may be worthwhile for you to participate in the EventPipe work that you care about. @brianrob is the owner of that feature. It would still probably be months before things are done (we have other work as well), and you may not be interested, but I thought I would throw it out there.

Feb 27 '18 21:02 vancem

@gleocadie has actually made this work. I believe that he is currently using a forked TraceEvent that has the interfaces in this PR and then an implementation that can connect to the listener exposed via LTTng live sessions. This is likely to be the most expedient option that you have, and @gleocadie may be able to help point you in the right direction if you want to go this way.

At the same time, as @vancem points out, EventPipe is the way that we'd like to expose tracing in a platform agnostic way. LTTng is valuable in that it allows for interleaving of runtime/framework events into a platform-level logger. You get a machine-wide trace as well. If you are interested in just application-level telemetry and/or runtime specific telemetry, then EventPipe is something that is likely the right direction for you, as it is cross-platform and will provide tracing data from the runtime, libraries, and application. It is true, however, that live session support does not exist yet, and it will be some time before we get to this. If you are interested in participating in this effort, EventPipe development is ongoing at http://github.com/dotnet/coreclr.

Feb 27 '18 22:02 brianrob

Also, @gleocadie, thanks for the status update!

Feb 27 '18 22:02 brianrob

This issue is probably dead by now, but I'd just like to leave a comment to state that a live CtfTraceEventSource would still be useful. The recommended approach with EventPipe would not cover the original poster's use case, which is similar to mine - as I am simply using ETW for centralized application/services logging, not for performance tracing.

The main problem for me is that an EventPipe session needs to be started with the process id of a running process. I cannot just let it sit there and look for events no matter whether any processes related to the specified providers are running.

Mar 16 '22 23:03 kwaclaw

Improve ctf trace event source

Codecov Report