google-gemini
Instrument Gemini CLI to Report Inference Metrics to CCPA
What would you like to be added?
see b/458304468
To enable comprehensive monitoring and acceptance rate calculation for Gemini CLI usage within GCA Cloud Logging, we request that Gemini CLI be instrumented to send inference and interaction metrics to CCPA.
Requirements:
Endpoint: Metrics should be sent to the RecordCodeAssistMetrics RPC method in the CCPA service (cloudcode.proto). Metric Events: ConversationOffered: This event should be sent for every unique inference response provided to the user by the Gemini CLI. Each event must be associated with the unique traceID (e.g., from the GenerateContentResponse.trace_id in the API response). ConversationInteraction: This event should be sent when a user interacts with or accepts a suggestion/response from the Gemini CLI. This event must also be linked to the same unique traceID as the corresponding ConversationOffered event. The key interaction to capture is "acceptance". Unique Identification: It is crucial that both ConversationOffered and ConversationInteraction events for the same inference result share the same unique traceID to allow CCPA and Cloud Logging to correlate them. Client Information: The ClientMetadata within the RecordCodeAssistMetricsRequest should be populated, ensuring ide_type is set to GEMINI_CLI.
Why is this needed?
Business Justification:
This instrumentation is essential for General Availability (GA) Readiness for Generate Content APIs.
Technical Details:
The Gemini CLI needs to capture the traceID returned by the underlying Generate Content API call. Upon displaying a result to the user, an ConversationOffered metric should be constructed and sent to CCPA, including the traceID. When the user accepts or acts upon the result, a ConversationInteraction metric should be constructed and sent to CCPA, including the same traceID.
Additional context
No response
Part one of two: https://github.com/google-gemini/gemini-cli/pull/15024
Now working through how we'll capture ConversationInteraction event from the CLI.
