cody
cody copied to clipboard
[WIP ]Adding the feature flag with tests
This PR introduces several key changes to improve the autocomplete functionality in Cody’s inline completions, primarily focused on user experience improvements, testing, and analysis.
Changes:
1. Feature Flag Implementation:
• Introduced a new feature flag to disable specific logic related to inline completions. This will allow controlled experimentation and toggling of the logic during development and A/B testing.
2. Autocomplete Behavior Improvements:
• Tested the current behavior of the autocomplete system when completing in the middle of a line.
• Addressed the issue where the model ignores the current line suffix and suggests conflicting completions.
3. UX Enhancements:
• Observed and documented cases where the model does not take the existing suffix into account.
• Considered implementing behavior from Cursor, where a different suffix shows the full line completion and replaces the line when accepted. This change might be added in a follow-up if necessary.
4. A/B Testing:
• Prepared for A/B testing of the changes to assess the user experience and determine the impact of the improvements.
5. Metric Analysis:
• Introduced methods to analyze key autocomplete metrics:
• CAR (Completion Acceptance Rate)
• wCAR (Weighted Completion Acceptance Rate)
• Number of accepted characters per user
• Latency metrics