feat: add sysinfo metrics

[nikhilsinhaparseable]

collect CPU usage, memory usage of the server collect disk usage of the volume - data, staging, hot-tier add these metrics to Prometheus Metrics

export these metrics to cluster metrics API add the metrics to pmeta stream add the querier node's sysinfo metrics to pmeta and cluster metrics API

Summary by CodeRabbit

• New Features
  • Added comprehensive system metrics collection, including disk, memory, and CPU usage, with Prometheus integration for real-time monitoring.
• Improvements
  • Enhanced metrics endpoints to include system and disk resource usage details.
  • Streamlined internal log ingestion for improved performance.
  • Initialization processes now consistently start system and cluster metrics schedulers asynchronously.
• Refactor
  • Simplified stream writing logic by always writing records to disk, removing conditional checks based on stream type.
• Bug Fixes
  • Corrected naming and usage of metrics scheduler functions for consistency.

[coveralls]

Pull Request Test Coverage Report for Build 13025834726

Details

• 0 of 390 (0.0%) changed or added relevant lines in 8 files are covered.
• 4 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.2%) to 12.761%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/handlers/http/modal/ingest_server.rs	0	1	0.0%
src/handlers/http/modal/server.rs	0	1	0.0%
src/handlers/http/modal/query_server.rs	0	2	0.0%
src/correlation.rs	0	3	0.0%
src/handlers/http/cluster/mod.rs	0	11	0.0%
src/handlers/http/ingest.rs	0	14	0.0%
src/metrics/mod.rs	0	174	0.0%
src/metrics/prom_utils.rs	0	184	0.0%
<!--	Total:	0	390

Files with Coverage Reduction	New Missed Lines	%
src/handlers/http/cluster/mod.rs	1	0.0%
src/metrics/mod.rs	3	0.0%
<!--	Total:	4

Totals
Change from base Build 13025432081:	-0.2%
Covered Lines:	2477
Relevant Lines:	19411

---

💛 - Coveralls

[coderabbitai[bot]]

Walkthrough

This set of changes introduces a comprehensive system metrics collection and reporting framework. Disk, memory, and CPU usage metrics are now gathered periodically using a scheduler and exposed via Prometheus-compatible metrics. The codebase is refactored to support these new metrics, with additional data structures and metric types added. Metric collection is integrated into server initialization routines, ensuring that system and cluster metrics are always gathered and available. Some internal APIs are streamlined, such as the removal of the stream_type parameter from stream push operations and simplification of the internal ingestion logic.

Changes

File(s)	Change Summary
src/metrics/mod.rs	Adds scheduled system resource monitoring: new Prometheus metrics for disk, memory, and CPU; new structs for encapsulating metrics; scheduler for periodic collection; new async functions for collecting and updating metrics; error handling for metric collection.
src/metrics/prom_utils.rs	Refactors Metrics struct and methods to handle new disk, memory, and CPU metrics; introduces new metric types and async processing; adds methods for collecting and processing system metrics; removes old synchronous metric processing; updates metric aggregation logic.
src/handlers/http/cluster/mod.rs	Adds collect_all_metrics to cluster metrics collection; updates function signatures to async; ensures system metrics are collected before cluster metrics; updates scheduler logic and error handling.
src/handlers/http/ingest.rs	Simplifies internal ingestion by removing explicit event construction and schema fetching; directly calls log flattening and pushing with parsed JSON and custom fields.
src/handlers/http/modal/ingest_server.rs src/handlers/http/modal/server.rs src/handlers/http/modal/query_server.rs	Adds calls to init_system_metrics_scheduler in server initialization methods; updates imports; ensures both system and cluster metrics schedulers are always started asynchronously during server startup; removes conditional checks and related logging.
src/event/mod.rs src/parseable/streams.rs	Removes stream_type parameter from stream push operations and related method signatures; eliminates conditional logic based on stream type in Stream::push; updates test code accordingly.

Sequence Diagram(s)

sequenceDiagram
    participant Scheduler
    participant MetricsModule
    participant Prometheus
    participant Disk
    participant System

    loop Every minute
        Scheduler->>MetricsModule: collect_all_metrics()
        MetricsModule->>Disk: collect_disk_metrics()
        Disk-->>MetricsModule: Disk usage stats
        MetricsModule->>System: collect_system_metrics()
        System-->>MetricsModule: Memory & CPU stats
        MetricsModule->>Prometheus: Update metrics (disk, memory, CPU)
    end

sequenceDiagram
    participant Server
    participant MetricsModule

    Server->>MetricsModule: init_system_metrics_scheduler().await
    Server->>MetricsModule: init_cluster_metrics_scheduler().await
    MetricsModule-->>Server: Scheduler tasks started

Suggested labels

for next release

Suggested reviewers

• parmesant

Poem

A bunny hops through metrics new,
Disk and memory, CPU too!
With every tick, the system checks,
Scheduler runs—no need for specs.
Prometheus smiles, the gauges rise,
All the stats beneath the skies.
🐇✨ System health, now crystal clear!

[!NOTE]

⚡️ AI Code Reviews for VS Code, Cursor, Windsurf

CodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback. Learn more here.

---

[!NOTE]

⚡️ Faster reviews with caching

CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 30th. To opt out, configure Review - Disable Cache at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the Data Retention setting under your Organization Settings. Enjoy the performance boost—your workflow just got faster.

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.