cloud-graphite-scripts Unable to use Python example

Hi there,

I'm trying to use the Python example to get started with Grafana and Graphtie, but I struggle with some of the basics.

What I've done (Python)

I've registered a new Grafana 30 day trial account
I've logged in and set Graphite as my default data source with the default settings
I generated an Editor API key
I downloaded the Python example to my PC and added my API key
I then tested the script by adding the following endpoints:

https://mfcsstest.hosted-metrics.grafana.net --> This results in Exception: {"code": "MethodNotAllowed", "message": "POST is not allowed"}

https://graphite-us-central1.grafana.net/metrics --> This results in Exception: "invalid authentication credentials"

https://mfcsstest.grafana.net/api/dashboards/home --> This results in Exception: {"message":"Not found"}

I'm unsure how to proceed from here - am I using an incorrect URL, or am I missing something else?

UPDATE: I seem to face similar struggles running the main.sh script as well. Could it be that the scripts are outdated vs. the latest syntax of the Grafana hosted cloud solutions?

Jul 23 '20 19:07 MatinF

Will handle in support ticket.

Jul 23 '20 20:07 adeverteuil

For future reference and searches, please see this documentation page to find the right URL and Authentication method for you: https://grafana.com/docs/grafana-cloud/metrics/graphite/http-api/#authentication

Also, please make sure not to confuse Grafana API keys with Grafana.com API keys. To Authenticate with Cloud Metrics, you need to create an API key at https://grafana.com/orgs/your_org/api-keys. Replace your_org with your actual org short name.

Jul 24 '20 19:07 adeverteuil

Let's improve the readme

Jul 25 '20 07:07 Dieterbe

I am also having problems posting data to the 30-day trial account. I have generated an API key on grafana.com under Security / API keys.

Using the https://lipi.hosted-metrics.grafana.net/metrics URL I get this error:

{ "code": "NotFound", "message": "/metrics does not exist" }

Using the https://lipi.grafana.net/metrics URL (which my account page indicates -- notice the missing 'hosted-metrics' subdomain) I get this error:

<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.15.10</center>
</body>

Confused if I should use the 'hosted-metrics' part as the tutorial suggest, or not.

Also, if I use a bogus company-name, I get the "/metrics does not exist" error.

Aug 22 '20 23:08 lipi

@adeverteuil can you please take this on through support. @lipi when our support reaches out to you, please share the page on which it misses the subdomain.

Ps: I <3 NZ !

Aug 24 '20 18:08 Dieterbe

Sure! @lipi I couldn't find your account. Can you please open a support ticket here? https://grafana.com/profile/org#support You can link to this GitHub issue for context info in the body of your message.

Aug 24 '20 18:08 adeverteuil

I updated my previous comment with correct URL (lipi.grafana.net) and logged a ticket at grafana.com (ticket #15914).

Aug 26 '20 23:08 lipi

Did you ever get this to work? It seems the scripts here are outdated compared to docs on the grafana cloud website.

Edit:

I got this to work, you need to use the general URL provided on the grafana Graphite page, for example https://graphite-us-central1.grafana.net/metrics

Then in the requests post, you need to add an extra argument auth=(<User ID>, <API Key>)

You do not need to include the authentication header as shown in these example scripts.

Dec 29 '20 18:12 OscarVanL

I'm having a very similar issue, here is my story https://community.grafana.com/t/invalid-metric-data-received-org-id-cannot-be-0-error/51074/3

Example given in this page https://grafana.com/docs/grafana-cloud/metrics-graphite/http-api/ also not working. How can I solve invalid metric data received: org-id cannot be 0 issue?

Jul 29 '21 20:07 ssaltin

Hi @sonerx that is caused a bug that was deployed mistakenly, and the fix should appear soon. I've asked that engineering team to post an update here when it's fixed. until then, you can use the following workaround: just add these 2 fields to the payload:

"org_id": 1,
"mtype": "gauge",

Jul 29 '21 20:07 Dieterbe

Can you also provide an example for adding tags to the HHTP request? Thanks a lot in advance!

I tried something like "tags" = ["key=value"] and seems like metric is being published, I get {"Published":1} but I don't see the datapoint on Grafana Explorer. What am I doing wrong here?

Aug 08 '21 14:08 ssaltin

can you try "key:value"

Aug 09 '21 06:08 Dieterbe

can you try "key:value"

I tried the following

{
	"name": "test_metrics.dummy",
	"org_id": 1,
	"interval": 10,
	"value": 1,
	"mtype": "counter",
	"time": 1628605910,
        "tags": ["key:value"]
}

and it returns invalid metric data received: invalid input

Aug 10 '21 14:08 ssaltin

I'm using grafana free trail and I'm unable to get the results with basic authentication. Below is my code

import requests

import json

base_url = "https://username:[email protected]"

resp = requests.get(base_url + "/api/dashboards/home", verify=True)

data = resp.json()

print(data)

output:

{'message': 'invalid username or password', 'traceID': '67d0619ed46f6634'}

Apr 25 '22 16:04 santosh-burada

Hi @santosh-burada , thank you for using our free trial! I'm sorry but this is not the right context for your question. You are trying to make an HTTP API call on your Grafana instance hosted on Grafana Cloud. This issue is about making API calls to hosted Graphite.

It is possible that you are trying to use a Grafana Cloud API key, but you should create an API key in Grafana instead. Login to https://santoshburadalearning.grafana.net/org/apikeys and create an API key there.

If this doesn't solve your issue, please do not update this GitHub issue, but instead please use our Community support forum: https://community.grafana.com/c/grafanacloud/34 See also Grafana Cloud support options.

Apr 25 '22 18:04 adeverteuil

This API is really hard to use, as it doesn't emit correct error codes.

auth failures result in a 302 redirect instead of a 403
including the obsolete hosted-metrics part of the subdomain gets a timeout instead of a DNS resolution failure
omitting the graphite/ part of the path to the API endpoint emits a 403 instead of a 404

Compounding that, the grafana.net dashboard now talks about "cloud access policies" and "service accounts", with nary an API key to be seen, and it isn't clear how that relates to the sample code.

Edit: just to be explicit, the policy access token I am using has "metrics:write" permissions, and the service account token has "editor" permissions. The various auth headers I have tried are:

Basic auth with service account name and API token (base64 encoded)
Basic auth with service account API token name and API token (base64 encoded)
Auth header with service account name and API token (no encoding, separated by colon)
Auth header with service account API token name and API token (no encoding, separated by colon)
Auth header with just the service account API token
Auth header with just the cloud access policy API token

Of all of these, the bare service account API token is the only one that doesn't generate a 302 redirect to an interactive login page (it generates a 404 error instead).

The other fun part of trying to deal with this API is that it will happily throw back reams of HTML in error responses to a JSON POST request.

Switching to console-relay-ng won't help, since its config files need the same endpoint and API key as the direct HTTPS POST requests do, and the whole problem here is that the docs (and this repo) are as clear as mud when it comes to working out what those actually are.

(I filed a Grafana Cloud support ticket in addition to posting here)

Dec 02 '24 10:12 ncoghlan

Based on https://grafana.com/blog/2024/03/21/how-to-use-http-apis-to-send-metrics-and-logs-to-grafana-cloud/ I decided to completely ignore the candidate URLs mentioned elsewhere in the documentation, and instead use the raw AWS hosted URL given on the page for the graphite data source definition. This finally got me an actual 401 Unauthorized error, so it looks like attempting to use the user friendly aliases may be a key part of the problem with the graphite data ingestion docs.

Edit: with two additional changes from that Prometheus ingestion blog post I got it working. Final details:

URL: append /metrics to the raw AWS instance URL from the graphite data source page, ignore the prettier (supposed) aliases
User ID: use the numeric user ID from the graphite data source page
API key: under "Users and access" -> "Cloud access policies" in the dashboard, create a new policy with "metrics: write" permission, and use it to generate a token
Auth header: set the auth header to f"Bearer {USER_ID}:{APIKEY}"

Problematic documentation:

this repo, since the scripts have incorrect default URLs and no pointers on where to find the correct ones
https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-graphite/http-api/ since it references incorrect URLs and doesn't say where to find the user ID for the auth header
https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-graphite/data-ingestion/ since that doesn't say how to work out the full API key by combining the instance user ID with the cloud policy access token

Dec 03 '24 11:12 ncoghlan