After upgrading our sentry to 24.9.0 we are facing sentry.auth.system: Trying to use `SystemToken` from non-internal IP

[balaG4046]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

Hi Team,

We have upgraded our sentry self hosted recently to 24.9.0, we are facing below issue intermittently, once after the restarting the container sentry-self-hosted-web-1 sentry came up, Error logs

15:06:20 [WARNING] django.request: Unauthorized: /api/0/projects/cds/evolio-masterdata-service/files/dsyms/ (status_code=401 request=<WSGIRequest: GET '/api/0/projects/cds/evolio-masterdata-service/files/dsyms/?debug_id=c80d19dc-4dc3-4765-9532-ea111a4c6061-e8be40ea'>) 15:06:20 [WARNING] django.request: Unauthorized: /api/0/projects/cds/evolio-masterdata-service/files/dsyms/ (status_code=401 request=<WSGIRequest: GET '/api/0/projects/cds/evolio-masterdata-service/files/dsyms/?debug_id=4702619c-5710-adc8-55f3-6ae683edbf5f-694b73fc'>) 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP

Kindly help us with the solution to resolve the issue.

Thanks!

Expected Result

it should work with any issue.

Actual Result

Please find the error logs-

15:06:20 [WARNING] django.request: Unauthorized: /api/0/projects/cds/evolio-masterdata-service/files/dsyms/ (status_code=401 request=<WSGIRequest: GET '/api/0/projects/cds/evolio-masterdata-service/files/dsyms/?debug_id=c80d19dc-4dc3-4765-9532-ea111a4c6061-e8be40ea'>) 15:06:20 [WARNING] django.request: Unauthorized: /api/0/projects/cds/evolio-masterdata-service/files/dsyms/ (status_code=401 request=<WSGIRequest: GET '/api/0/projects/cds/evolio-masterdata-service/files/dsyms/?debug_id=4702619c-5710-adc8-55f3-6ae683edbf5f-694b73fc'>) 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP 15:06:20 [ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP

Product Area

Issues

Link

No response

DSN

No response

Version

24.9.0

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[balaG4046]

hi team, any update here

[aldy505]

Did you modify your Docker network IP range to something that's not in the private IP ranges? See here: https://develop.sentry.dev/self-hosted/troubleshooting/#docker-network-conflicting-ip-address

The easiest solution is just restart your Docker engine. sudo systemctl restart docker

[balaG4046]

Hi @aldy505 Thanks for your update, we have not modified any ip range, but this issue is coming after we moved to 24.9.0 version, however we are restarting the container to resolve the issue, it would be better if we can permanently fix the issue rather than restarting the container.

Thanks!

[aldy505]

Okay, another guess, do you happen to have these lines on your sentry/sentry.conf.py file?

https://github.com/getsentry/self-hosted/blob/bdf8d3ff918d92b04d10906f3869900ca41ec3bf/sentry/sentry.conf.example.py#L16-L42

[balaG4046]

yes we do have this line in our conf file @aldy505 today also we faced the same issue again, please help us with the solution. is this issue fixed in 24.10.0 ? ref(feedback): 401 for unauth'd POSTs to projectUserReports (https://github.com/getsentry/sentry/pull/79069) ?

[arussell]

I am also seeing this issue under 24.10.0, get_internal_network() returns WAN IP. Hard coding internal Docker range with INTERNAL_SYSTEM_IPS = ("172.16.0.0/12") and rerunning ./install.sh does not fix the issue, still see the error:

[ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP

[balaG4046]

Hi @aldy505 Kindly help me with an update.

Thanks!

[aamarques]

The same is true here. I can't access account/notifications or account/security.

[balaG4046]

Hi @aldy505 Kindly help me with an update.

Thanks!

[aldy505]

Sorry, at this point I have no idea on how to fix this (and how to debug this).

[aamarques]

This has been working until 24.7.0, and suddenly, the problem is ours. Someone somewhere removed a piece that works, and we cannot change the notify and security page.

[balaG4046]

@aldy505 can you forward this issue to anyone from your team to look it,

[aamarques]

@hubertdeng123 Did you see this issue ? Can you help us ?

[aldy505]

@bc-sentry Hello, can you help by transferring this to getsentry/sentry for triage? Thank you!

[bc-sentry]

@aldy505 Sentry internal won't be able to help with this. We would have to know the IP address that is not being considered internal. Unfortunately, there is no easy way for the developer to add a print statement to the code to show that IP address. The IP addresses that are considered internal are listed in server.py. Perhaps they could re-check their container IP ranges and see if anything looks odd.

[balaG4046]

hi @bc-sentry we have checked and our Container IPs is falls under the server.py list of IPS which is mentioned in the file.

[aamarques]

Here too. This happened from scratch, so something has been changed somewhere.

[aamarques]

After researching the Internet, I found this issue: https://github.com/getsentry/self-hosted/issues/2632. My sentry.conf.py has a function to resolve the INTERNAL_SYSTEM_IPS = (get_internal_network(),), and this "solution" the guy seems to put the values by hand. (https://github.com/getsentry/self-hosted/issues/2632#issuecomment-1860400590).

1 - What is the correct behaviour: use function or add manually? 2 - How can I check if the sentry.conf.py is running ok ? Executing it in the container ?

@aldy505 @bc-sentry

[arussell]

@aamarques I tried this earlier in this thread, but it didn't seem to help:

I am also seeing this issue under 24.10.0, get_internal_network() returns WAN IP. Hard coding internal Docker range with INTERNAL_SYSTEM_IPS = ("172.16.0.0/12") and rerunning ./install.sh does not fix the issue, still see the error:

[ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP

[aamarques]

@aamarques I tried this earlier in this thread, but it didn't seem to help:

I am also seeing this issue under 24.10.0, get_internal_network() returns WAN IP. Hard coding internal Docker range with INTERNAL_SYSTEM_IPS = ("172.16.0.0/12") and rerunning ./install.sh does not fix the issue, still see the error:

[ERROR] sentry.auth.system: Trying to use SystemToken from non-internal IP

Sorry man. I can't understand how a thing that works, stop to work and nobody has an answer.

[bc-sentry]

@balaG4046 Sorry that we haven't been able to help you fix this. As I was looking again at your initial post, it doesn't say exactly what the failure does or blocks you from doing. You showed two HTTP 401's trying to access project dsym files and then the 4 errors Trying to use SystemToken from non-internal IP. As a quick test, I just uploaded a dsym to a self-hosted project using the sentry-cli and it worked. To help us diagnoses further, what is the actual problem caused by these errors in the log? Thanks.

[aamarques]

Hey @bc-sentry this block these features: https://github.com/getsentry/self-hosted/issues/3420

[hubertdeng123]

Just to confirm, are all you folks using symbolicator?

The INTERNAL_SYSTEM_IPS = (get_internal_network(),) is a function we use as an attempt to automatically add a list of internal IPs for you. If that fails, you should manually add them.

Here is the line of code that is throwing the error: https://github.com/getsentry/sentry/blob/master/src/sentry/auth/system.py#L51

[hubertdeng123]

Wondering if symbolicator for some reason is being treated as a non-internal IP

[balaG4046]

Hi @hubertdeng123 Yes we are using symbolicator, i guess as you said symbolicator is being treated as non internal Ip, when we checked the symbolicator logs we found 401 unauthorized error - please refer the screenshot

Kindly let us know how to fix this issue - need your help and suggestion

[aamarques]

Hi @hubertdeng123 and @balaG4046 that the same here.

sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:47.637676325Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:47.743927042Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:52.223670594Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:52.341258233Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:52.445518454Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:52.528391598Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:52.636199742Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
sentry-self-hosted-symbolicator-1               | 2024-11-25T10:09:52.741299372Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized

[aamarques]

Hi @hubertdeng123 I change the sentry.conf.py and put manually : INTERNAL_SYSTEM_IPS = ("172.0.0.0/8",) and now I have no more the Trying to use SystemToken from non-internal IP error.

But symbolicator still giving symbolicator-1 | 2024-11-25T13:56:56.897232926Z WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized

And I still have no access to these pages: https://github.com/getsentry/self-hosted/issues/3420

[balaG4046]

Hi @hubertdeng123 Any update please.

Thanks!

[aamarques]

Morning, The workaround I did : https://forum.sentry.io/t/symbolicator-401-unauthorized-error/13878/5

I added the following and the external IP (XX.XX.XX.XX) that reaches the web. In my case, Sentry is internal, and all traffic comes from one IP.

Now I don't have that error. I'll try other ways also.

INTERNAL_SYSTEM_IPS = (
    "0.0.0.0/8",
    "10.0.0.0/8",
    "100.64.0.0/10",
    "127.0.0.0/8",
    "169.254.0.0/16",
    "172.16.0.0/12",
    "192.0.0.0/29",
    "192.0.2.0/24",
    "192.88.99.0/24",
    "192.168.0.0/16",
    "198.18.0.0/15",
    "198.51.100.0/24",
    "224.0.0.0/4",
    "240.0.0.0/4",
    "255.255.255.255/32",
   "XX.XX.XX.XX", 
)

BUT THE PAGES NOTIFICATION AND SECURITY STILL DON'T WORK.