self-hosted Upgrade to 24.4.2 failed

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

git fetch git checkout 24.4.2 sudo ./install.sh

Expected Result

The upgrade task was executed successfully

Actual Result

sudo ./install.sh

...

Running migrations for default
Operations to perform:
  Apply all migrations: auth, contenttypes, feedback, hybridcloud, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
Alert Rules: 100% |#                                           | ETA:  --:--:--
 OK
  Applying sentry.0702_alert_rule_project_backfill_migration_2... OK
  Applying sentry.0703_add_team_user_to_rule... OK
Rules: 100% |##################################################| Time:  0:00:00
 OK
  Applying sentry.0705_grouphistory_add_userteam... OK
Group Historys: 100% |#########################################| Time:  0:00:00
 OK
  Applying sentry.0707_alert_rule_activations_incidents_fk... OK
  Applying sentry.0708_rule_remove_owner_state... OK
Creating missing DSNs
Correcting Group.num_comments counter
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/base.py", line 84, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.IndexCorrupted: table tid from new index tuple (107,8) overlaps with invalid duplicate tuple at offset 27 of block 5 in index "sentry_groupedmessage_logger_993cb6d5"


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 93, in inner
    raise type(e)(f"{e!r}\nSQL: {sql}").with_traceback(e.__traceback__)
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/base.py", line 84, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.IndexCorrupted: IndexCorrupted('table tid from new index tuple (107,8) overlaps with invalid duplicate tuple at offset 27 of block 5 in index "sentry_groupedmessage_logger_993cb6d5"\n')
SQL:
        UPDATE sentry_groupedmessage SET num_comments = (
            SELECT COUNT(*) from sentry_activity
            WHERE type = %s and group_id = sentry_groupedmessage.id
        )


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/__init__.py", line 195, in main
    func(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 35, in inner
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/upgrade.py", line 135, in upgrade
    _upgrade(
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/upgrade.py", line 79, in _upgrade
    call_command("sentry.runner.commands.repair.repair")
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/__init__.py", line 162, in call_command
    command.invoke(ctx)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 35, in inner
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/repair.py", line 94, in repair
    fix_group_counters()
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/repair.py", line 62, in fix_group_counters
    cursor.execute(
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1711, in runner
    return sentry_patched_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/__init__.py", line 629, in execute
    result = real_execute(self, sql, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 77, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 75, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 18, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 93, in inner
    raise type(e)(f"{e!r}\nSQL: {sql}").with_traceback(e.__traceback__)
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/decorators.py", line 91, in inner
    return func(self, sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/postgres/base.py", line 84, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.InternalError: IndexCorrupted('table tid from new index tuple (107,8) overlaps with invalid duplicate tuple at offset 27 of block 5 in index "sentry_groupedmessage_logger_993cb6d5"\n')
SQL:
        UPDATE sentry_groupedmessage SET num_comments = (
            SELECT COUNT(*) from sentry_activity
            WHERE type = %s and group_id = sentry_groupedmessage.id
        )

Error in install/set-up-and-migrate-database.sh:30.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:35
--> install/set-up-and-migrate-database.sh:source:30

Cleaning up...

Product Area

Other

Link

No response

DSN

No response

Version

24.4.1

May 07 '24 08:05 MingNiu

Assigning to @getsentry/support for routing ⏲️

May 07 '24 08:05 getsantry[bot]

REINDEX index sentry_groupedmessage_logger_993cb6d5;

May 07 '24 09:05 MingNiu

Just to clarify - did that reindexing operation work?

May 09 '24 18:05 azaslavsky

I just run into the same problem and I can confirm it works

docker exec -it sentry_onpremise-postgres-1 bash psql -u postgres REINDEX index sentry_groupedmessage_logger_993cb6d5;

May 11 '24 13:05 cocorossello

I changed image: "postgres:14.11-alpine" with image: "postgres:14.11" in the docker-compose.yml file and it fixed the problem

May 11 '24 19:05 sydro

Yes, reindex works.

May 13 '24 01:05 MingNiu

What version of self-hosted are you coming from? We did not face this issue ourselves, so am wondering.

May 14 '24 20:05 hubertdeng123

upgrade from 24.4.1 to 24.4.2

May 27 '24 05:05 MingNiu

Given that you were able to find a fix for this, I'm going to close this issue. Please let me know if there is any other way we can help.

May 29 '24 21:05 hubertdeng123

I've just got the same issue while upgrading from 23.12.1 to 24.5.0. Can you please reopen this issue and find a proper fix for it?

May 30 '24 05:05 afaianswq

The root cause of the issue is most probably the change from image: "postgres:14.5" to image: "postgres:14.11-alpine". PostgreSQL have issues with indexes when libc version changes which is most probably what happened when debian changed to alpine.

May 30 '24 06:05 afaianswq