self-hosted Backup import failure

Self-Hosted Version

23.12.1

CPU Architecture

x86_64

Docker Version

26.1.3

Docker Compose Version

2.27.0

Steps to Reproduce

cd /path/to/sentry-dist/sentry
Create a global backup: docker compose run --rm -T web export global > backup.json
Import backup: docker compose run --rm -T web import global --no-prompt /etc/sentry/backup.json

Expected Result

Backup imported without issues

Actual Result

Restore fails with the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: 'snuba_query'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/sentry/runner/__init__.py", line 195, in main
    func(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/sentry/runner/commands/backup.py", line 722, in import_global
    import_in_global_scope(
  File "/usr/local/lib/python3.10/site-packages/sentry/backup/imports.py", line 479, in import_in_global_scope
    return _import(
  File "/usr/local/lib/python3.10/site-packages/sentry/backup/imports.py", line 358, in _import
    _clear_model_tables_before_import()
  File "/usr/local/lib/python3.10/site-packages/sentry/backup/imports.py", line 60, in _clear_model_tables_before_import
    manager.all().delete()  # type: ignore
  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 156, in override
    return original_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 746, in delete
    deleted, _rows_count = collector.delete()
  File "/usr/local/lib/python3.10/site-packages/django/db/models/deletion.py", line 435, in delete
    signals.post_delete.send(
  File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/usr/local/lib/python3.10/site-packages/sentry/incidents/models.py", line 394, in clear_alert_rule_subscription_caches
    snuba_query=instance.snuba_query
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
sentry.snuba.models.SnubaQuery.DoesNotExist: SnubaQuery matching query does not exist.

Event ID

No response

May 28 '24 13:05 afaianswq

This seems like some code that was written a while ago that does not directly influence the import process. Are you importing into a clean install of sentry? Is there information in your db?

cc @azaslavsky

May 29 '24 21:05 hubertdeng123

Is this error repeatable? I think it's a racy interaction of post-delete signals, and would work on the following try.

May 29 '24 22:05 azaslavsky

Yes, it's 100% reproducible.

May 30 '24 06:05 afaianswq

This seems like some code that was written a while ago that does not directly influence the import process. Are you importing into a clean install of sentry? Is there information in your db?

cc @azaslavsky

I am importing into the same instance, so yes, there is information in the db.

May 30 '24 06:05 afaianswq

I have just tested it with version 24.5.0 and the issue reproduced.

May 30 '24 09:05 afaianswq

Are you also exporting from version 24.5.0? It's important to note that we only support imports from versions that are 3 versions from the version you're importing into. So, in your case, it would be 24.3.0

May 31 '24 23:05 hubertdeng123

Yes, export/import is attempted on the same version always.

Jun 04 '24 03:06 afaianswq

You may need to drop this table manually. Is your instance currently working (ie, you failed to load the backup, but the instance still works as before)? Or is it completely bricked (which is what I would expect from this stack trace).

In any case, you'll need to use psql to drop the offending table manually - but please do a full backup first!

Jun 05 '24 23:06 azaslavsky

Which table do I need to drop? And why? I am not looking for a quick workaround. I want backup/restore feature to work reliably.

Jun 06 '24 03:06 afaianswq

I think I have a solution for this, will try to get it up by Monday.

Jun 07 '24 23:06 azaslavsky

Update: it's a bit more involved than I had anticipated, so it may take a few days to get to the bottom of this and resolve. But I do believe this is fundamentally a bug on our end.

Jun 12 '24 20:06 azaslavsky

Hello. Any news on this?

Jun 27 '24 14:06 shadowlmd

Got a bit derailed by some other work, hoping to pick this up next week.

Jun 28 '24 22:06 azaslavsky

Bump...

Jul 24 '24 08:07 shadowlmd

Another bump. Can someone please address this quite critical issue?

Aug 12 '24 12:08 shadowlmd

Bump

Aug 22 '24 11:08 shadowlmd

Bump

Aug 27 '24 02:08 shadowlmd

Please keep in mind that this issue is on our radar, we just haven't gotten to it yet. Please avoid bumping this as it is not helpful to us for prioritization.

Aug 28 '24 21:08 hubertdeng123

Why is it still on your radar after 3 months? Restoring from a backup is a essential and critical part of the product, this bug is supposed to have critical priority.

Aug 28 '24 21:08 shadowlmd

Any progress? This bug is holding us from upgrading from a more than one year old version.

Oct 23 '24 15:10 shadowlmd

No progress at the moment.

cc @azaslavsky

Nov 12 '24 01:11 hubertdeng123

Can you please raise the priority of this bug and assign someone to work on it? This is a critical feature and it's completely broken for more than a year already.

Dec 17 '24 10:12 shadowlmd

A bit of a sad news. This will be backlogged for a while, Alex has been moved to another team. He's not on the OSS/devinfra team anymore.

Dec 30 '24 11:12 aldy505