synapse
synapse copied to clipboard
element-hq
Fails to start with `IncorrectDatabaseSetup` error (inconsistent stream positions with sequences)
Description
Synapse fails to start with an IncorrectDatabaseSetup exception, logging the following:
Postgres sequence 'account_data_sequence' is inconsistent with associated stream position of 'account_data' in the 'stream_positions' table.
This is likely a programming error and should be reported at https://github.com/matrix-org/synapse.
Steps to reproduce
Try to start synapse.
Homeserver
fsmpi.rwth-aachen.de
Synapse Version
v1.128.0
Installation Method
Docker (matrixdotorg/synapse)
Database
Postgres 14
Workers
Single process
Platform
Running in a Kubernetes cluster using the ghcr.io/element-hq/synapse image
Configuration
No response
Relevant log output
{"log":"Exception during startup","namespace":"synapse.app._base","level":"ERROR","time":1749741788.04,"request":"main","server_name":"fsmpi.rwth-aachen.de","exc_type":"IncorrectDatabaseSetup","exc_value":"\nPostgres sequence 'account_data_sequence' is inconsistent with associated stream position\nof 'account_data' in the 'stream_positions' table.\n\nThis is likely a programming error and should be reported at\nhttps://github.com/matrix-org/synapse.\n\nA temporary workaround to fix this error is to shut down Synapse (including\nany and all workers) and run the following SQL:\n\n DELETE FROM stream_positions WHERE stream_name = 'account_data';\n\nThis will need to be done every time the server is restarted.\n"}
**********************************************************************************
Error during initialisation:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/synapse/app/homeserver.py", line 373, in setup
hs.setup()
File "/usr/local/lib/python3.12/site-packages/synapse/server.py", line 359, in setup
self.datastores = Databases(self.DATASTORE_CLASS, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/__init__.py", line 104, in __init__
main = main_store_class(database, db_conn, hs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/__init__.py", line 174, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 117, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/experimental_features.py", line 44, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/devices.py", line 1770, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/devices.py", line 95, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/roommember.py", line 1935, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/room.py", line 2418, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/room.py", line 1961, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/room.py", line 150, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/registration.py", line 2324, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/profile.py", line 53, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/presence.py", line 80, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/presence.py", line 61, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/transactions.py", line 81, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/state.py", line 974, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/state.py", line 118, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/state.py", line 765, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/event_federation.py", line 2075, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/media_repository.py", line 226, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/media_repository.py", line 119, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/filtering.py", line 49, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1894, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/metrics.py", line 75, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/end_to_end_keys.py", line 1512, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/e2e_room_keys.py", line 69, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/search.py", line 433, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/search.py", line 144, in __init__
super().__init__(database, db_conn, hs)
File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/account_data.py", line 73, in __init__
self._account_data_id_gen = MultiWriterIdGenerator(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
self._sequence_gen.check_consistency(
File "/usr/local/lib/python3.12/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'account_data_sequence' is inconsistent with associated stream position
of 'account_data' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'account_data';
This will need to be done every time the server is restarted.
There may be more information in the logs.
**********************************************************************************
Found file in config directory that does not end in '.yaml': '/config-ro/fsmpi.rwth-aachen.de.log.config'. IGNORING.
Found file in config directory that does not end in '.yaml': '/config-ro/database.yaml.tmpl'. IGNORING.
Found file in config directory that does not end in '.yaml': '/config-ro/oidc.yaml.tmpl'. IGNORING.
Found file in config directory that does not end in '.yaml': '/config-ro/fsmpi.rwth-aachen.de.signing.key'. IGNORING.
Found file in config directory that does not end in '.yaml': '/config-ro/oidc_client_secret'. IGNORING.
Found subdirectory in config directory: '/config-ro/..data'. IGNORING.
Found subdirectory in config directory: '/config-ro/..2025_06_12_15_06_27.209063034'. IGNORING.
Stream closed EOF for matrix-synapse/synapse-7946c4bb9b-6x9qf (matrix-synapse)
Anything else that would be useful to know?
The URL it says to report the bug at is wrong.
I poked around in the database a bit and found the following:
synapse=> select stream_id from account_data order by stream_id desc limit 1;
stream_id
-----------
81210
(1 row)
synapse=> select * from stream_positions where stream_name = 'account_data';
stream_name | instance_name | stream_id
--------------+---------------+-----------
account_data | master | 81239
(1 row)
# grep account_data_sequence dump.sql
[…]
-- Name: account_data_sequence; Type: SEQUENCE SET; Schema: data; Owner: synapse_owner_user
SELECT pg_catalog.setval('data.account_data_sequence', 81210, true);
[…]
Hi !
We're hitting this issue on regular basis during upgrades... The last one was today while upgrading from 1.127 to 1.130 on K8s with Postgresql 15 & Redis 6.
After executing the recommended DELETE it starts again. But the problem will be back next upgrade.
I guess the database is borked in a way, but I'm having a hard time finding how. This script don't detect issues on those sequences (after the DELETE + restart).
Hints from a database expert would be most welcome !
How are you doing the upgrades? Is it possible you are not stopping Synapse before starting the second copy? (Just thinking since both of you are using Kubernetes)
We have used the rolling strategy, so yes we might have had 2 versions running in parallel at some point. But we are now using the recreate strategy that is supposed to terminate the pod with the old version before starting the new one.
Is it possible the database has been borked for a long time, but synapse is still running properly except for upgrades ?
Does this look like a proper shutdown ? The redis connection is suspect.
2025-07-23 09:23:07,225 - twisted - 279 - INFO - sentinel - Received SIGTERM, shutting down.
2025-07-23 09:23:07,226 - synapse.storage.databases.main.lock - 113 - INFO - LockStore._on_shutdown-0 - Dropping held locks due to shutdown
2025-07-23 09:23:07,226 - synapse.storage.databases.main.lock - 124 - INFO - LockStore._on_shutdown-0 - Dropped locks due to shutdown
2025-07-23 09:23:07,226 - synapse.handlers.presence - 891 - INFO - presence.on_shutdown-0 - Performing _on_shutdown. Persisting 3 unpersisted changes
2025-07-23 09:23:07,226 - synapse.handlers.presence - 903 - INFO - presence.on_shutdown-0 - Finished _on_shutdown
2025-07-23 09:23:07,227 - synapse.app._base - 509 - INFO - sentinel - Shutting down...
2025-07-23 09:23:07,227 - twisted - 279 - INFO - sentinel - (TCP Port 9090 Closed)
2025-07-23 09:23:07,227 - synapse.replication.tcp.redis - 306 - INFO - sentinel - Connection to redis server chat--sy-re.beta-liiibre:6379 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
2025-07-23 09:23:07,228 - twisted - 279 - INFO - sentinel - (TCP Port 8008 Closed)
2025-07-23 09:23:07,228 - synapse.replication.tcp.redis - 204 - INFO - sentinel - Lost connection to redis
2025-07-23 09:23:07,228 - synapse.replication.tcp.redis - 306 - INFO - sentinel - Connection to redis server chat--sy-re.beta-liiibre:6379 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
2025-07-23 09:23:07,229 - twisted - 279 - INFO - sentinel - (TCP Port 9093 Closed)
2025-07-23 09:23:07,233 - twisted - 279 - INFO - sentinel - Main loop terminated.
I am encountering this issue having migrated from sqlite to postgres just now. I'm not running containerized or anything else and the server was definitely shutdown during the synapse_port_db runs. First run ran successfully, but subsequent runs of the tool popped an error like this, as did restarting with the postgres db backend after migration completed.
Running the recommended psql commands to delete rows from that table did fix the second run of synapse_port_db, but then a new sequence popped up on first start.
Running synapse_port_db:
root@elementsynapse:/etc/matrix-synapse# synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db --postgres-config /etc/matrix-synapse/homeserver-pg.yaml --curses
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/_scripts/synapse_port_db.py", line 784, in run
self.postgres_store = self.build_db_store(
~~~~~~~~~~~~~~~~~~~^
self.hs_config.database.get_single_database()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/_scripts/synapse_port_db.py", line 688, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs) # type: ignore[arg-type]
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1894, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/client_ips.py", line 93, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/deviceinbox.py", line 1082, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/devices.py", line 2424, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 122, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/media_repository.py", line 119, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/registration.py", line 2641, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/registration.py", line 185, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/room.py", line 2028, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/roommember.py", line 1886, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/search.py", line 145, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/state/bg_updates.py", line 299, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/state.py", line 765, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/user_directory.py", line 93, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/end_to_end_keys.py", line 92, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/e2e_room_keys.py", line 69, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/stats.py", line 128, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/account_data.py", line 68, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/push_rule.py", line 139, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/appservice.py", line 103, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/filtering.py", line 49, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/profile.py", line 53, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/pusher.py", line 82, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/roommember.py", line 102, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/receipts.py", line 138, in __init__
self._receipts_id_gen = MultiWriterIdGenerator(
~~~~~~~~~~~~~~~~~~~~~~^
db_conn=db_conn,
^^^^^^^^^^^^^^^^
...<7 lines>...
writers=hs.config.worker.writers.receipts,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/id_generators.py", line 309, in __init__
self._sequence_gen.check_consistency(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
db_conn,
^^^^^^^^
...<3 lines>...
positive=positive,
^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
...<2 lines>...
)
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'receipts_sequence' is inconsistent with associated stream position
of 'receipts' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'receipts';
This will need to be done every time the server is restarted.
Postgres sequence 'receipts_sequence' is inconsistent with associated stream position
of 'receipts' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'receipts';
This will need to be done every time the server is restarted.
root@elementsynapse:/etc/matrix-synapse# synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db --postgres-config /etc/matrix-synapse/homeserver-pg.yaml --curses
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/_scripts/synapse_port_db.py", line 784, in run
self.postgres_store = self.build_db_store(
~~~~~~~~~~~~~~~~~~~^
self.hs_config.database.get_single_database()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/_scripts/synapse_port_db.py", line 688, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs) # type: ignore[arg-type]
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1894, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/client_ips.py", line 93, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/deviceinbox.py", line 1082, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/devices.py", line 2424, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 122, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/media_repository.py", line 119, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/registration.py", line 2641, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/registration.py", line 185, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/room.py", line 2028, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/roommember.py", line 1886, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/search.py", line 145, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/state/bg_updates.py", line 299, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/state.py", line 765, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/user_directory.py", line 93, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/end_to_end_keys.py", line 92, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/e2e_room_keys.py", line 69, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/stats.py", line 128, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/account_data.py", line 68, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/push_rule.py", line 139, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/appservice.py", line 103, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/filtering.py", line 49, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/profile.py", line 53, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/pusher.py", line 82, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/roommember.py", line 102, in __init__
super().__init__(database, db_conn, hs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/receipts.py", line 138, in __init__
self._receipts_id_gen = MultiWriterIdGenerator(
~~~~~~~~~~~~~~~~~~~~~~^
db_conn=db_conn,
^^^^^^^^^^^^^^^^
...<7 lines>...
writers=hs.config.worker.writers.receipts,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/id_generators.py", line 309, in __init__
self._sequence_gen.check_consistency(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
db_conn,
^^^^^^^^
...<3 lines>...
positive=positive,
^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
...<2 lines>...
)
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'receipts_sequence' is inconsistent with associated stream position
of 'receipts' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'receipts';
This will need to be done every time the server is restarted.
Postgres sequence 'receipts_sequence' is inconsistent with associated stream position
of 'receipts' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'receipts';
This will need to be done every time the server is restarted.
Journalctl entry at first start:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/deviceinbox.py", line 107, in __init__
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: self._to_device_msg_id_gen: MultiWriterIdGenerator = MultiWriterIdGenerator(
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ~~~~~~~~~~~~~~~~~~~~~~^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: db_conn=db_conn,
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ^^^^^^^^^^^^^^^^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ...<10 lines>...
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: writers=hs.config.worker.writers.to_device,
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: )
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/id_generators.py", line 309, in __init__
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: self._sequence_gen.check_consistency(
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: db_conn,
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ^^^^^^^^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ...<3 lines>...
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: positive=positive,
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ^^^^^^^^^^^^^^^^^^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: )
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ^
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: raise IncorrectDatabaseSetup(
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: ...<2 lines>...
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: )
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: synapse.storage.engines._base.IncorrectDatabaseSetup:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: Postgres sequence 'device_inbox_sequence' is inconsistent with associated stream position
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: of 'to_device' in the 'stream_positions' table.
Sep 05 13:54:50 elementsynapse matrix-synapse[389]:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: This is likely a programming error and should be reported at
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: https://github.com/matrix-org/synapse.
Sep 05 13:54:50 elementsynapse matrix-synapse[389]:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: A temporary workaround to fix this error is to shut down Synapse (including
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: any and all workers) and run the following SQL:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: DELETE FROM stream_positions WHERE stream_name = 'to_device';
Sep 05 13:54:50 elementsynapse matrix-synapse[389]:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: This will need to be done every time the server is restarted.
Sep 05 13:54:50 elementsynapse matrix-synapse[389]:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]:
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: There may be more information in the logs.
Sep 05 13:54:50 elementsynapse matrix-synapse[389]: **********************************************************************************
Journalctl entry at second attempt:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/pusher.py", line 579, in __init__
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: super().__init__(database, db_conn, hs)
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/databases/main/receipts.py", line 138, in __init__
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: self._receipts_id_gen = MultiWriterIdGenerator(
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ~~~~~~~~~~~~~~~~~~~~~~^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: db_conn=db_conn,
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ^^^^^^^^^^^^^^^^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ...<7 lines>...
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: writers=hs.config.worker.writers.receipts,
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: )
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/id_generators.py", line 309, in __init__
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: self._sequence_gen.check_consistency(
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: db_conn,
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ^^^^^^^^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ...<3 lines>...
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: positive=positive,
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ^^^^^^^^^^^^^^^^^^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: )
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ^
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: File "/opt/venvs/matrix-synapse/lib/python3.13/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: raise IncorrectDatabaseSetup(
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: ...<2 lines>...
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: )
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: synapse.storage.engines._base.IncorrectDatabaseSetup:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: Postgres sequence 'receipts_sequence' is inconsistent with associated stream position
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: of 'receipts' in the 'stream_positions' table.
Sep 05 13:55:21 elementsynapse matrix-synapse[474]:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: This is likely a programming error and should be reported at
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: https://github.com/matrix-org/synapse.
Sep 05 13:55:21 elementsynapse matrix-synapse[474]:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: A temporary workaround to fix this error is to shut down Synapse (including
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: any and all workers) and run the following SQL:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: DELETE FROM stream_positions WHERE stream_name = 'receipts';
Sep 05 13:55:21 elementsynapse matrix-synapse[474]:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: This will need to be done every time the server is restarted.
Sep 05 13:55:21 elementsynapse matrix-synapse[474]:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]:
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: There may be more information in the logs.
Sep 05 13:55:21 elementsynapse matrix-synapse[474]: **********************************************************************************
It looks like it's pretty unhappy with that table...
I am encountering this issue having migrated from sqlite to postgres just now. I'm not running containerized or anything else and the server was definitely shutdown during the synapse_port_db runs. First run ran successfully, but subsequent runs of the tool popped an error like this, as did restarting with the postgres db backend after migration completed.
Running the recommended psql commands to delete rows from that table did fix the second run of synapse_port_db, but then a new sequence popped up on first start.
+1
