synapse icon indicating copy to clipboard operation
synapse copied to clipboard

`compute_state_delta` pulls out the full state when doing gappy syncs

Open erikjohnston opened this issue 3 years ago • 1 comments

This causes spikes in memory usage when handling /sync requests.

The callstack:

  File "/home/erikj/synapse/synapse/handlers/sync.py", line 1562, in handle_room_entries
    await self._generate_room_entry(
  File "/home/erikj/synapse/synapse/handlers/sync.py", line 2072, in _generate_room_entry
    state = await self.compute_state_delta(
  File "/home/erikj/synapse/synapse/handlers/sync.py", line 966, in compute_state_delta
    state_at_previous_sync = await self.get_state_at(
  File "/home/erikj/synapse/synapse/handlers/sync.py", line 680, in get_state_at
    state = await self.get_state_after_event(
  File "/home/erikj/synapse/synapse/handlers/sync.py", line 639, in get_state_after_event
    state_ids = await self._state_storage_controller.get_state_ids_for_event(
  File "/home/erikj/synapse/synapse/storage/controllers/state.py", line 303, in get_state_ids_for_event
    state_map = await self.get_state_ids_for_events(
  File "/home/erikj/synapse/synapse/storage/controllers/state.py", line 253, in get_state_ids_for_events
    group_to_state = await self.stores.state._get_state_for_groups(
  File "/home/erikj/synapse/synapse/storage/databases/state/store.py", line 288, in _get_state_for_groups
    group_to_state_dict = await self._get_state_groups_from_groups(
  File "/home/erikj/synapse/synapse/storage/databases/state/store.py", line 173, in _get_state_groups_from_groups

erikjohnston avatar Jul 25 '22 14:07 erikjohnston

It looks like we did this to handle a bug where membership changes in the gap get missed: https://github.com/matrix-org/synapse/blob/845732be450b3f9c991df35b2f07d600a0eca6dd/synapse/handlers/sync.py#L948-L951

See the comment here: https://github.com/vector-im/element-web/issues/7211#issuecomment-419668549

I don't think there's an easy fix for this one.

squahtx avatar Aug 09 '22 11:08 squahtx