diff options
45 files changed, 136 insertions, 57 deletions
diff --git a/CHANGES.md b/CHANGES.md index 4c413b72ee..ab6fce3e7d 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,3 +1,50 @@ +Synapse 1.10.0rc1 (2020-01-31) +============================== + +**WARNING**: As of this release Synapse validates `client_secret` parameters in the Client-Server API as per the spec. See [\#6766](https://github.com/matrix-org/synapse/issues/6766) for details. + + +Features +-------- + +- Add experimental support for updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260). ([\#6787](https://github.com/matrix-org/synapse/issues/6787), [\#6790](https://github.com/matrix-org/synapse/issues/6790), [\#6794](https://github.com/matrix-org/synapse/issues/6794)) + + +Bugfixes +-------- + +- Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS). ([\#6734](https://github.com/matrix-org/synapse/issues/6734)) +- Minor fixes to `PUT /_synapse/admin/v2/users` admin api. ([\#6761](https://github.com/matrix-org/synapse/issues/6761)) +- Validate `client_secret` parameter using the regex provided by the Client-Server API, temporarily allowing `:` characters for older clients. The `:` character will be removed in a future release. ([\#6767](https://github.com/matrix-org/synapse/issues/6767)) +- Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key). ([\#6771](https://github.com/matrix-org/synapse/issues/6771)) +- Fix outbound federation request metrics. ([\#6795](https://github.com/matrix-org/synapse/issues/6795)) +- Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device. ([\#6796](https://github.com/matrix-org/synapse/issues/6796)) +- Fix race in federation sender worker that delayed sending of device updates. ([\#6799](https://github.com/matrix-org/synapse/issues/6799), [\#6800](https://github.com/matrix-org/synapse/issues/6800)) +- Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room. ([\#6801](https://github.com/matrix-org/synapse/issues/6801)) +- Fix waking up other workers when remote server is detected to have come back online. ([\#6811](https://github.com/matrix-org/synapse/issues/6811)) + + +Improved Documentation +---------------------- + +- Clarify documentation related to `user_dir` and `federation_reader` workers. ([\#6775](https://github.com/matrix-org/synapse/issues/6775)) + + +Internal Changes +---------------- + +- Record room versions in the `rooms` table. ([\#6729](https://github.com/matrix-org/synapse/issues/6729), [\#6788](https://github.com/matrix-org/synapse/issues/6788), [\#6810](https://github.com/matrix-org/synapse/issues/6810)) +- Propagate cache invalidates from workers to other workers. ([\#6748](https://github.com/matrix-org/synapse/issues/6748)) +- Remove some unnecessary admin handler abstraction methods. ([\#6751](https://github.com/matrix-org/synapse/issues/6751)) +- Add some debugging for media storage providers. ([\#6757](https://github.com/matrix-org/synapse/issues/6757)) +- Detect unknown remote devices and mark cache as stale. ([\#6776](https://github.com/matrix-org/synapse/issues/6776), [\#6819](https://github.com/matrix-org/synapse/issues/6819)) +- Attempt to resync remote users' devices when detected as stale. ([\#6786](https://github.com/matrix-org/synapse/issues/6786)) +- Delete current state from the database when server leaves a room. ([\#6792](https://github.com/matrix-org/synapse/issues/6792)) +- When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale. ([\#6797](https://github.com/matrix-org/synapse/issues/6797)) +- Add background update to clean out left rooms from current state. ([\#6802](https://github.com/matrix-org/synapse/issues/6802), [\#6816](https://github.com/matrix-org/synapse/issues/6816)) +- Refactoring work in preparation for changing the event redaction algorithm. ([\#6803](https://github.com/matrix-org/synapse/issues/6803), [\#6805](https://github.com/matrix-org/synapse/issues/6805), [\#6806](https://github.com/matrix-org/synapse/issues/6806), [\#6807](https://github.com/matrix-org/synapse/issues/6807), [\#6820](https://github.com/matrix-org/synapse/issues/6820)) + + Synapse 1.9.1 (2020-01-28) ========================== diff --git a/UPGRADE.rst b/UPGRADE.rst index 470246f128..3cad8c2837 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -76,13 +76,13 @@ for example: dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb -Upgrading to **<NEXT_VERSION>** -=============================== +Upgrading to v1.10.0 +==================== Synapse will now log a warning on start up if used with a PostgreSQL database that has a non-recommended locale set. -See [docs/postgres.md](docs/postgres.md) for details. +See `docs/postgres.md <docs/postgres.md>`_ for details. Upgrading to v1.8.0 diff --git a/changelog.d/6729.misc b/changelog.d/6729.misc deleted file mode 100644 index 5537355bea..0000000000 --- a/changelog.d/6729.misc +++ /dev/null @@ -1 +0,0 @@ -Record room versions in the `rooms` table. diff --git a/changelog.d/6734.bugfix b/changelog.d/6734.bugfix deleted file mode 100644 index 79c6bab4d1..0000000000 --- a/changelog.d/6734.bugfix +++ /dev/null @@ -1 +0,0 @@ -Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS). diff --git a/changelog.d/6748.misc b/changelog.d/6748.misc deleted file mode 100644 index de320d4cd9..0000000000 --- a/changelog.d/6748.misc +++ /dev/null @@ -1 +0,0 @@ -Propagate cache invalidates from workers to other workers. diff --git a/changelog.d/6751.misc b/changelog.d/6751.misc deleted file mode 100644 index 7222520528..0000000000 --- a/changelog.d/6751.misc +++ /dev/null @@ -1 +0,0 @@ -Remove some unnecessary admin handler abstraction methods. \ No newline at end of file diff --git a/changelog.d/6757.misc b/changelog.d/6757.misc deleted file mode 100644 index a50c5e974a..0000000000 --- a/changelog.d/6757.misc +++ /dev/null @@ -1 +0,0 @@ -Add some debugging for media storage providers. diff --git a/changelog.d/6761.bugfix b/changelog.d/6761.bugfix deleted file mode 100644 index 1c664c02df..0000000000 --- a/changelog.d/6761.bugfix +++ /dev/null @@ -1 +0,0 @@ -Minor fixes to `PUT /_synapse/admin/v2/users` admin api. diff --git a/changelog.d/6767.bugfix b/changelog.d/6767.bugfix deleted file mode 100644 index 63c7c63315..0000000000 --- a/changelog.d/6767.bugfix +++ /dev/null @@ -1 +0,0 @@ -Validate `client_secret` parameter using the regex provided by the Client-Server API, temporarily allowing `:` characters for older clients. The `:` character will be removed in a future release. diff --git a/changelog.d/6771.bugfix b/changelog.d/6771.bugfix deleted file mode 100644 index 623ba24acb..0000000000 --- a/changelog.d/6771.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key). diff --git a/changelog.d/6775.doc b/changelog.d/6775.doc deleted file mode 100644 index c6078ef82d..0000000000 --- a/changelog.d/6775.doc +++ /dev/null @@ -1 +0,0 @@ -Clarify documentation related to `user_dir` and `federation_reader` workers. diff --git a/changelog.d/6776.misc b/changelog.d/6776.misc deleted file mode 100644 index 4f9a4ac7a5..0000000000 --- a/changelog.d/6776.misc +++ /dev/null @@ -1 +0,0 @@ -Detect unknown remote devices and mark cache as stale. diff --git a/changelog.d/6786.misc b/changelog.d/6786.misc deleted file mode 100644 index 94c692e53a..0000000000 --- a/changelog.d/6786.misc +++ /dev/null @@ -1 +0,0 @@ -Attempt to resync remote users' devices when detected as stale. diff --git a/changelog.d/6787.feature b/changelog.d/6787.feature deleted file mode 100644 index df9e4b77ab..0000000000 --- a/changelog.d/6787.feature +++ /dev/null @@ -1 +0,0 @@ -Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260). diff --git a/changelog.d/6788.misc b/changelog.d/6788.misc deleted file mode 100644 index 5537355bea..0000000000 --- a/changelog.d/6788.misc +++ /dev/null @@ -1 +0,0 @@ -Record room versions in the `rooms` table. diff --git a/changelog.d/6790.feature b/changelog.d/6790.feature deleted file mode 100644 index df9e4b77ab..0000000000 --- a/changelog.d/6790.feature +++ /dev/null @@ -1 +0,0 @@ -Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260). diff --git a/changelog.d/6792.misc b/changelog.d/6792.misc deleted file mode 100644 index fa31d509b3..0000000000 --- a/changelog.d/6792.misc +++ /dev/null @@ -1 +0,0 @@ -Delete current state from the database when server leaves a room. diff --git a/changelog.d/6794.feature b/changelog.d/6794.feature deleted file mode 100644 index df9e4b77ab..0000000000 --- a/changelog.d/6794.feature +++ /dev/null @@ -1 +0,0 @@ -Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260). diff --git a/changelog.d/6795.bugfix b/changelog.d/6795.bugfix deleted file mode 100644 index d1585653b1..0000000000 --- a/changelog.d/6795.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix outbound federation request metrics. diff --git a/changelog.d/6796.bugfix b/changelog.d/6796.bugfix deleted file mode 100644 index 206a157311..0000000000 --- a/changelog.d/6796.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device. diff --git a/changelog.d/6797.misc b/changelog.d/6797.misc deleted file mode 100644 index e9127bac51..0000000000 --- a/changelog.d/6797.misc +++ /dev/null @@ -1 +0,0 @@ -When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale. diff --git a/changelog.d/6799.bugfix b/changelog.d/6799.bugfix deleted file mode 100644 index 322a2758af..0000000000 --- a/changelog.d/6799.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix race in federation sender worker that delayed sending of device updates. diff --git a/changelog.d/6800.bugfix b/changelog.d/6800.bugfix deleted file mode 100644 index 322a2758af..0000000000 --- a/changelog.d/6800.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix race in federation sender worker that delayed sending of device updates. diff --git a/changelog.d/6801.bugfix b/changelog.d/6801.bugfix deleted file mode 100644 index f401fa5d69..0000000000 --- a/changelog.d/6801.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room. diff --git a/changelog.d/6802.misc b/changelog.d/6802.misc deleted file mode 100644 index a77ba1d7a5..0000000000 --- a/changelog.d/6802.misc +++ /dev/null @@ -1 +0,0 @@ -Add background update to clean out left rooms from current state. diff --git a/changelog.d/6803.misc b/changelog.d/6803.misc deleted file mode 100644 index 08aa80bcd9..0000000000 --- a/changelog.d/6803.misc +++ /dev/null @@ -1 +0,0 @@ -Refactoring work in preparation for changing the event redaction algorithm. diff --git a/changelog.d/6805.misc b/changelog.d/6805.misc deleted file mode 100644 index 08aa80bcd9..0000000000 --- a/changelog.d/6805.misc +++ /dev/null @@ -1 +0,0 @@ -Refactoring work in preparation for changing the event redaction algorithm. diff --git a/changelog.d/6806.misc b/changelog.d/6806.misc deleted file mode 100644 index 08aa80bcd9..0000000000 --- a/changelog.d/6806.misc +++ /dev/null @@ -1 +0,0 @@ -Refactoring work in preparation for changing the event redaction algorithm. diff --git a/changelog.d/6807.misc b/changelog.d/6807.misc deleted file mode 100644 index 08aa80bcd9..0000000000 --- a/changelog.d/6807.misc +++ /dev/null @@ -1 +0,0 @@ -Refactoring work in preparation for changing the event redaction algorithm. diff --git a/changelog.d/6810.misc b/changelog.d/6810.misc deleted file mode 100644 index 5537355bea..0000000000 --- a/changelog.d/6810.misc +++ /dev/null @@ -1 +0,0 @@ -Record room versions in the `rooms` table. diff --git a/changelog.d/6811.bugfix b/changelog.d/6811.bugfix deleted file mode 100644 index 361f2fc2e8..0000000000 --- a/changelog.d/6811.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix waking up other workers when remote server is detected to have come back online. diff --git a/changelog.d/6816.misc b/changelog.d/6816.misc deleted file mode 100644 index a77ba1d7a5..0000000000 --- a/changelog.d/6816.misc +++ /dev/null @@ -1 +0,0 @@ -Add background update to clean out left rooms from current state. diff --git a/changelog.d/6819.misc b/changelog.d/6819.misc deleted file mode 100644 index 4f9a4ac7a5..0000000000 --- a/changelog.d/6819.misc +++ /dev/null @@ -1 +0,0 @@ -Detect unknown remote devices and mark cache as stale. diff --git a/changelog.d/6820.misc b/changelog.d/6820.misc deleted file mode 100644 index 08aa80bcd9..0000000000 --- a/changelog.d/6820.misc +++ /dev/null @@ -1 +0,0 @@ -Refactoring work in preparation for changing the event redaction algorithm. diff --git a/changelog.d/6844.bugfix b/changelog.d/6844.bugfix new file mode 100644 index 0000000000..e84aa1029f --- /dev/null +++ b/changelog.d/6844.bugfix @@ -0,0 +1 @@ +Fix an issue with cross-signing where device signatures were not sent to remote servers. diff --git a/changelog.d/6848.bugfix b/changelog.d/6848.bugfix new file mode 100644 index 0000000000..65688e5d57 --- /dev/null +++ b/changelog.d/6848.bugfix @@ -0,0 +1 @@ +Fix detecting unknown devices from remote encrypted events. diff --git a/changelog.d/6850.misc b/changelog.d/6850.misc new file mode 100644 index 0000000000..418569113f --- /dev/null +++ b/changelog.d/6850.misc @@ -0,0 +1 @@ +Detect unexpected sender keys on inbound encrypted events and resync device lists. diff --git a/synapse/__init__.py b/synapse/__init__.py index a236888d3c..bd942d3e1c 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -36,7 +36,7 @@ try: except ImportError: pass -__version__ = "1.9.1" +__version__ = "1.10.0rc1" if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): # We import here so that we don't have to install a bunch of deps when diff --git a/synapse/api/constants.py b/synapse/api/constants.py index 0ade47e624..cc8577552b 100644 --- a/synapse/api/constants.py +++ b/synapse/api/constants.py @@ -77,12 +77,11 @@ class EventTypes(object): Aliases = "m.room.aliases" Redaction = "m.room.redaction" ThirdPartyInvite = "m.room.third_party_invite" - Encryption = "m.room.encryption" RelatedGroups = "m.room.related_groups" RoomHistoryVisibility = "m.room.history_visibility" CanonicalAlias = "m.room.canonical_alias" - Encryption = "m.room.encryption" + Encrypted = "m.room.encrypted" RoomAvatar = "m.room.avatar" RoomEncryption = "m.room.encryption" GuestAccess = "m.room.guest_access" diff --git a/synapse/handlers/device.py b/synapse/handlers/device.py index 26ef5e150c..a9bd431486 100644 --- a/synapse/handlers/device.py +++ b/synapse/handlers/device.py @@ -598,7 +598,13 @@ class DeviceListUpdater(object): # happens if we've missed updates. resync = yield self._need_to_do_resync(user_id, pending_updates) - logger.debug("Need to re-sync devices for %r? %r", user_id, resync) + if logger.isEnabledFor(logging.INFO): + logger.info( + "Received device list update for %s, requiring resync: %s. Devices: %s", + user_id, + resync, + ", ".join(u[0] for u in pending_updates), + ) if resync: yield self.user_device_resync(user_id) diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py index c86d3177e9..e9441bbeff 100644 --- a/synapse/handlers/federation.py +++ b/synapse/handlers/federation.py @@ -752,29 +752,75 @@ class FederationHandler(BaseHandler): # For encrypted messages we check that we know about the sending device, # if we don't then we mark the device cache for that user as stale. - if event.type == EventTypes.Encryption: + if event.type == EventTypes.Encrypted: device_id = event.content.get("device_id") + sender_key = event.content.get("sender_key") + + cached_devices = await self.store.get_cached_devices_for_user(event.sender) + + resync = False # Whether we should resync device lists. + + device = None if device_id is not None: - cached_devices = await self.store.get_cached_devices_for_user( - event.sender - ) - if device_id not in cached_devices: + device = cached_devices.get(device_id) + if device is None: logger.info( "Received event from remote device not in our cache: %s %s", event.sender, device_id, ) - await self.store.mark_remote_user_device_cache_as_stale( - event.sender + resync = True + + # We also check if the `sender_key` matches what we expect. + if sender_key is not None: + # Figure out what sender key we're expecting. If we know the + # device and recognize the algorithm then we can work out the + # exact key to expect. Otherwise check it matches any key we + # have for that device. + if device: + keys = device.get("keys", {}).get("keys", {}) + + if event.content.get("algorithm") == "m.megolm.v1.aes-sha2": + # For this algorithm we expect a curve25519 key. + key_name = "curve25519:%s" % (device_id,) + current_keys = [keys.get(key_name)] + else: + # We don't know understand the algorithm, so we just + # check it matches a key for the device. + current_keys = keys.values() + elif device_id: + # We don't have any keys for the device ID. + current_keys = [] + else: + # The event didn't include a device ID, so we just look for + # keys across all devices. + current_keys = ( + key + for device in cached_devices + for key in device.get("keys", {}).get("keys", {}).values() ) - # Immediately attempt a resync in the background - if self.config.worker_app: - return run_in_background(self._user_device_resync, event.sender) - else: - return run_in_background( - self._device_list_updater.user_device_resync, event.sender - ) + # We now check that the sender key matches (one of) the expected + # keys. + if sender_key not in current_keys: + logger.info( + "Received event from remote device with unexpected sender key: %s %s: %s", + event.sender, + device_id or "<no device_id>", + sender_key, + ) + resync = True + + if resync: + await self.store.mark_remote_user_device_cache_as_stale(event.sender) + + # Immediately attempt a resync in the background + if self.config.worker_app: + return run_in_background(self._user_device_resync, event.sender) + else: + return run_in_background( + self._device_list_updater.user_device_resync, event.sender + ) @log_function async def backfill(self, dest, room_id, limit, extremities): diff --git a/synapse/handlers/room.py b/synapse/handlers/room.py index 1382399557..b609a65f47 100644 --- a/synapse/handlers/room.py +++ b/synapse/handlers/room.py @@ -360,7 +360,7 @@ class RoomCreationHandler(BaseHandler): (EventTypes.RoomHistoryVisibility, ""), (EventTypes.GuestAccess, ""), (EventTypes.RoomAvatar, ""), - (EventTypes.Encryption, ""), + (EventTypes.RoomEncryption, ""), (EventTypes.ServerACL, ""), (EventTypes.RelatedGroups, ""), (EventTypes.PowerLevels, ""), diff --git a/synapse/handlers/stats.py b/synapse/handlers/stats.py index 7f7d56390e..68e6edace5 100644 --- a/synapse/handlers/stats.py +++ b/synapse/handlers/stats.py @@ -286,7 +286,7 @@ class StatsHandler(StateDeltasHandler): room_state["history_visibility"] = event_content.get( "history_visibility" ) - elif typ == EventTypes.Encryption: + elif typ == EventTypes.RoomEncryption: room_state["encryption"] = event_content.get("algorithm") elif typ == EventTypes.Name: room_state["name"] = event_content.get("name") diff --git a/synapse/storage/data_stores/main/devices.py b/synapse/storage/data_stores/main/devices.py index ea0503476f..b7617efb80 100644 --- a/synapse/storage/data_stores/main/devices.py +++ b/synapse/storage/data_stores/main/devices.py @@ -320,6 +320,11 @@ class DeviceWorkerStore(SQLBaseStore): device_display_name = device.get("device_display_name", None) if device_display_name: result["device_display_name"] = device_display_name + if "signatures" in device: + for sig_user_id, sigs in device["signatures"].items(): + result["keys"].setdefault("signatures", {}).setdefault( + sig_user_id, {} + ).update(sigs) else: result["deleted"] = True @@ -524,6 +529,11 @@ class DeviceWorkerStore(SQLBaseStore): device_display_name = device.get("device_display_name", None) if device_display_name: result["device_display_name"] = device_display_name + if "signatures" in device: + for sig_user_id, sigs in device["signatures"].items(): + result["keys"].setdefault("signatures", {}).setdefault( + sig_user_id, {} + ).update(sigs) results.append(result) diff --git a/synapse/storage/data_stores/main/stats.py b/synapse/storage/data_stores/main/stats.py index 7bc186e9a1..7af1495e47 100644 --- a/synapse/storage/data_stores/main/stats.py +++ b/synapse/storage/data_stores/main/stats.py @@ -744,7 +744,7 @@ class StatsStore(StateDeltasStore): EventTypes.Create, EventTypes.JoinRules, EventTypes.RoomHistoryVisibility, - EventTypes.Encryption, + EventTypes.RoomEncryption, EventTypes.Name, EventTypes.Topic, EventTypes.RoomAvatar, @@ -816,7 +816,7 @@ class StatsStore(StateDeltasStore): room_state["history_visibility"] = event.content.get( "history_visibility" ) - elif event.type == EventTypes.Encryption: + elif event.type == EventTypes.RoomEncryption: room_state["encryption"] = event.content.get("algorithm") elif event.type == EventTypes.Name: room_state["name"] = event.content.get("name") |