summary refs log tree commit diff
Commit message (Collapse)AuthorAgeFilesLines
* support federation queries through http connect proxy (#10475)Dirk Klimpel2021-08-119-191/+555
| | | | | Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu> Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Only return state events that the AS passed in via `state_events_at_start` ↵Eric Eastwood2021-08-102-1/+4
| | | | | | | | | | | | | | (MSC2716) (#10552) * Only return state events that the AS passed in via state_events_at_start As discovered by @Half-Shot in https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684158448 Part of MSC2716 * Add changelog * Fix changelog extension
* Add local support for the new spaces summary endpoint (MSC2946) (#10549)Patrick Cloke2021-08-106-112/+521
| | | | | This adds support for the /hierarchy endpoint, which is an update to MSC2946. Currently this only supports rooms known locally to the homeserver.
* Fix an edge-case with invited rooms over federation in the spaces summary. ↵Patrick Cloke2021-08-103-62/+138
| | | | | | | (#10560) If a room which the requesting user was invited to was queried over federation it will now properly appear in the spaces summary (instead of being stripped out by the requesting server).
* Update contributing.md to warn against rebasing an open PR. (#10563)Hillery Shay2021-08-102-0/+2
| | | | Signed-off-by: H.Shay <shaysquared@gmail.com>
* Merge branch 'master' into developBrendan Abolivier2021-08-103-1/+13
|\
| * 1.40.0 v1.40.0Brendan Abolivier2021-08-103-1/+13
| |
* | When redacting, keep event fields around that maintain the historical event ↵Eric Eastwood2021-08-093-6/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | structure intact (MSC2716) (#10538) * Keep event fields that maintain the historical event structure intact Fix https://github.com/matrix-org/synapse/issues/10521 * Add changelog * Bump room version * Better changelog text * Fix up room version after develop merge
* | Do not remove `status_msg` when user going offline (#10550)Dirk Klimpel2021-08-093-9/+166
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Moved homeserver documentation above reverse proxy examples (#10551)Drew Short2021-08-092-10/+14
| | | | | | | | Signed-off-by: Drew Short <warrick@sothr.com>
* | Merge branch 'release-v1.40' into developBrendan Abolivier2021-08-0911-38/+61
|\|
| * 1.40.0rc3 v1.40.0rc3Brendan Abolivier2021-08-096-4/+28
| |
| * Support MSC3289: Room version 8 (#10449)Patrick Cloke2021-08-099-36/+35
| | | | | | This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.
* | Convert Transaction and Edu object to attrs (#10542)Patrick Cloke2021-08-068-194/+75
| | | | | | | | | | Instead of wrapping the JSON into an object, this creates concrete instances for Transaction and Edu. This allows for improved type hints and simplified code.
* | Fix exceptions in logs when failing to get remote room list (#10541)Erik Johnston2021-08-065-38/+134
| |
* | Clean up federation event auth code (#10539)Richard van der Hoff2021-08-063-56/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * drop old-room hack pretty sure we don't need this any more. * Remove incorrect comment about modifying `context` It doesn't look like the supplied context is ever modified. * Stop `_auth_and_persist_event` modifying its parameters This is only called in three places. Two of them don't pass `auth_events`, and the third doesn't use the dict after passing it in, so this should be non-functional. * Stop `_check_event_auth` modifying its parameters `_check_event_auth` is only called in three places. `on_send_membership_event` doesn't pass an `auth_events`, and `prep` and `_auth_and_persist_event` do not use the map after passing it in. * Stop `_update_auth_events_and_context_for_auth` modifying its parameters Return the updated auth event dict, rather than modifying the parameter. This is only called from `_check_event_auth`. * Improve documentation on `_auth_and_persist_event` Rename `auth_events` parameter to better reflect what it contains. * Improve documentation on `_NewEventInfo` * Improve documentation on `_check_event_auth` rename `auth_events` parameter to better describe what it contains * changelog
* | Update the API response for spaces summary over federation. (#10530)Patrick Cloke2021-08-062-19/+39
| | | | | | | | | | | | This adds 'allowed_room_ids' (in addition to 'allowed_spaces', for backwards compatibility) to the federation response of the spaces summary. A future PR will remove the 'allowed_spaces' flag.
* | Add a setting to disable TLS for sending email (#10546)Richard van der Hoff2021-08-068-50/+138
| | | | | | This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
* | Mark all MSC2716 events as historical (#10537)Eric Eastwood2021-08-052-5/+11
| | | | | | * Mark all MSC2716 events as historical
* | Merge remote-tracking branch 'origin/release-v1.40' into developRichard van der Hoff2021-08-052-28/+24
|\|
| * fix broken links in `upgrade.md` (#10543)Dirk Klimpel2021-08-052-28/+24
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Refactoring before implementing the updated spaces summary. (#10527)Patrick Cloke2021-08-054-136/+198
| | | | | | | | | | This should have no user-visible changes, but refactors some pieces of the SpaceSummaryHandler before adding support for the updated MSC2946.
* | Only return an appservice protocol if it has a service providing it. (#10532)Will Hunt2021-08-053-5/+125
| | | | | | | | | | | | If there are no services providing a protocol, omit it completely instead of returning an empty dictionary. This fixes a long-standing spec compliance bug.
* | Add documentation for configuring a forward proxy. (#10443)Dirk Klimpel2021-08-053-0/+76
| |
* | Don't fail CI when lint-newfile job was skipped (#10529)Erik Johnston2021-08-052-1/+7
| |
* | Merge tag 'v1.40.0rc2' into developErik Johnston2021-08-057-3/+38
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.40.0rc2 (2021-08-04) ============================== Bugfixes -------- - Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](https://github.com/matrix-org/synapse/issues/10517)) - Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](https://github.com/matrix-org/synapse/issues/10531)) Internal Changes ---------------- - Fix release script to open the correct URL for the release. ([\#10516](https://github.com/matrix-org/synapse/issues/10516))
| * Fixup changelog v1.40.0rc2Brendan Abolivier2021-08-041-1/+1
| |
| * Fixup changelogBrendan Abolivier2021-08-041-1/+1
| |
| * Fixup changelogBrendan Abolivier2021-08-041-1/+1
| |
| * 1.40.0rc2Brendan Abolivier2021-08-046-4/+23
| |
| * Don't fail on empty bodies when sending out read receipts (#10531)Brendan Abolivier2021-08-043-1/+14
| | | | | | | | | | Fixes a bug introduced in rc1 that would cause Synapse to 400 on read receipts requests with empty bodies. Broken in #10413
| * Fix PeriodicallyFlushingMemoryHandler inhibiting application shutdown (#10517)reivilibre2021-08-032-0/+2
| |
| * Fix release script URL (#10516)Erik Johnston2021-08-032-1/+2
| |
* | Send unstable-prefixed room_type in store-invite IS API requests (#10435)Michael Telatynski2021-08-043-1/+19
| | | | | | | | | | | | | | | | The room type is per MSC3288 to allow the identity-server to change invitation wording based on whether the invitation is to a room or a space. The prefixed key will be replaced once MSC3288 is accepted into the spec.
* | Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-048-35/+265
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* | Improve event caching code (#10119)Erik Johnston2021-08-044-43/+158
| | | | | | | | Ensure we only load an event from the DB once when the same event is requested multiple times at once.
* | Fix `could not serialize access` errors for `claim_e2e_one_time_keys` (#10504)Erik Johnston2021-08-042-62/+127
| |
* | Add `get_userinfo_by_id` method to `ModuleApi` (#9581)Jason Robinson2021-08-045-2/+80
| | | | | | | | | | | | Makes it easier to fetch user details in for example spam checker modules, without needing to use api._store or figure out database interactions. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Add warnings to ip_range_blacklist usage with proxies (#10129)Kento Okamoto2021-08-034-5/+26
| | | | | | | | | | | | | | | | Per issue #9812 using `url_preview_ip_range_blacklist` with a proxy via `HTTPS_PROXY` or `HTTP_PROXY` environment variables has some inconsistent bahavior than mentioned. This PR changes the following: - Changes the Sample Config file to include a note mentioning that `url_preview_ip_range_blacklist` and `ip_range_blacklist` is ignored when using a proxy - Changes some logic in synapse/config/repository.py to send a warning when both `*ip_range_blacklist` configs and a proxy environment variable are set and but no longer throws an error. Signed-off-by: Kento Okamoto <kentokamoto@protonmail.com>
* | Fix debian package triggers (#10481)Richard van der Hoff2021-08-033-9/+17
| | | | | | | | Replace the outdated list of dpkg triggers with an autogenerated one.
* | Make sync response cache time configurable. (#10513)Richard van der Hoff2021-08-034-3/+34
| |
* | Include room ID in ignored EDU log messages (#10507)Dagfinn Ilmari Mannsåker2021-08-033-2/+5
|/ | | | Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
* Fixup changelog v1.40.0rc1Erik Johnston2021-08-031-1/+1
|
* Fixup changelogErik Johnston2021-08-031-3/+1
|
* Fixup changelogErik Johnston2021-08-031-6/+6
|
* 1.40.0rc1Erik Johnston2021-08-0347-47/+67
|
* Revert use of PeriodicallyFlushingMemoryHandler by default (#10515)reivilibre2021-08-033-8/+3
|
* Extend the release script to tag and create the releases. (#10496)Erik Johnston2021-08-033-36/+278
|
* Add developer documentation to explain room DAG concepts like `outliers` and ↵Eric Eastwood2021-08-033-0/+81
| | | | `state_groups` (#10464)
* Fix the `tests-done` github actions step, again (#10512)Richard van der Hoff2021-08-022-9/+13
|
* Fix codestyle CI from #10440 (#10511)reivilibre2021-08-022-1/+2
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Prune inbound federation queues if they get too long (#10390)Erik Johnston2021-08-024-2/+177
|
* Allow setting transaction limit for db connections (#10440)Toni Spets2021-08-026-0/+69
| | | | | | | Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
* Fix deb build script to set prerelease flag correctly (#10500)Richard van der Hoff2021-08-012-5/+5
|
* Fix explicit assignment of PL 0 from being misinterpreted in rare ↵reivilibre2021-07-302-1/+2
| | | | circumstances (#10499)
* Remove shebang line from module files (#10415)V024602021-07-2917-16/+1
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Merge remote-tracking branch 'origin/master' into developErik Johnston2021-07-293-5/+14
|\
| * Merge tag 'v1.39.0'Erik Johnston2021-07-293-5/+14
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0 (2021-07-29) =========================== No significant changes. Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465)) Synapse 1.39.0rc2 (2021-07-22) ============================== This release also includes the changes in v1.38.1. Internal Changes ---------------- - Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416)) Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))
| | * Fixup changelog v1.39.0Erik Johnston2021-07-291-4/+1
| | |
| | * 1.39.0Erik Johnston2021-07-293-1/+13
| | |
* | | Improve failover logic for MSC3083 restricted rooms. (#10447)Patrick Cloke2021-07-292-4/+40
| | | | | | | | | | | | | | | If the federation client receives an M_UNABLE_TO_AUTHORISE_JOIN or M_UNABLE_TO_GRANT_JOIN response it will attempt another server before giving up completely.
* | | Merge branch 'master' into developErik Johnston2021-07-290-0/+0
|\| |
| * | Merge tag 'v1.39.0rc3'Erik Johnston2021-07-29265-2479/+4141
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
* | | Fix backfilled events being rejected for no `state_groups` (#10439)Eric Eastwood2021-07-292-0/+132
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reproducible on a federated homeserver when there is a membership auth event as a floating outlier. Then when we try to backfill one of that persons messages, it has missing membership auth to fetch which caused us to mistakenly replace the `context` for the message with that of the floating membership `outlier` event. Since `outliers` have no `state` or `state_group`, the error bubbles up when we continue down the persisting route: `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` Call stack: ``` backfill _auth_and_persist_event _check_event_auth _update_auth_events_and_context_for_auth ```
* | | Generics for `ObservableDeferred` (#10491)Richard van der Hoff2021-07-284-9/+15
| | | | | | | | | | | | | | | Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
* | | Make historical events discoverable from backfill for servers without any ↵Eric Eastwood2021-07-2812-26/+338
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489 We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509 * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
* | | Merge tag 'v1.39.0rc3' into developErik Johnston2021-07-2811-17/+81
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
| * | Fixup changelog v1.39.0rc3Erik Johnston2021-07-281-2/+2
| | |
| * | 1.39.0rc3Erik Johnston2021-07-288-6/+25
| | |
| * | Fix import of the default SAML mapping provider. (#10477)Jason Robinson2021-07-272-2/+1
| | | | | | | | | | | | Fix a circular import, which was causing exceptions on boot if SAML was configured.
| * | Fix typo that causes R30v2 to actually be old R30 (#10486)reivilibre2021-07-272-1/+2
| | | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
| * | Always communicate device OTK counts to clients (#10485)Andrew Morgan2021-07-275-6/+36
| | | | | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * | Change release script to update debian changelog for RCs (#10465)Erik Johnston2021-07-272-6/+21
| | |
* | | Fix up type hints for Twisted 21.7 (#10490)Richard van der Hoff2021-07-286-15/+25
| | | | | | | | | Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
* | | improve typing annotations in CachedCall (#10450)Richard van der Hoff2021-07-282-10/+18
| | | | | | | | | | | | | | | tighten up some of the typing in CachedCall, which is going to be needed when Twisted 21.7 brings better typing on Deferred.
* | | Restricted rooms (MSC3083) should not have their allow key redacted. (#10489)Patrick Cloke2021-07-283-0/+46
| | |
* | | Support for MSC2285 (hidden read receipts) (#10413)Šimon Brandner2021-07-2811-12/+495
| | | | | | | | | Implementation of matrix-org/matrix-doc#2285
* | | Document Complement dev usage (#10483)Eric Eastwood2021-07-272-2/+40
| | |
* | | Support underscores (in addition to hyphens) for charset detection. (#10410)sri-vidyut2021-07-273-2/+18
| | |
* | | Fix `oldest_pdu_in_federation_staging` (#10455)Erik Johnston2021-07-272-2/+6
| | | | | | | | | | | | If the staging area was empty we'd report an age of 51 years, which is not true or helpful.
* | | allow specifying https:// proxy (#10411)Dirk Klimpel2021-07-273-133/+450
| | |
* | | Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407)reivilibre2021-07-274-2/+97
| | | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | | Use new go test running syntax for complement. (#10488)Patrick Cloke2021-07-273-2/+3
| | | | | | | | | Updates CI and the helper script t ensures all tests are run (in parallel).
* | | Mitigate media repo XSSs on IE11. (#10468)Denis Kasak2021-07-272-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | IE11 doesn't support Content-Security-Policy but it has support for a non-standard X-Content-Security-Policy header, which only supports the sandbox directive. This prevents script execution, so it at least offers some protection against media repo-based attacks. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* | | Support MSC2033: Device ID on whoami (#9918)Travis Ralston2021-07-273-2/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix no-access-token bug in deactivation tests * Support MSC2033: Device ID on whoami * Test for appservices too MSC: https://github.com/matrix-org/matrix-doc/pull/2033 The MSC has passed FCP, which means stable endpoints can be used.
* | | Add type hints to state handler. (#10482)Patrick Cloke2021-07-264-19/+29
| | |
* | | Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-2617-98/+632
| | | | | | | | | | | | (#10254)
* | | Add type hints to synapse.federation.transport.client. (#10408)Patrick Cloke2021-07-262-201/+299
| | |
* | | Enable docker image caching for the deb build (#10431)Richard van der Hoff2021-07-263-13/+65
| | |
* | | Merge branch 'release-v1.39' into developErik Johnston2021-07-232-1/+2
|\| |
| * | Merge tag 'v1.38.1' into release-v1.39Erik Johnston2021-07-232-0/+15
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.1 (2021-07-22) =========================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457))
| * | | Fix an error in the docker workflow (#10461)Richard van der Hoff2021-07-222-1/+2
| | | |
* | | | Merge tag 'v1.39.0rc2' into developErik Johnston2021-07-233-2/+16
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc2 (2021-07-22) ============================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457)) Internal Changes ---------------- - Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416))
| * | | 1.39.0rc2 v1.39.0rc2Erik Johnston2021-07-224-3/+16
| | | |
| * | | Merge commit '7da24b975dfb10c277cf963dfddb88f55b1ca598' into release-v1.39Erik Johnston2021-07-222-4/+8
| |\ \ \
* | \ \ \ Merge branch 'master' into developErik Johnston2021-07-233-4/+22
|\ \ \ \ \ | | |_|_|/ | |/| | |
| * | | | Merge tag 'v1.38.1'Richard van der Hoff2021-07-234-5/+23
| |\ \ \ \ | | | |_|/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.1 (2021-07-22) =========================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457))
| | * | | 1.38.1 v1.38.1Erik Johnston2021-07-224-2/+16
| | | |/ | | |/|
| | * | Always send device_one_time_keys_count (#10457)David Baker2021-07-222-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | As per comment Fixes https://github.com/matrix-org/synapse/issues/10456 See also https://github.com/vector-im/element-android/issues/3725
* | | | Disable msc2716 until Complement update is merged (#10463)Eric Eastwood2021-07-222-1/+2
| | | |
* | | | Add `creation_ts` to list users admin API (#10448)Dirk Klimpel2021-07-226-33/+46
| | | | | | | | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | | | Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442)Erik Johnston2021-07-226-99/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.
* | | | Fix a handful of type annotations. (#10446)Richard van der Hoff2021-07-224-5/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog
* | | | Move dev/ docs to development/ (#10453)Eric Eastwood2021-07-229-7/+8
| | | |
* | | | Cancel redundant GHA workflows (#10451)Richard van der Hoff2021-07-223-0/+9
| | | |
* | | | Fix the tests-done Github Actions job (#10444)Richard van der Hoff2021-07-222-1/+19
| | | |
* | | | Drop xenial-support hacks (#10429)Richard van der Hoff2021-07-217-18/+22
| | | |
* | | | Add type hints to additional servlet functions (#10437)Patrick Cloke2021-07-216-66/+176
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.
* | | | Fix the hierarchy of OpenID providers in the docs. (#10445)Patrick Cloke2021-07-212-1/+2
| | | |
* | | | Update the notification email subject when invited to a space. (#10426)Patrick Cloke2021-07-213-2/+21
| | | |
* | | | Merge remote-tracking branch 'origin/release-v1.39' into developRichard van der Hoff2021-07-213-78/+73
|\ \ \ \ | | |_|/ | |/| |
| * | | Move the docker image build to Github Actions (#10416)Richard van der Hoff2021-07-213-78/+73
| | | | | | | | | | | | | | | | it's flaky on circleCI, and having to manage multiple CI providers is painful.
* | | | Add a return type to parse_string. (#10438)Patrick Cloke2021-07-2113-45/+86
| | | | | | | | | | | | | | | | And set the required attribute in a few places which will error if a parameter is not provided.
* | | | Switch to `chunk` events so we can auth via power_levels (MSC2716) (#10432)Eric Eastwood2021-07-213-6/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room.
* | | | Merge tag 'v1.39.0rc1' into developAndrew Morgan2021-07-2055-52/+94
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))
| * | | 1.39.0rc1 v1.39.0rc1Andrew Morgan2021-07-201-0/+6
| | | |
| * | | Incorporate changelog of #10433Andrew Morgan2021-07-202-1/+1
| | | |
| * | | Fix dropping locks on shut down (#10433)Erik Johnston2021-07-203-1/+19
| | | |
| * | | Combine some changelog lines in the documentation sectionAndrew Morgan2021-07-201-6/+3
| | | |
| * | | Update account validity feature line in changelogAndrew Morgan2021-07-201-1/+1
| | | |
| * | | fix typo in changelogAndrew Morgan2021-07-201-1/+1
| | | |
| * | | changelog word fixesAndrew Morgan2021-07-201-1/+1
| | | |
| * | | 1.39.0rc1Andrew Morgan2021-07-2052-51/+72
| | | |
* | | | MSC3244 room capabilities implementation (#10283)Michael Telatynski2021-07-205-3/+93
| | | |
* | | | Revert "Fix dropping locks on shut down"Erik Johnston2021-07-201-5/+1
| | | | | | | | | | | | | | | | This reverts commit 83f1ccfcaba76785ab4bd91e3177724e2dbb85ed.
* | | | Fix dropping locks on shut downErik Johnston2021-07-201-1/+5
|/ / /
* | | Add a github actions job recording success of other jobs. (#10430)Richard van der Hoff2021-07-202-0/+13
| | |
* | | Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier2021-07-2012-108/+403
| | | | | | | | | | | | | | | (#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
* | | Fix exception when failing to get remote room list (#10414)Erik Johnston2021-07-202-1/+6
| | |
* | | Factorise `get_datastore` calls in phone_stats_home. (#10427)reivilibre2021-07-192-16/+19
| | | | | | | | | Follow-up to #10332.
* | | Add a new version of the R30 phone-home metric, which removes a false ↵reivilibre2021-07-196-5/+416
| | | | | | | | | | | | | | | impression of retention given by the old R30 metric (#10332) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | | [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-1929-102/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* | | Remove unused `events_by_room` (#10421)Eric Eastwood2021-07-192-4/+1
| | | | | | | | | | | | | | | It looks like it was first used and introduced in https://github.com/matrix-org/synapse/commit/5130d80d79fe1f95ce03b8f1cfd4fbf0a32f5ac8#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305 but the But the usage was removed in https://github.com/matrix-org/synapse/commit/4c6a31cd6efa25be4c9f1b357e8f92065fac63eb#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8
* | | One last inline type hint (for the whole repo) (#10418)Jonathan de Jong2021-07-162-1/+2
| | |
* | | Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-1643-215/+212
| | |
* | | Add a module type for account validity (#9884)Brendan Abolivier2021-07-1613-228/+438
| | | | | | | | | | | | | | | This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
* | | Do not include signatures/hashes in make_{join,leave,knock} responses. (#10404)Patrick Cloke2021-07-163-6/+18
| | | | | | | | | | | | These signatures would end up invalid since the joining/leaving/knocking server would modify the response before calling send_{join,leave,knock}.
* | | Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-1538-162/+150
| | |
* | | Reduce likelihood of Postgres table scanning `state_groups_state`. (#10359)Erik Johnston2021-07-152-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | The postgres statistics collector sometimes massively underestimates the number of distinct state groups are in the `state_groups_state`, which can cause postgres to use table scans for queries for multiple state groups. We fix this by manually setting `n_distinct` on the column.
* | | Merge branch 'master' into developBrendan Abolivier2021-07-152-2/+5
|\| |
| * | Make deprecation notice of the spam checker doc more obvious (#10395)Brendan Abolivier2021-07-152-2/+5
| |/
* | Add tests to characterise the current behaviour of R30 phone-home metrics ↵reivilibre2021-07-152-0/+154
| | | | | | | | | | | | (#10315) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | Updating install prerequisites for newer macOS & ARM Macs. (#9971)Luke Walsh2021-07-152-3/+7
| |
* | Docs: Use something other than the document name to describe a page (#10399)Andrew Morgan2021-07-1512-23/+22
| | | | | | | | | | Our documentation has a history of using a document's name as a way to link to it, such as "See [workers.md]() for details". This makes sense when you're traversing a directory of files, but less sense when the files are abstracted away - as they are on the documentation website. This PR changes the links to various documentation pages to something that fits better into the surrounding sentence, as you would when making any hyperlink on the web.
* | Stagger send presence to remotes (#10398)Erik Johnston2021-07-154-5/+116
| | | | | | | | | | | | This is to help with performance, where trying to connect to thousands of hosts at once can consume a lot of CPU (due to TLS etc). Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Update the logcontext doc (#10353)Brendan Abolivier2021-07-152-230/+102
| | | | | | By referring to awaitables instead of deferreds.
* | Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-1579-336/+329
| |
* | Allow providing credentials to `http_proxy` (#10360)Dirk Klimpel2021-07-153-14/+64
| |
* | Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston2021-07-155-19/+66
| |
* | doc: Add delegation example to the caddy reverse proxy section (#10368)Moritz Dietz2021-07-142-0/+28
| |
* | Instructions on installing RC debs (#10396)Richard van der Hoff2021-07-142-9/+13
| |
* | Set section for prerelease debs (#10391)Richard van der Hoff2021-07-142-0/+15
| | | | | | This is part of fixing #6116: we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.
* | Make GHA config more efficient (#10383)Richard van der Hoff2021-07-143-15/+21
| | | | | | | | | | | | | | | | A few things here: * Build the debs for single distro for each PR, so that we can see if it breaks. Do the same for develop. Building all the debs ties up the GHA workers for ages. * Stop building the debs for release branches. Again, it takes ages, and I don't think anyone is actually going to stop and look at them. We'll know they are working when we make an RC. * Change the configs so that if we manually cancel a workflow, it actually does something.
* | Add type hints and comments to event auth code. (#10393)Patrick Cloke2021-07-144-10/+18
| |
* | Fix messages from multiple senders in historical chunk (MSC2716) (#10276)Eric Eastwood2021-07-133-11/+76
| | | | | | | | | | | | | | Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint. Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
* | Add type hints to get_domain_from_id and get_localpart_from_id. (#10385)Patrick Cloke2021-07-133-26/+75
| |
* | Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-136-38/+237
| | | | | | | | | | | | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* | Merge branch 'master' into developRichard van der Hoff2021-07-135-9/+133
|\|
| * Move upgrade blurb v1.38.0Richard van der Hoff2021-07-131-2/+2
| |
| * Update CHANGES.mdRichard van der Hoff2021-07-131-1/+1
| |
| * fix debian changelogRichard van der Hoff2021-07-131-1/+7
| |
| * 1.38.0Richard van der Hoff2021-07-133-2/+8
| |
| * 1.38.0rc3 v1.38.0rc3Richard van der Hoff2021-07-135-5/+16
| |
| * Github Actions workflow to attach release artifacts to release (#10379)Richard van der Hoff2021-07-132-1/+28
| |
| * Build the python release artifacts in GHA tooRichard van der Hoff2021-07-131-3/+23
| |
| * build debs in GHA (#10247)Richard van der Hoff2021-07-123-4/+58
| | | | | | GHA workflow to build the debs
* | Fix some links in `docs` and `contrib` (#10370)Dirk Klimpel2021-07-139-17/+23
| |
* | Bugfix `make_room_admin` fails for users that have left a private room (#10367)Dirk Klimpel2021-07-132-1/+9
| | | | | | Fixes: #10338
* | Use inline type hints in `tests/` (#10350)Jonathan de Jong2021-07-1318-63/+62
| | | | | | | | | | | | | | | | This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
* | [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-1320-58/+45
| |
* | Fix federation inbound age metric. (#10355)Erik Johnston2021-07-132-1/+4
| | | | | | We should be reporting the age rather than absolute timestamp.
* | Replace `room_depth.min_depth` with a BIGINT (#10289)Richard van der Hoff2021-07-124-7/+186
| | | | | | | | | | while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth with a BIGINT.
* | Additional unit tests for spaces summary. (#10305)Patrick Cloke2021-07-122-1/+204
| |
* | Unblacklist fixed tests (#10357)Erik Johnston2021-07-093-5/+2
| |
* | Re-enable room v6 sytest (#10345)Richard van der Hoff2021-07-092-1/+1
| | | | | | ... now that it has been fixed in https://github.com/matrix-org/sytest/pull/1061.
* | Revert "Newsfile"Erik Johnston2021-07-091-1/+0
| | | | | | | | This reverts commit 944428d1163d1521ef96db88040852520ad6cbff.
* | NewsfileErik Johnston2021-07-091-0/+1
| |
* | Merge remote-tracking branch 'origin/release-v1.38' into developErik Johnston2021-07-091-2/+2
|\|
| * Fix README rstErik Johnston2021-07-091-5/+4
| |
| * Fix README rstErik Johnston2021-07-091-4/+5
| |
* | Fix the user directory becoming broken (and noisy errors being logged) when ↵reivilibre2021-07-092-2/+7
| | | | | | | | | | knocking and room statistics are in use. (#10344) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | Switch `application_services_txns.txn_id` to BIGINT (#10349)Richard van der Hoff2021-07-092-0/+24
| |
* | Merge tag 'v1.38.0rc2' into developErik Johnston2021-07-0916-621/+659
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
| * Fixup changelog v1.38.0rc2Erik Johnston2021-07-091-1/+1
| |
| * Fixup changelogErik Johnston2021-07-091-1/+1
| |
| * 1.38.0rc2Erik Johnston2021-07-095-4/+17
| |
| * Ensure we always drop the federation inbound lock (#10336)Erik Johnston2021-07-093-2/+15
| |
| * Fix broken links in INSTALL.md (#10331)Dirk Klimpel2021-07-0813-617/+629
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Upsert redactions in case they already exists (#10343)Andreas Rammhold2021-07-093-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Upsert redactions in case they already exists Occasionally, in combination with retention, redactions aren't deleted from the database whenever they are due for deletion. The server will eventually try to backfill the deleted events and trip over the already existing redaction events. Switching to an UPSERT for those events allows us to recover from there situations. The retention code still needs fixing but that is outside of my current comfort zone on this code base. This is related to #8707 where the error was discussed already. Signed-off-by: Andreas Rammhold <andreas@rammhold.de> * Also purge redactions when purging events Previously redacints where left behind leading to backfilling issues when the server stumbled across the already existing yet to be backfilled redactions. This issues has been discussed in #8707. Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
* | Add base starting insertion event when no chunk ID is provided (MSC2716) ↵Eric Eastwood2021-07-083-23/+98
| | | | | | | | | | | | | | | | | | (#10250) * Add base starting insertion point when no chunk ID is provided This is so we can have the marker event point to this initial insertion event and be able to traverse the events in the first chunk.
* | Fix formatting in the logcontext doc (#10337)Brendan Abolivier2021-07-082-1/+2
| |
* | Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-0810-572/+22
| | | | | | Fixes #9602
* | Minor changes to `user_daily_visits` (#10324)reivilibre2021-07-082-2/+3
| | | | | | | | | | | | | | * Use fake time in tests in _get_start_of_day. * Change the inequality of last_seen in user_daily_visits Co-authored-by: Erik Johnston <erik@matrix.org>
* | Rebuild event auth when rebuilding an event after a call to a ↵Brendan Abolivier2021-07-082-4/+7
| | | | | | | | | | `ThirdPartyEventRules` module (#10316) Because modules might send extra state events when processing an event (e.g. matrix-org/synapse-dinsic#100), and in some cases these extra events might get dropped if we don't recalculate the initial event's auth.
* | Fix a broken link in the admin api docs (#10322)Richard van der Hoff2021-07-072-2/+3
| | | | | | | | | | | | | | * Fix a broken link in the admin api docs * Rename 10321.doc to 10321.docs * Rename 10321.docs to 10322.doc
* | Merge remote-tracking branch 'origin/release-v1.38' into developRichard van der Hoff2021-07-0716-46/+51
|\| | | | | | | merge @dklimpel's fixes to the sample config
| * Update links to documentation in sample config (#10287)Dirk Klimpel2021-07-0716-46/+51
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Simplify structure of room admin API docs (#10313)Dirk Klimpel2021-07-072-45/+25
| |
* | Merge tag 'v1.38.0rc1' into developBrendan Abolivier2021-07-072-3/+2
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.0rc1 (2021-07-06) ============================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. Features -------- - Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450)) - Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263)) - Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284)) - Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314), [\#10326](https://github.com/matrix-org/synapse/issues/10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114)) - Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239)) - Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292))
| * Fix broken link v1.38.0rc1Brendan Abolivier2021-07-071-1/+1
| |
| * Merge latest fix into the changelogBrendan Abolivier2021-07-072-2/+1
| |
* | Merge branch 'release-v1.38' into developRichard van der Hoff2021-07-0736-64/+87
|\|
| * Merge branch 'master' into release-v1.38Richard van der Hoff2021-07-071-30/+27
| |\
| | * build the docs for master (#10323)Richard van der Hoff2021-07-071-31/+28
| | |
| * | ANALYZE new stream ordering column (#10326)Brendan Abolivier2021-07-072-0/+11
| | | | | | | | | Fixes #10325
| * | Small changelog tweaksBrendan Abolivier2021-07-061-6/+4
| | |
| * | 1.38.0rc1Brendan Abolivier2021-07-0634-33/+50
| | |
* | | Ignore EDUs for rooms we're not in (#10317)Dagfinn Ilmari Mannsåker2021-07-064-0/+67
|/ /
* | Fix deactivate a user if he does not have a profile (#10252)Dirk Klimpel2021-07-063-22/+73
| |
* | Add script for getting info about recently registered users (#10290)Erik Johnston2021-07-0617-153/+309
| |
* | Handle old staged inbound events (#10303)Erik Johnston2021-07-063-10/+67
| | | | | | | | | | | | | | We might have events in the staging area if the service was restarted while there were unhandled events in the staging area. Fixes #10295
* | Add upgrade notes about disk space for events migration (#10314)Richard van der Hoff2021-07-063-1/+44
| |
* | Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-059-95/+485
| |
* | Fix bad link in modules documentation (#10302)Brendan Abolivier2021-07-022-1/+2
| | | | | | Fix link in modules doc to point at instructions on registering a callback instead of ones on registering a web resource.
* | Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-0111-106/+112
| | | | | | Instead of mixing them with user authentication methods.
* | fix ordering of bg update (#10291)Richard van der Hoff2021-07-012-1/+2
| | | | | | | | | | this was a typo introduced in #10282. We don't want to end up doing the `replace_stream_ordering_column` update after anything that comes up in migration 60/03.
* | Reenable 'Backfilled events whose prev_events...' sytest (#10292)Erik Johnston2021-07-012-1/+1
| | | | | | Now that we've fixed it.
* | Add SSO `external_ids` to Query User Account admin API (#10261)Dirk Klimpel2021-07-014-85/+159
| | | | | | Related to #10251
* | Add some metrics to staging area (#10284)Erik Johnston2021-07-012-0/+40
| |
* | Fix the homeserver config example in presence router docs (#10288)Andrew Morgan2021-06-302-1/+4
| | | | | | | | | | The presence router docs include some sample homeserver config. At some point we changed the name of the [config option](https://github.com/matrix-org/synapse/blob/859dc05b3692a3672c1a0db8deaaa9274b6aa6f5/docs/sample_config.yaml#L104-L113), but forgot to update the docs. I've also added `presence.enabled: true` to the example, as that's the new way to enable presence (the `presence_enabled` option has been deprecated).
* | Change more stream_ordering columns to BIGINT (#10286)Richard van der Hoff2021-06-302-0/+31
| |
* | Rebuild other indexes using `stream_ordering` (#10282)Richard van der Hoff2021-06-303-6/+56
| | | | | | | | We need to rebuild *all* of the indexes that use the current `stream_ordering` column.
* | Merge branch 'master' into developErik Johnston2021-06-303-1/+15
|\|
| * Fixup changelog v1.37.1Erik Johnston2021-06-301-1/+1
| |
| * Add note to changelogErik Johnston2021-06-301-1/+3
| |
| * 1.37.1Erik Johnston2021-06-303-1/+13
| |
* | Correct type hints for synapse.event_auth. (#10253)Patrick Cloke2021-06-306-38/+51
| |
* | Fix the inbound PDU metric (#10279)Erik Johnston2021-06-306-27/+93
| | | | | | This broke in #10272
* | Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-294-3/+10
|\|
| * 1.37.1rc1 v1.37.1rc1Richard van der Hoff2021-06-294-3/+10
| |
* | Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-2912-6/+718
|\|
| * Update newsfiles v1.37.1a1Richard van der Hoff2021-06-292-1/+1
| |
| * v1.37.1a1Richard van der Hoff2021-06-291-1/+1
| |
| * Merge remote-tracking branch 'origin/erikj/async_federation_base_branch' ↵Richard van der Hoff2021-06-2911-5/+717
| |\ | | | | | | | | | into release-v1.37
| | * Handle inbound events from federation asynchronously (#10272)Erik Johnston2021-06-295-5/+241
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #9490 This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous. Two drawbacks: Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately. We now only process one event per room at a time. This can be fixed up further down the line.
| | * Add a distributed lock (#10269)Erik Johnston2021-06-296-0/+476
| | | | | | | | | This adds a simple best effort locking mechanism that works cross workers.
* | | Do not recurse into non-spaces in the spaces summary. (#10256)Patrick Cloke2021-06-295-26/+43
| | | | | | | | | | | | | | | Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.
* | | Fix `populate_stream_ordering2` background job (#10267)Richard van der Hoff2021-06-292-16/+13
| | | | | | | | | | | | It was possible for us not to find any rows in a batch, and hence conclude that we had finished. Let's not do that.
* | | Migrate stream_ordering to a bigint (#10264)Richard van der Hoff2021-06-294-16/+163
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move background update names out to a separate class `EventsBackgroundUpdatesStore` gets inherited and we don't really want to further pollute the namespace. * Migrate stream_ordering to a bigint * changelog
* | | Soft-fail spammy events received over federation (#10263)Richard van der Hoff2021-06-292-6/+7
| | |
* | | Merge branch 'master' into developBrendan Abolivier2021-06-293-3/+11
|\| |
| * | Move deprecation notices to the top of the changelog v1.37.0Brendan Abolivier2021-06-291-6/+2
| | |
| * | 1.37.0Brendan Abolivier2021-06-293-1/+13
| |/
* | Add additional types to the federation transport server. (#10213)Patrick Cloke2021-06-283-118/+521
| |
* | Adjust the URL in the README.rst file to point to LiberaChat instead of ↵Felix Kronlage-Dammers2021-06-282-1/+2
| | | | | | | | freenode (#10258)
* | Create an index.html file when generating a docs build (#10242)Andrew Morgan2021-06-242-1/+7
| | | | | | | | | | | | | | | | | | Currently when a new build of the docs is created, an `index.html` file does not exist. Typically this would be generated from a`docs/README.md` file - which we have - however we're currently using [docs/README.md](https://github.com/matrix-org/synapse/blob/394673055db4df49bfd58c2f6118834a6d928563/docs/README.md) to explain the docs and point to the website. It is not part of the content of the website. So we end up not having an `index.html` file, which will result in a 404 page if one tries to navigate to `https://matrix-org.github.io/synapse/<docs_version>/index.html`. This isn't a really problem for the default version of the documentation (currently `develop`), as [navigating to the top-level root](https://matrix-org.github.io/synapse/) of the website (without specifying a version) will [redirect](https://github.com/matrix-org/synapse/blob/a77e6925f26597958eccf0ef9956cb13c536e57e/index.html#L2) you to the Welcome and Overview page of the `develop` docs version. However, ideally once we add a GUI for switching between versions, we'll want to send the user to `matrix-org.github.io/synapse/<version>/index.html`, which currently isn't generated. This PR modifies the CI that builds the docs to simply copy the rendered [Welcome & Overview page](https://matrix-org.github.io/synapse/develop/welcome_and_overview.html) to `index.html`.
* | Return errors from `send_join` etc if the event is rejected (#10243)Richard van der Hoff2021-06-243-10/+41
| | | | | | Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
* | Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-246-183/+132
| | | | | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* | MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-2415-61/+892
| | | | | | | | | | | | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* | Merge tag 'v1.37.0rc1' into developBrendan Abolivier2021-06-2450-53/+155
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-27 10:52:08 +0000