summary refs log tree commit diff
path: root/tests (unfollow)
Commit message (Collapse)AuthorFilesLines
2021-08-11Fix type hints in space summary tests. (#10575)Patrick Cloke2-9/+8
And ensure that the file is checked via mypy.
2021-08-11support federation queries through http connect proxy (#10475)Dirk Klimpel9-191/+555
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu> Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-10Only return state events that the AS passed in via `state_events_at_start` ↵Eric Eastwood2-1/+4
(MSC2716) (#10552) * Only return state events that the AS passed in via state_events_at_start As discovered by @Half-Shot in https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684158448 Part of MSC2716 * Add changelog * Fix changelog extension
2021-08-10Add local support for the new spaces summary endpoint (MSC2946) (#10549)Patrick Cloke6-112/+521
This adds support for the /hierarchy endpoint, which is an update to MSC2946. Currently this only supports rooms known locally to the homeserver.
2021-08-10Fix an edge-case with invited rooms over federation in the spaces summary. ↵Patrick Cloke3-62/+138
(#10560) If a room which the requesting user was invited to was queried over federation it will now properly appear in the spaces summary (instead of being stripped out by the requesting server).
2021-08-10Update contributing.md to warn against rebasing an open PR. (#10563)Hillery Shay2-0/+2
Signed-off-by: H.Shay <shaysquared@gmail.com>
2021-08-10 1.40.0 v1.40.0Brendan Abolivier3-1/+13
2021-08-09When redacting, keep event fields around that maintain the historical event ↵Eric Eastwood3-6/+40
structure intact (MSC2716) (#10538) * Keep event fields that maintain the historical event structure intact Fix https://github.com/matrix-org/synapse/issues/10521 * Add changelog * Bump room version * Better changelog text * Fix up room version after develop merge
2021-08-09Do not remove `status_msg` when user going offline (#10550)Dirk Klimpel3-9/+166
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-09Moved homeserver documentation above reverse proxy examples (#10551)Drew Short2-10/+14
Signed-off-by: Drew Short <warrick@sothr.com>
2021-08-09 1.40.0rc3 v1.40.0rc3Brendan Abolivier6-4/+28
2021-08-09Support MSC3289: Room version 8 (#10449)Patrick Cloke9-36/+35
This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.
2021-08-06Convert Transaction and Edu object to attrs (#10542)Patrick Cloke8-194/+75
Instead of wrapping the JSON into an object, this creates concrete instances for Transaction and Edu. This allows for improved type hints and simplified code.
2021-08-06Fix exceptions in logs when failing to get remote room list (#10541)Erik Johnston5-38/+134
2021-08-06Clean up federation event auth code (#10539)Richard van der Hoff3-56/+69
* drop old-room hack pretty sure we don't need this any more. * Remove incorrect comment about modifying `context` It doesn't look like the supplied context is ever modified. * Stop `_auth_and_persist_event` modifying its parameters This is only called in three places. Two of them don't pass `auth_events`, and the third doesn't use the dict after passing it in, so this should be non-functional. * Stop `_check_event_auth` modifying its parameters `_check_event_auth` is only called in three places. `on_send_membership_event` doesn't pass an `auth_events`, and `prep` and `_auth_and_persist_event` do not use the map after passing it in. * Stop `_update_auth_events_and_context_for_auth` modifying its parameters Return the updated auth event dict, rather than modifying the parameter. This is only called from `_check_event_auth`. * Improve documentation on `_auth_and_persist_event` Rename `auth_events` parameter to better reflect what it contains. * Improve documentation on `_NewEventInfo` * Improve documentation on `_check_event_auth` rename `auth_events` parameter to better describe what it contains * changelog
2021-08-06Update the API response for spaces summary over federation. (#10530)Patrick Cloke2-19/+39
This adds 'allowed_room_ids' (in addition to 'allowed_spaces', for backwards compatibility) to the federation response of the spaces summary. A future PR will remove the 'allowed_spaces' flag.
2021-08-06Add a setting to disable TLS for sending email (#10546)Richard van der Hoff8-50/+138
This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
2021-08-05Mark all MSC2716 events as historical (#10537)Eric Eastwood2-5/+11
* Mark all MSC2716 events as historical
2021-08-05fix broken links in `upgrade.md` (#10543)Dirk Klimpel2-28/+24
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-05Refactoring before implementing the updated spaces summary. (#10527)Patrick Cloke4-136/+198
This should have no user-visible changes, but refactors some pieces of the SpaceSummaryHandler before adding support for the updated MSC2946.
2021-08-05Only return an appservice protocol if it has a service providing it. (#10532)Will Hunt3-5/+125
If there are no services providing a protocol, omit it completely instead of returning an empty dictionary. This fixes a long-standing spec compliance bug.
2021-08-05Add documentation for configuring a forward proxy. (#10443)Dirk Klimpel3-0/+76
2021-08-05Don't fail CI when lint-newfile job was skipped (#10529)Erik Johnston2-1/+7
2021-08-04Send unstable-prefixed room_type in store-invite IS API requests (#10435)Michael Telatynski3-1/+19
The room type is per MSC3288 to allow the identity-server to change invitation wording based on whether the invitation is to a room or a space. The prefixed key will be replaced once MSC3288 is accepted into the spec.
2021-08-04Add support for MSC2716 marker events (#10498)Eric Eastwood8-35/+265
* Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
2021-08-04Fixup changelog v1.40.0rc2Brendan Abolivier1-1/+1
2021-08-04Fixup changelogBrendan Abolivier1-1/+1
2021-08-04Fixup changelogBrendan Abolivier1-1/+1
2021-08-04 1.40.0rc2Brendan Abolivier6-4/+23
2021-08-04Don't fail on empty bodies when sending out read receipts (#10531)Brendan Abolivier3-1/+14
Fixes a bug introduced in rc1 that would cause Synapse to 400 on read receipts requests with empty bodies. Broken in #10413
2021-08-04Improve event caching code (#10119)Erik Johnston4-43/+158
Ensure we only load an event from the DB once when the same event is requested multiple times at once.
2021-08-04Fix `could not serialize access` errors for `claim_e2e_one_time_keys` (#10504)Erik Johnston2-62/+127
2021-08-04Add `get_userinfo_by_id` method to `ModuleApi` (#9581)Jason Robinson5-2/+80
Makes it easier to fetch user details in for example spam checker modules, without needing to use api._store or figure out database interactions. Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-08-03Add warnings to ip_range_blacklist usage with proxies (#10129)Kento Okamoto4-5/+26
Per issue #9812 using `url_preview_ip_range_blacklist` with a proxy via `HTTPS_PROXY` or `HTTP_PROXY` environment variables has some inconsistent bahavior than mentioned. This PR changes the following: - Changes the Sample Config file to include a note mentioning that `url_preview_ip_range_blacklist` and `ip_range_blacklist` is ignored when using a proxy - Changes some logic in synapse/config/repository.py to send a warning when both `*ip_range_blacklist` configs and a proxy environment variable are set and but no longer throws an error. Signed-off-by: Kento Okamoto <kentokamoto@protonmail.com>
2021-08-03Fix debian package triggers (#10481)Richard van der Hoff3-9/+17
Replace the outdated list of dpkg triggers with an autogenerated one.
2021-08-03Make sync response cache time configurable. (#10513)Richard van der Hoff4-3/+34
2021-08-03Include room ID in ignored EDU log messages (#10507)Dagfinn Ilmari Mannsåker3-2/+5
Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2021-08-03Fix PeriodicallyFlushingMemoryHandler inhibiting application shutdown (#10517)reivilibre2-0/+2
2021-08-03Fix release script URL (#10516)Erik Johnston2-1/+2
2021-08-03Fixup changelog v1.40.0rc1Erik Johnston1-1/+1
2021-08-03Fixup changelogErik Johnston1-3/+1
2021-08-03Fixup changelogErik Johnston1-6/+6
2021-08-03 1.40.0rc1Erik Johnston47-47/+67
2021-08-03Revert use of PeriodicallyFlushingMemoryHandler by default (#10515)reivilibre3-8/+3
2021-08-03Extend the release script to tag and create the releases. (#10496)Erik Johnston3-36/+278
2021-08-03Add developer documentation to explain room DAG concepts like `outliers` and ↵Eric Eastwood3-0/+81
`state_groups` (#10464)
2021-08-02Fix the `tests-done` github actions step, again (#10512)Richard van der Hoff2-9/+13
2021-08-02Fix codestyle CI from #10440 (#10511)reivilibre2-1/+2
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-08-02Prune inbound federation queues if they get too long (#10390)Erik Johnston4-2/+177
2021-08-02Allow setting transaction limit for db connections (#10440)Toni Spets6-0/+69
Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
2021-08-01Fix deb build script to set prerelease flag correctly (#10500)Richard van der Hoff2-5/+5
2021-07-30Fix explicit assignment of PL 0 from being misinterpreted in rare ↵reivilibre2-1/+2
circumstances (#10499)
2021-07-29Remove shebang line from module files (#10415)V0246017-16/+1
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2021-07-29Improve failover logic for MSC3083 restricted rooms. (#10447)Patrick Cloke2-4/+40
If the federation client receives an M_UNABLE_TO_AUTHORISE_JOIN or M_UNABLE_TO_GRANT_JOIN response it will attempt another server before giving up completely.
2021-07-29Fixup changelog v1.39.0Erik Johnston1-4/+1
2021-07-29 1.39.0Erik Johnston3-1/+13
2021-07-29Fix backfilled events being rejected for no `state_groups` (#10439)Eric Eastwood2-0/+132
Reproducible on a federated homeserver when there is a membership auth event as a floating outlier. Then when we try to backfill one of that persons messages, it has missing membership auth to fetch which caused us to mistakenly replace the `context` for the message with that of the floating membership `outlier` event. Since `outliers` have no `state` or `state_group`, the error bubbles up when we continue down the persisting route: `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` Call stack: ``` backfill _auth_and_persist_event _check_event_auth _update_auth_events_and_context_for_auth ```
2021-07-28Generics for `ObservableDeferred` (#10491)Richard van der Hoff4-9/+15
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
2021-07-28Make historical events discoverable from backfill for servers without any ↵Eric Eastwood12-26/+338
scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489 We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509 * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
2021-07-28Fixup changelog v1.39.0rc3Erik Johnston1-2/+2
2021-07-28 1.39.0rc3Erik Johnston8-6/+25
2021-07-28Fix up type hints for Twisted 21.7 (#10490)Richard van der Hoff6-15/+25
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
2021-07-28improve typing annotations in CachedCall (#10450)Richard van der Hoff2-10/+18
tighten up some of the typing in CachedCall, which is going to be needed when Twisted 21.7 brings better typing on Deferred.
2021-07-28Restricted rooms (MSC3083) should not have their allow key redacted. (#10489)Patrick Cloke3-0/+46
2021-07-28Support for MSC2285 (hidden read receipts) (#10413)Šimon Brandner11-12/+495
Implementation of matrix-org/matrix-doc#2285
2021-07-27Document Complement dev usage (#10483)Eric Eastwood2-2/+40
2021-07-27Support underscores (in addition to hyphens) for charset detection. (#10410)sri-vidyut3-2/+18
2021-07-27Fix `oldest_pdu_in_federation_staging` (#10455)Erik Johnston2-2/+6
If the staging area was empty we'd report an age of 51 years, which is not true or helpful.
2021-07-27allow specifying https:// proxy (#10411)Dirk Klimpel3-133/+450
2021-07-27Fix import of the default SAML mapping provider. (#10477)Jason Robinson2-2/+1
Fix a circular import, which was causing exceptions on boot if SAML was configured.
2021-07-27Fix typo that causes R30v2 to actually be old R30 (#10486)reivilibre2-1/+2
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-27Always communicate device OTK counts to clients (#10485)Andrew Morgan5-6/+36
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-07-27Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407)reivilibre4-2/+97
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-27Use new go test running syntax for complement. (#10488)Patrick Cloke3-2/+3
Updates CI and the helper script t ensures all tests are run (in parallel).
2021-07-27Mitigate media repo XSSs on IE11. (#10468)Denis Kasak2-0/+3
IE11 doesn't support Content-Security-Policy but it has support for a non-standard X-Content-Security-Policy header, which only supports the sandbox directive. This prevents script execution, so it at least offers some protection against media repo-based attacks. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-07-27Change release script to update debian changelog for RCs (#10465)Erik Johnston2-6/+21
2021-07-27Support MSC2033: Device ID on whoami (#9918)Travis Ralston3-2/+51
* Fix no-access-token bug in deactivation tests * Support MSC2033: Device ID on whoami * Test for appservices too MSC: https://github.com/matrix-org/matrix-doc/pull/2033 The MSC has passed FCP, which means stable endpoints can be used.
2021-07-26Add type hints to state handler. (#10482)Patrick Cloke4-19/+29
2021-07-26Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke17-98/+632
(#10254)
2021-07-26Add type hints to synapse.federation.transport.client. (#10408)Patrick Cloke2-201/+299
2021-07-26Enable docker image caching for the deb build (#10431)Richard van der Hoff3-13/+65
2021-07-22Fix an error in the docker workflow (#10461)Richard van der Hoff2-1/+2
2021-07-22Disable msc2716 until Complement update is merged (#10463)Eric Eastwood2-1/+2
2021-07-22 1.39.0rc2 v1.39.0rc2Erik Johnston4-3/+16
2021-07-22 1.38.1 v1.38.1Erik Johnston4-2/+16
2021-07-22Always send device_one_time_keys_count (#10457)David Baker2-4/+8
As per comment Fixes https://github.com/matrix-org/synapse/issues/10456 See also https://github.com/vector-im/element-android/issues/3725
2021-07-22Add `creation_ts` to list users admin API (#10448)Dirk Klimpel6-33/+46
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-22Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442)Erik Johnston6-99/+44
Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.
2021-07-22Fix a handful of type annotations. (#10446)Richard van der Hoff4-5/+6
* switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog
2021-07-22Move dev/ docs to development/ (#10453)Eric Eastwood9-7/+8
2021-07-22Cancel redundant GHA workflows (#10451)Richard van der Hoff3-0/+9
2021-07-22Fix the tests-done Github Actions job (#10444)Richard van der Hoff2-1/+19
2021-07-21Drop xenial-support hacks (#10429)Richard van der Hoff7-18/+22
2021-07-21Add type hints to additional servlet functions (#10437)Patrick Cloke6-66/+176
Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.
2021-07-21Fix the hierarchy of OpenID providers in the docs. (#10445)Patrick Cloke2-1/+2
2021-07-21Update the notification email subject when invited to a space. (#10426)Patrick Cloke3-2/+21
2021-07-21Add a return type to parse_string. (#10438)Patrick Cloke13-45/+86
And set the required attribute in a few places which will error if a parameter is not provided.
2021-07-21Move the docker image build to Github Actions (#10416)Richard van der Hoff3-78/+73
it's flaky on circleCI, and having to manage multiple CI providers is painful.
2021-07-21Switch to `chunk` events so we can auth via power_levels (MSC2716) (#10432)Eric Eastwood3-6/+18
Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room.
2021-07-201.39.0rc1 v1.39.0rc1Andrew Morgan1-0/+6
2021-07-20Incorporate changelog of #10433Andrew Morgan2-1/+1
2021-07-20Fix dropping locks on shut down (#10433)Erik Johnston3-1/+19
2021-07-20Combine some changelog lines in the documentation sectionAndrew Morgan1-6/+3
2021-07-20Update account validity feature line in changelogAndrew Morgan1-1/+1
2021-07-20MSC3244 room capabilities implementation (#10283)Michael Telatynski5-3/+93
2021-07-20fix typo in changelogAndrew Morgan1-1/+1
2021-07-20changelog word fixesAndrew Morgan1-1/+1
2021-07-20 1.39.0rc1Andrew Morgan52-51/+72
2021-07-20Revert "Fix dropping locks on shut down"Erik Johnston1-5/+1
This reverts commit 83f1ccfcaba76785ab4bd91e3177724e2dbb85ed.
2021-07-20Fix dropping locks on shut downErik Johnston1-1/+5
2021-07-20Add a github actions job recording success of other jobs. (#10430)Richard van der Hoff2-0/+13
2021-07-20Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier12-108/+403
(#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
2021-07-20Fix exception when failing to get remote room list (#10414)Erik Johnston2-1/+6
2021-07-19Factorise `get_datastore` calls in phone_stats_home. (#10427)reivilibre2-16/+19
Follow-up to #10332.
2021-07-19Add a new version of the R30 phone-home metric, which removes a false ↵reivilibre6-5/+416
impression of retention given by the old R30 metric (#10332) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-19[pyupgrade] `synapse/` (#10348)Jonathan de Jong29-102/+86
This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
2021-07-19Remove unused `events_by_room` (#10421)Eric Eastwood2-4/+1
It looks like it was first used and introduced in https://github.com/matrix-org/synapse/commit/5130d80d79fe1f95ce03b8f1cfd4fbf0a32f5ac8#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305 but the But the usage was removed in https://github.com/matrix-org/synapse/commit/4c6a31cd6efa25be4c9f1b357e8f92065fac63eb#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8
2021-07-16One last inline type hint (for the whole repo) (#10418)Jonathan de Jong2-1/+2
2021-07-16Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong43-215/+212
2021-07-16Add a module type for account validity (#9884)Brendan Abolivier13-228/+438
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16Do not include signatures/hashes in make_{join,leave,knock} responses. (#10404)Patrick Cloke3-6/+18
These signatures would end up invalid since the joining/leaving/knocking server would modify the response before calling send_{join,leave,knock}.
2021-07-15Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong38-162/+150
2021-07-15Reduce likelihood of Postgres table scanning `state_groups_state`. (#10359)Erik Johnston2-0/+35
The postgres statistics collector sometimes massively underestimates the number of distinct state groups are in the `state_groups_state`, which can cause postgres to use table scans for queries for multiple state groups. We fix this by manually setting `n_distinct` on the column.
2021-07-15Make deprecation notice of the spam checker doc more obvious (#10395)Brendan Abolivier2-2/+5
2021-07-15Add tests to characterise the current behaviour of R30 phone-home metrics ↵reivilibre2-0/+154
(#10315) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-15Updating install prerequisites for newer macOS & ARM Macs. (#9971)Luke Walsh2-3/+7
2021-07-15Docs: Use something other than the document name to describe a page (#10399)Andrew Morgan12-23/+22
Our documentation has a history of using a document's name as a way to link to it, such as "See [workers.md]() for details". This makes sense when you're traversing a directory of files, but less sense when the files are abstracted away - as they are on the documentation website. This PR changes the links to various documentation pages to something that fits better into the surrounding sentence, as you would when making any hyperlink on the web.
2021-07-15Stagger send presence to remotes (#10398)Erik Johnston4-5/+116
This is to help with performance, where trying to connect to thousands of hosts at once can consume a lot of CPU (due to TLS etc). Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-07-15Update the logcontext doc (#10353)Brendan Abolivier2-230/+102
By referring to awaitables instead of deferreds.
2021-07-15Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong79-336/+329
2021-07-15Allow providing credentials to `http_proxy` (#10360)Dirk Klimpel3-14/+64
2021-07-15Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston5-19/+66
2021-07-14doc: Add delegation example to the caddy reverse proxy section (#10368)Moritz Dietz2-0/+28
2021-07-14Instructions on installing RC debs (#10396)Richard van der Hoff2-9/+13
2021-07-14Set section for prerelease debs (#10391)Richard van der Hoff2-0/+15
This is part of fixing #6116: we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.
2021-07-14Make GHA config more efficient (#10383)Richard van der Hoff3-15/+21
A few things here: * Build the debs for single distro for each PR, so that we can see if it breaks. Do the same for develop. Building all the debs ties up the GHA workers for ages. * Stop building the debs for release branches. Again, it takes ages, and I don't think anyone is actually going to stop and look at them. We'll know they are working when we make an RC. * Change the configs so that if we manually cancel a workflow, it actually does something.
2021-07-14Add type hints and comments to event auth code. (#10393)Patrick Cloke4-10/+18
2021-07-13Fix messages from multiple senders in historical chunk (MSC2716) (#10276)Eric Eastwood3-11/+76
Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint. Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
2021-07-13Add type hints to get_domain_from_id and get_localpart_from_id. (#10385)Patrick Cloke3-26/+75
2021-07-13Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke6-38/+237
Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
2021-07-13Move upgrade blurb v1.38.0Richard van der Hoff1-2/+2
2021-07-13Update CHANGES.mdRichard van der Hoff1-1/+1
2021-07-13fix debian changelogRichard van der Hoff1-1/+7
2021-07-13 1.38.0Richard van der Hoff3-2/+8
2021-07-131.38.0rc3 v1.38.0rc3Richard van der Hoff5-5/+16
2021-07-13Fix some links in `docs` and `contrib` (#10370)Dirk Klimpel9-17/+23
2021-07-13Bugfix `make_room_admin` fails for users that have left a private room (#10367)Dirk Klimpel2-1/+9
Fixes: #10338
2021-07-13Use inline type hints in `tests/` (#10350)Jonathan de Jong18-63/+62
This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
2021-07-13Github Actions workflow to attach release artifacts to release (#10379)Richard van der Hoff2-1/+28
2021-07-13[pyupgrade] `tests/` (#10347)Jonathan de Jong20-58/+45
2021-07-13Fix federation inbound age metric. (#10355)Erik Johnston2-1/+4
We should be reporting the age rather than absolute timestamp.
2021-07-13Build the python release artifacts in GHA tooRichard van der Hoff1-3/+23
2021-07-12build debs in GHA (#10247)Richard van der Hoff3-4/+58
GHA workflow to build the debs
2021-07-12Replace `room_depth.min_depth` with a BIGINT (#10289)Richard van der Hoff4-7/+186
while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth with a BIGINT.
2021-07-12Additional unit tests for spaces summary. (#10305)Patrick Cloke2-1/+204
2021-07-09Unblacklist fixed tests (#10357)Erik Johnston3-5/+2
2021-07-09Re-enable room v6 sytest (#10345)Richard van der Hoff2-1/+1
... now that it has been fixed in https://github.com/matrix-org/sytest/pull/1061.
2021-07-09Revert "Newsfile"Erik Johnston1-1/+0
This reverts commit 944428d1163d1521ef96db88040852520ad6cbff. </