summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Implement MSC3231: Token authenticated registration (#10142)Callum Brown2021-08-213-0/+382
| | | | | Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
* Do not include rooms with an unknown room version in a sync response. (#10644)Patrick Cloke2021-08-192-3/+6
| | | | A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.
* Remove not needed database updates in modify user admin API (#10627)Dirk Klimpel2021-08-191-7/+18
|
* Merge branch 'release-v1.41' into developErik Johnston2021-08-183-204/+68
|\
| * Allow /createRoom to be run on workers (#10564)Andrew Morgan2021-08-171-34/+34
| | | | | | Fixes https://github.com/matrix-org/synapse/issues/7867
| * Remove the unused public_room_list_stream (#10565)Andrew Morgan2021-08-173-181/+45
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Fix weakref_slot parameter for room member storage attrs. (#10642)Patrick Cloke2021-08-181-4/+4
| | | | | | Follow-up to #10629 which set it to true, not false.
* | Convert room member storage tuples to attrs. (#10629)Patrick Cloke2021-08-183-17/+36
|/ | | | Instead of using namedtuples. This helps with asserting type hints and code completion.
* Allow to edit `external_ids` by Edit User admin API (#10598)Dirk Klimpel2021-08-171-0/+22
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* update links to schema doc (#10620)Richard van der Hoff2021-08-172-3/+3
|
* Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-045-28/+150
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* Improve event caching code (#10119)Erik Johnston2021-08-042-43/+107
| | | | Ensure we only load an event from the DB once when the same event is requested multiple times at once.
* Fix `could not serialize access` errors for `claim_e2e_one_time_keys` (#10504)Erik Johnston2021-08-041-62/+126
|
* Add `get_userinfo_by_id` method to `ModuleApi` (#9581)Jason Robinson2021-08-041-1/+29
| | | | | | Makes it easier to fetch user details in for example spam checker modules, without needing to use api._store or figure out database interactions. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Prune inbound federation queues if they get too long (#10390)Erik Johnston2021-08-021-2/+102
|
* Allow setting transaction limit for db connections (#10440)Toni Spets2021-08-021-0/+21
| | | | | | | Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
* Generics for `ObservableDeferred` (#10491)Richard van der Hoff2021-07-281-1/+3
| | | | | Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
* Make historical events discoverable from backfill for servers without any ↵Eric Eastwood2021-07-284-20/+258
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489 We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509 * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
* Merge tag 'v1.39.0rc3' into developErik Johnston2021-07-281-1/+8
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
| * Always communicate device OTK counts to clients (#10485)Andrew Morgan2021-07-271-1/+8
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Fix `oldest_pdu_in_federation_staging` (#10455)Erik Johnston2021-07-271-2/+5
| | | | | | | | If the staging area was empty we'd report an age of 51 years, which is not true or helpful.
* | Add type hints to state handler. (#10482)Patrick Cloke2021-07-262-8/+13
| |
* | Add `creation_ts` to list users admin API (#10448)Dirk Klimpel2021-07-222-12/+9
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442)Erik Johnston2021-07-225-99/+43
| | | | | | | | | | | | | | | | Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.
* | Add type hints to additional servlet functions (#10437)Patrick Cloke2021-07-211-1/+1
| | | | | | | | | | | | | | | | | | Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.
* | Add a return type to parse_string. (#10438)Patrick Cloke2021-07-213-3/+3
|/ | | | And set the required attribute in a few places which will error if a parameter is not provided.
* Fix dropping locks on shut down (#10433)Erik Johnston2021-07-201-1/+5
|
* Add a new version of the R30 phone-home metric, which removes a false ↵reivilibre2021-07-191-0/+129
| | | | | impression of retention given by the old R30 metric (#10332) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-195-7/+9
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Remove unused `events_by_room` (#10421)Eric Eastwood2021-07-191-4/+0
| | | | | It looks like it was first used and introduced in https://github.com/matrix-org/synapse/commit/5130d80d79fe1f95ce03b8f1cfd4fbf0a32f5ac8#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305 but the But the usage was removed in https://github.com/matrix-org/synapse/commit/4c6a31cd6efa25be4c9f1b357e8f92065fac63eb#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-1520-96/+86
|
* Reduce likelihood of Postgres table scanning `state_groups_state`. (#10359)Erik Johnston2021-07-151-0/+34
| | | | | | | | The postgres statistics collector sometimes massively underestimates the number of distinct state groups are in the `state_groups_state`, which can cause postgres to use table scans for queries for multiple state groups. We fix this by manually setting `n_distinct` on the column.
* Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-131-2/+11
| | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* Fix federation inbound age metric. (#10355)Erik Johnston2021-07-131-1/+3
| | | We should be reporting the age rather than absolute timestamp.
* Replace `room_depth.min_depth` with a BIGINT (#10289)Richard van der Hoff2021-07-123-7/+185
| | | | | while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth with a BIGINT.
* Fix the user directory becoming broken (and noisy errors being logged) when ↵reivilibre2021-07-091-2/+6
| | | | | knocking and room statistics are in use. (#10344) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Switch `application_services_txns.txn_id` to BIGINT (#10349)Richard van der Hoff2021-07-091-0/+23
|
* Merge tag 'v1.38.0rc2' into developErik Johnston2021-07-091-2/+13
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
| * Ensure we always drop the federation inbound lock (#10336)Erik Johnston2021-07-091-2/+13
| |
* | Upsert redactions in case they already exists (#10343)Andreas Rammhold2021-07-092-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Upsert redactions in case they already exists Occasionally, in combination with retention, redactions aren't deleted from the database whenever they are due for deletion. The server will eventually try to backfill the deleted events and trip over the already existing redaction events. Switching to an UPSERT for those events allows us to recover from there situations. The retention code still needs fixing but that is outside of my current comfort zone on this code base. This is related to #8707 where the error was discussed already. Signed-off-by: Andreas Rammhold <andreas@rammhold.de> * Also purge redactions when purging events Previously redacints where left behind leading to backfilling issues when the server stumbled across the already existing yet to be backfilled redactions. This issues has been discussed in #8707. Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
* | Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-083-292/+6
| | | | | | Fixes #9602
* | Minor changes to `user_daily_visits` (#10324)reivilibre2021-07-081-2/+2
|/ | | | | | | * Use fake time in tests in _get_start_of_day. * Change the inequality of last_seen in user_daily_visits Co-authored-by: Erik Johnston <erik@matrix.org>
* ANALYZE new stream ordering column (#10326)Brendan Abolivier2021-07-071-0/+10
| | | Fixes #10325
* Fix deactivate a user if he does not have a profile (#10252)Dirk Klimpel2021-07-061-4/+4
|
* Add script for getting info about recently registered users (#10290)Erik Johnston2021-07-061-1/+1
|
* Handle old staged inbound events (#10303)Erik Johnston2021-07-061-0/+9
| | | | | | | We might have events in the staging area if the service was restarted while there were unhandled events in the staging area. Fixes #10295
* fix ordering of bg update (#10291)Richard van der Hoff2021-07-011-1/+1
| | | | | this was a typo introduced in #10282. We don't want to end up doing the `replace_stream_ordering_column` update after anything that comes up in migration 60/03.
* Add some metrics to staging area (#10284)Erik Johnston2021-07-011-0/+39
|
* Change more stream_ordering columns to BIGINT (#10286)Richard van der Hoff2021-06-301-0/+30
|
* Rebuild other indexes using `stream_ordering` (#10282)Richard van der Hoff2021-06-302-6/+55
| | | | We need to rebuild *all* of the indexes that use the current `stream_ordering` column.
* Fix the inbound PDU metric (#10279)Erik Johnston2021-06-304-10/+72
| | | This broke in #10272
* Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-295-3/+511
|\
| * Handle inbound events from federation asynchronously (#10272)Erik Johnston2021-06-292-3/+138
| | | | | | | | | | | | | | | | | | | | | | Fixes #9490 This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous. Two drawbacks: Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately. We now only process one event per room at a time. This can be fixed up further down the line.
| * Add a distributed lock (#10269)Erik Johnston2021-06-293-0/+373
| | | | | | This adds a simple best effort locking mechanism that works cross workers.
* | Fix `populate_stream_ordering2` background job (#10267)Richard van der Hoff2021-06-291-16/+12
| | | | | | | | It was possible for us not to find any rows in a batch, and hence conclude that we had finished. Let's not do that.
* | Migrate stream_ordering to a bigint (#10264)Richard van der Hoff2021-06-293-16/+162
| | | | | | | | | | | | | | | | | | | | * Move background update names out to a separate class `EventsBackgroundUpdatesStore` gets inherited and we don't really want to further pollute the namespace. * Migrate stream_ordering to a bigint * changelog
* | MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-242-4/+237
|/ | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* Fix schema delta to not take as long on large servers (#10227)Erik Johnston2021-06-221-2/+5
| | | Introduced in #6739
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-5/+45
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Fix performance of responding to user key requests over federation (#10221)Erik Johnston2021-06-211-1/+8
| | | | | We were repeatedly looking up a config option in a loop (using the unclassed config style), which is expensive enough that it can cause large CPU usage.
* Fix incorrect time magnitude on delayed call (#10195)Andrew Morgan2021-06-171-1/+1
| | | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/10030. We were expecting milliseconds where we should have provided a value in seconds. The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup. This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.
* Fix persist_events to stop leaking opentracing contexts (#10193)Richard van der Hoff2021-06-171-1/+1
|
* Make opentracing trace into event persistence (#10134)Richard van der Hoff2021-06-161-5/+41
| | | | | | | | | | | | | | * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans
* Refactor `EventPersistenceQueue` (#10145)Richard van der Hoff2021-06-141-77/+88
| | | some cleanup, pulled out of #10134.
* Support for database schema version ranges (#9933)Richard van der Hoff2021-06-114-76/+108
| | | This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
* Fix bug when running presence off master (#10149)Erik Johnston2021-06-112-1/+16
| | | Hopefully fixes #10027.
* Integrate knock rooms with the public rooms directory (#9359)Andrew Morgan2021-06-091-5/+9
| | | | | | | | | | This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403. Specifically, it: * Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory * Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72
* Implement knock feature (#6739)Sorunome2021-06-092-0/+18
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Fix logging context when opening new DB connection (#10141)Erik Johnston2021-06-081-3/+9
| | | | Fixes #10140
* More database opentracing (#10136)Richard van der Hoff2021-06-071-2/+5
| | | Add a couple of extra logs/spans, to give a bit of a better idea.
* Add OpenTracing for database activity. (#10113)Richard van der Hoff2021-06-031-33/+53
| | | | | | | | | | This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
* Add new admin APIs to remove media by media ID from quarantine. (#10044)Dirk Klimpel2021-06-021-10/+20
| | | | | Related to: #6681, #5956, #10040 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-013-14/+74
| | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* Make reason and score optional for report_event (#10077)Callum Brown2021-05-271-1/+1
| | | | | | Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414 See #8551 Signed-off-by: Callum Brown <callum@calcuode.com>
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-275-13/+11
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Add an admin API for unprotecting local media from quarantine (#10040)Dirk Klimpel2021-05-261-3/+4
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-241-1/+1
|
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-243-3/+2
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Fix `get_state_ids_for_event` return type typo to match what the function ↵Eric Eastwood2021-05-241-1/+1
| | | | | actually does (#10050) It looks like a typo copy/paste from `get_state_for_event` above.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-212-31/+39
|
* Fixed removal of new presence stream states (#10014)Marek Matys2021-05-211-9/+9
| | | | | | | | | Fixes: https://github.com/matrix-org/synapse/issues/9962 This is a fix for above problem. I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts. Signed-off-by: Marek Matys <themarcq@gmail.com>
* Use a database table to hold the users that should have full presence sent ↵Andrew Morgan2021-05-182-1/+91
| | | | to them, instead of something in-memory (#9823)
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-143-11/+8
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Remove unnecessary SystemRandom from SQLBaseStore (#9987)Dan Callahan2021-05-142-3/+2
| | | | | | | | It's not obvious that instances of SQLBaseStore each need their own instances of random.SystemRandom(); let's just use random directly. Introduced by 52839886d664576831462e033b88e5aba4c019e3 Signed-off-by: Dan Callahan <danc@element.io>
* Add config option to hide device names over federation (#9945)Aaron Raimist2021-05-111-1/+3
| | | | | Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers. Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add debug logging for issue #9533 (#9959)Richard van der Hoff2021-05-111-0/+18
| | | | | Hopefully this will help us track down where to-device messages are getting lost/delayed.
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-07282-44/+79
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Use get_current_users_in_room from store and not StateHandler (#9910)Erik Johnston2021-05-053-5/+8
|
* Use the parent's logging context name for runWithConnection. (#9895)Patrick Cloke2021-04-281-1/+3
| | | | | This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".
* Revert "Experimental Federation Speedup (#9702)"Andrew Morgan2021-04-281-12/+16
| | | | This reverts commit 05e8c70c059f8ebb066e029bc3aa3e0cefef1019.
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-272-10/+7
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Split presence out of master (#9820)Erik Johnston2021-04-234-48/+129
|
* Only store data in caches, not "smart" objects (#9845)Erik Johnston2021-04-231-72/+89
|
* Clear the resync bit after resyncing device lists (#9867)Richard van der Hoff2021-04-221-10/+9
| | | Fixes #9866.
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-2210-19/+34
| | | This is no longer required, since we have dropped support for Python 3.5.
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-5/+5
|
* Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan2021-04-192-16/+64
| | | | | hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
* User directory: use calculated room membership state instead (#9821)Andrew Morgan2021-04-161-0/+27
| | | | | Fixes: #9797. Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
* Small speed up joining large remote rooms (#9825)Erik Johnston2021-04-161-21/+33
| | | | | There are a couple of points in `persist_events` where we are doing a query per event in series, which we can replace.
* Experimental Federation Speedup (#9702)Jonathan de Jong2021-04-141-16/+12
| | | | | This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max). Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1471-71/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-131-4/+4
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Add an admin API to manage ratelimit for a specific user (#9648)Dirk Klimpel2021-04-131-5/+59
|
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-088-26/+58
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Merge remote-tracking branch 'origin/develop' into rav/drop_py35Richard van der Hoff2021-04-082-3/+40
|\
| * Remove outdated constraint on remote_media_cache_thumbnails (#9725)Richard van der Hoff2021-04-062-3/+40
| | | | | | | | | | | | | | | | | | | | | | The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key` constraint is superceded by `remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds `thumbnail_method` to the unique key). PR #7124 made an attempt to remove the old constraint, but got the name wrong, so it didn't work. Here we update the bg update and rerun it. Fixes #8649.
* | remove unused param on `make_tuple_comparison_clause`Richard van der Hoff2021-04-084-7/+2
| |
* | Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-084-80/+12
| |
* | Require py36 and Postgres 9.6Richard van der Hoff2021-04-081-2/+2
|/
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-2/+2
|
* Add `order_by` to list user admin API (#9691)Dirk Klimpel2021-04-012-7/+44
|
* Deprecate imp (#9718)Cristina2021-03-311-3/+8
| | | | | Fixes #9642. Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
* Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff2021-03-301-2/+2
| | | | | | | `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-4/+5
|
* Add a storage method for returning all current presence from all users (#9650)Andrew Morgan2021-03-252-3/+68
| | | | | | | Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133 Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-241-1/+1
|
* Add type hints to misc. files. (#9676)Patrick Cloke2021-03-241-2/+2
|
* Add a type hints for service notices to the HomeServer object. (#9675)Patrick Cloke2021-03-242-5/+5
|
* Fix federation stall on concurrent access errors (#9639)Jonathan de Jong2021-03-231-36/+9
|
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-237-7/+7
|
* Consistently check whether a password may be set for a user. (#9636)Dirk Klimpel2021-03-181-0/+1
|
* Fix bad naming of storage function (#9637)Erik Johnston2021-03-171-1/+1
| | | | | | We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.
* Prep work for removing `outlier` from `internal_metadata` (#9411)Richard van der Hoff2021-03-172-3/+21
| | | | | | | | | | | | * Populate `internal_metadata.outlier` based on `events` table Rather than relying on `outlier` being in the `internal_metadata` column, populate it based on the `events.outlier` column. * Move `outlier` out of InternalMetadata._dict Ultimately, this will allow us to stop writing it to the database. For now, we have to grandfather it back in so as to maintain compatibility with older versions of Synapse.
* Don't go into federation catch up mode so easily (#9561)Erik Johnston2021-03-151-5/+5
| | | | | | | | | | Federation catch up mode is very inefficient if the number of events that the remote server has missed is small, since handling gaps can be very expensive, c.f. #9492. Instead of going into catch up mode whenever we see an error, we instead do so only if we've backed off from trying the remote for more than an hour (the assumption being that in such a case it is more than a transient failure).
* Optimise missing prev_event handling (#9601)Richard van der Hoff2021-03-151-7/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | Background: When we receive incoming federation traffic, and notice that we are missing prev_events from the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events, we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote server for the state at those prev_events, and then we may need to then ask the remote server for any events in that state which we don't already have, as well as the auth events for those missing state events, so that we can auth them. This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the observation that we are currently loading all of those auth events into memory at the start of the operation, but we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're actually going to need, but it doesn't.) The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about 60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache churn. (NB I've already tripled the size of the cache from its default of 10K). Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different (ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting that we clean it up.
* Convert Requester to attrs (#9586)Richard van der Hoff2021-03-101-3/+3
| | | | | | ... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+145
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* Fix a bug in the background task for purging chain cover. (#9583)Patrick Cloke2021-03-101-1/+1
|
* Add a background task to purge unused chain IDs. (#9542)Patrick Cloke2021-03-093-6/+98
| | | | | This is a companion change to apply the fix in #9498 / 922788c6043138165c025c78effeda87de842bab to previously purged rooms.
* Fix 'rejected_events_metadata' background update (#9537)Erik Johnston2021-03-031-1/+3
| | | | Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
* Purge chain cover tables when purging events. (#9498)Patrick Cloke2021-03-032-10/+37
|
* Add type hints to user admin API. (#9521)Dirk Klimpel2021-03-032-6/+6
|
* Bump the mypy and mypy-zope versions. (#9529)Patrick Cloke2021-03-031-1/+1
|
* Make deleting stale pushers a background update (#9536)Erik Johnston2021-03-032-1/+54
|
* Re-run rejected metadata background update. (#9503)Erik Johnston2021-03-021-0/+9
| | | | | | | It landed in schema version 58 after 59 had been created, causing some servers to not run it. The main effect of was that not all rooms had their chain cover calculated correctly. After the BG updates complete the chain covers will get fixed when a new state event in the affected rooms is received.
* Fix SQL delta file taking a long time to run (#9516)Erik Johnston2021-03-022-5/+57
| | | | Fixes #9504
* Ensure pushers are deleted for deactivated accounts (#9285)Erik Johnston2021-02-252-0/+64
|
* Add SQL delta for deleting stale pushers (#9479)Erik Johnston2021-02-241-0/+19
|
* Refactor to ensure we call check_consistency (#9470)Erik Johnston2021-02-247-28/+71
| | | The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
* Include newly added sequences in the port DB script. (#9449)Patrick Cloke2021-02-232-2/+13
| | | And ensure the consistency of `event_auth_chain_id`.
* Add an `order_by` field to list users' media admin API. (#8978)Dirk Klimpel2021-02-221-3/+38
|
* Remove cache for get_shared_rooms_for_users (#9416)Andrew Morgan2021-02-221-3/+1
| | | | | | | This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed. The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately. Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
* Regenerate exact thumbnails if missing (#9438)Erik Johnston2021-02-191-9/+9
|\
| * Regenerate exact thumbnails if missingErik Johnston2021-02-181-9/+9
| |
* | Add a config option to prioritise local users in user directory search ↵Andrew Morgan2021-02-191-9/+50
|/ | | | | | | | | results (#9383) This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default. Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical. This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
* Add back the guard against the user directory stream position not existing. ↵Patrick Cloke2021-02-181-1/+7
| | | | | | (#9428) As the comment says, this guard was there for when the initial user directory update has yet to happen.
* Add the shadow-banning status to the display user admin API. (#9400)Dirk Klimpel2021-02-172-3/+6
|
* Add type hints to groups code. (#9393)Patrick Cloke2021-02-171-2/+7
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1643-215/+337
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix some typos.Patrick Cloke2021-02-1212-20/+20
|
* Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) ↵Eric Eastwood2021-02-101-10/+0
| | | | | | | (#9003) Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests. Fix #8996
* Update type hints for Cursor to match PEP 249. (#9299)Jonathan de Jong2021-02-054-17/+46
|
* Add experimental support for PyPy. (#9123)Jonathan de Jong2021-02-042-6/+8
| | | | * Adds proper dependencies. * Minor fixes in database layer.
* Merge branch 'social_login' into developRichard van der Hoff2021-02-011-20/+20
|\
| * Fix SSO on workers (#9271)Richard van der Hoff2021-02-011-20/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource
| * Merge tag 'v1.26.0rc2' into social_loginRichard van der Hoff2021-01-255-7/+79
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
* | | Add phone home stats for encrypted messages. (#9283)Jan Christian Grünhage2021-02-011-0/+56
| | | | | | | | | Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
* | | Add type hints to E2E handler. (#9232)Patrick Cloke2021-01-281-2/+2
| | | | | | | | | This finishes adding type hints to the `synapse.handlers` module.
* | | Add type hints to various handlers. (#9223)Patrick Cloke2021-01-263-11/+16
| | | | | | | | | | | | With this change all handlers except the e2e_* ones have type hints enabled.
* | | Fix get forward extremities queryJason Robinson2021-01-261-1/+1
| | | | | | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | | Merge branch 'develop' into jaywink/admin-forward-extremitiesJason Robinson2021-01-262-1/+32
|\ \ \ | | | | | | | | | | | | | | | | # Conflicts: # synapse/rest/admin/__init__.py
| * | | Don't error if deleting a non-existent pusher. (#9121)Patrick Cloke2021-01-251-1/+3
| | | |
| * | | Add an admin API for shadow-banning users. (#9209)Patrick Cloke2021-01-251-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This expands the current shadow-banning feature to be usable via the admin API and adds documentation for it. A shadow-banned users receives successful responses to their client-server API requests, but the events are not propagated into rooms. Shadow-banning a user should be used as a tool of last resort and may lead to confusing or broken behaviour for the client.
* | | | Make natural join inner joinJason Robinson2021-01-261-2/+2
| | | | | | | | | | | | Co-authored-by: Erik Johnston <erik@matrix.org>
* | | | Merge branch 'develop' into jaywink/admin-forward-extremitiesJason Robinson2021-01-2339-410/+2048
|\| | |
| * | | Speed up chain cover calculation (#9176)Erik Johnston2021-01-212-55/+160
| | | |
| * | | Increase perf of handling concurrent use of StreamIDGenerators. (#9190)Erik Johnston2021-01-211-8/+13
| | | | | | | | | | | | | | | | | | | | We have seen a failure mode here where if there are many in flight unfinished IDs then marking an ID as finished takes a lot of CPU (as calling deque.remove iterates over the list)
| * | | Merge remote-tracking branch 'origin/release-v1.26.0' into developErik Johnston2021-01-212-1/+19
| |\ \ \ | | | |/ | | |/|
| | * | Add schema update to fix existing DBs affected by #9193 (#9195)Erik Johnston2021-01-212-1/+19
| | | |
| * | | Merge remote-tracking branch 'origin/release-v1.26.0' into developErik Johnston2021-01-214-7/+61
| |\| |
| | * | Fix receipts or account data not being sent down sync (#9193)Erik Johnston2021-01-214-7/+61
| | | | | | | | | | | | | | | | | | | | Introduced in #9104 This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.
| * | | Use execute_batch in more places (#9188)Erik Johnston2021-01-2111-31/+25
| | |/ | |/| | | | | | | | | | * Use execute_batch in more places * Newsfile
| * | Use execute_batch instead of executemany in places (#9181)Erik Johnston2021-01-212-12/+11
| |/ | | | | `execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
| * Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-4/+2
| |
| * Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-1810-160/+315
| |
| * Make chain cover index bg update go faster (#9124)Erik Johnston2021-01-151-150/+179
| | | | | | | | | | | | We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.
| * Ensure we store pusher data as text (#9117)Richard van der Hoff2021-01-151-3/+2
| | | | | | | | | | I don't think there's any need to use canonicaljson here. Fixes: #4475.
| * Add type hints to media rest resources. (#9093)Patrick Cloke2021-01-151-1/+2
| |
| * Fix perf of get_cross_signing_keys (#9116)Erik Johnston2021-01-141-13/+28
| |
| * Fix chain cover background update to work with split out event persisters ↵Erik Johnston2021-01-144-29/+37
| | | | | | | | (#9115)
| * Fix get destinations to catch up query. (#9114)Erik Johnston2021-01-141-13/+11
| | | | | | | | t was doing a sequential scan on `destination_rooms`, which took minutes.
| * Add background update for add chain cover index (#9029)Erik Johnston2021-01-143-16/+243
| |
| * Register the /devices endpoint on workers. (#9092)Patrick Cloke2021-01-131-11/+30
| |
| * Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-121-1/+1
| | | | | | This only applies if the user's data is to be erased.
| * Move more encryption endpoints off master (#9068)Erik Johnston2021-01-111-44/+44
| |
| * Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-116-22/+839
| |
| * Remove old tables after schema version bump (#9055)Erik Johnston2021-01-115-59/+36
| | | | | | | | | | These tables are unused, and can be dropped now the schema version has been bumped.
| * Ensure rejected events get added to some metadata tables (#9016)Erik Johnston2021-01-113-24/+166
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
| * Drop last_used column from access_tokens (#9025)Jerin J Titus2021-01-112-0/+78
| | | | | | | | | | * Dropped last_used column from access_tokens Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
* | Add depth and received_ts to forward_extremities admin API responseJason Robinson2021-01-231-1/+2
| | | | | | | | | | | | Also add a warning on the admin API documentation. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Remove trailing whitespace to appease the linterJason Robinson2021-01-111-2/+2
| | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Simplify delete_forward_extremities_for_room_txn SQLJason Robinson2021-01-111-10/+4
| | | | | | | | | | | | As per feedback. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Address pr feedbackJason Robinson2021-01-112-26/+40
| | | | | | | | | | | | | | | | | | | | * docs updates * prettify SQL * add missing copyright * cursor_to_dict * update touched files copyright years Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Don't try to use f-stringsJason Robinson2021-01-111-1/+1
| | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Merge branch 'develop' into jaywink/admin-forward-extremitiesJason Robinson2021-01-091-31/+23
|\|
| * Fix error handling during insertion of client IPs (#9051)Erik Johnston2021-01-081-31/+23
| | | | | | | | | | You can't continue using a transaction once an exception has been raised, so catching and dropping the error here is pointless and just causes more errors.
* | Implement clearing cache after deleting forward extremitiesJason Robinson2021-01-091-8/+33
| | | | | | | | | | | | Also run linter. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Add Rooms admin forward extremities DELETE endpointJason Robinson2021-01-081-1/+48
| | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Add forward extremities endpoint to rooms admin APIJason Robinson2021-01-072-0/+22
|/ | | | | | GET /_synapse/admin/v1/rooms/<identifier>/forward_extremities now gets forward extremities for a room, returning count and the list of extremities. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-074-65/+158
|
* Some cleanups to device inbox store. (#9041)Erik Johnston2021-01-071-49/+58
|
* Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-073-30/+175
| | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* Handle updating schema version without any deltas. (#9033)Erik Johnston2021-01-071-7/+10
| | | | This can happen when using a split out state database and we've upgraded the schema version without there being any changes in the state schema.
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-3/+5
|
* Add type hints to the crypto module. (#8999)Patrick Cloke2021-01-041-5/+5
|
* Add additional type hints to the storage module. (#8980)Patrick Cloke2020-12-308-144/+211
|
* Add type hints to admin and room list handlers. (#8973)Patrick Cloke2020-12-291-2/+5
|
* Drop the unused local_invites table. (#8979)Patrick Cloke2020-12-291-0/+18
| | | This table has been unused since Synapse v1.17.0.
* Allow re-using a UI auth validation for a period of time (#8970)Patrick Cloke2020-12-182-0/+56
|
* Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-16/+184
| | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* Use the simple dictionary in full text search for the user directory (#8959)Brendan Abolivier2020-12-171-12/+12
| | | | | * Use the simple dictionary in fts for the user directory * Clarify naming
* Make search statement in List Room and User Admin API case-insensitive (#8931)Dirk Klimpel2020-12-172-5/+6
|
* Convert internal pusher dicts to attrs classes. (#8940)Patrick Cloke2020-12-163-41/+59
| | | This improves type hinting and should use less memory.
* Do not assume that the contents dictionary includes history_visibility. (#8945)Patrick Cloke2020-12-161-2/+5
|
* Fix handling of stream tokens for push. (#8943)Patrick Cloke2020-12-151-10/+0
| | | | Removes faulty assertions and fixes the logic to ensure the max stream token is always set.
* Add number of local devices to Room Details Admin API (#8886)Dirk Klimpel2020-12-111-0/+32
|
* Pass room_id to get_auth_chain_difference (#8879)Erik Johnston2020-12-041-1/+3
| | | | | This is so that we can choose which algorithm to use based on the room ID.
* UIA: offer only available auth flowsRichard van der Hoff2020-12-022-0/+42
| | | | | | | During user-interactive auth, do not offer password auth to users with no password, nor SSO auth to users with no SSO. Fixes #7559.
* Add missing `ordering` to background updates (#8850)Richard van der Hoff2020-12-014-12/+12
| | | | | | | It's important that we make sure our background updates happen in a defined order, to avoid disasters like #6923. Add an ordering to all of the background updates that have landed since #7190.
* Drop (almost) unused index on event_json (#8845)Richard van der Hoff2020-11-302-1/+20
|
* Speed up remote invite rejection database call (#8815)Andrew Morgan2020-11-251-1/+33
| | | | | | | | | | | | | This is another PR that grew out of #6739. The existing code for checking whether a user is currently invited to a room when they want to leave the room looks like the following: https://github.com/matrix-org/synapse/blob/f737368a26bb9eea401fcc3a5bdd7e0b59e91f09/synapse/handlers/room_member.py#L518-L540 It calls `get_invite_for_local_user_in_room`, which will actually query *all* rooms the user has been invited to, before iterating over them and matching via the room ID. It will then return a tuple of a lot of information which we pull the event ID out of. I need to do a similar check for knocking, but this code wasn't very efficient. I then tried to write a different implementation using `StateHandler.get_current_state` but this actually didn't work as we haven't *joined* the room yet - we've only been invited to it. That means that only certain tables in Synapse have our desired `invite` membership state. One of those tables is `local_current_membership`. So I wrote a store method that just queries that table instead
* Improve appservice handler to send only the most recent read receipts when ↵Will Hunt2020-11-181-1/+6
| | | | | | | | | | | | | | | | | | | no stream_id is stored. (#8744) * Make this line debug (it's noisy) * Don't include from_key for presence if we are at 0 * Limit read receipts for all rooms to 100 * changelog.d/8744.bugfix * Allow from_key to be None * Update 8744.bugfix * The from_key is superflous * Update comment
* Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-0/+2
|
* Generalise _maybe_store_room_on_invite (#8754)Andrew Morgan2020-11-131-4/+6
| | | | | | | | | There's a handy function called maybe_store_room_on_invite which allows us to create an entry in the rooms table for a room and its version for which we aren't joined to yet, but we can reference when ingesting events about. This is currently used for invites where we receive some stripped state about the room and pass it down via /sync to the client, without us being in the room yet. There is a similar requirement for knocking, where we will eventually do the same thing, and need an entry in the rooms table as well. Thus, reusing this function works, however its name needs to be generalised a bit. Separated out from #6739.
* Cache event ID to auth event IDs lookups (#8752)Erik Johnston2020-11-131-12/+70
| | | | | This should hopefully speed up `get_auth_chain_difference` a bit in the case of repeated state res on the same rooms. `get_auth_chain_difference` does a breadth first walk of the auth graphs by repeatedly looking up events' auth events. Different state resolutions on the same room will end up doing a lot of the same event to auth events lookups, so by caching them we should speed things up in cases of repeated state resolutions on the same room.
* Enable reconnection in DB pool (#8726)Erik Johnston2020-11-121-1/+6
| | | | | | `adbapi.ConnectionPool` let's you turn on auto reconnect of DB connections. This is off by default. As far as I can tell if its not enabled dead connections never get removed from the pool. Maybe helps #8574
* Add an admin API for users' media statistics (#8700)Dirk Klimpel2020-11-051-0/+127
| | | | | | | | Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users. Related to #6094 It is the first API for statistics. Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats) Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Improve start time by adding index to e2e_cross_signing_keys (#8694)Erik Johnston2020-11-022-2/+33
| | | We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
* Fix error handling around when completing an AS transaction (#8693)Brendan Abolivier2020-10-301-1/+1
|
* Fix race for concurrent downloads of remote media. (#8682)Erik Johnston2020-10-301-0/+27
| | | Fixes #6755
* Fix optional parameter in stripped state storage method (#8688)Andrew Morgan2020-10-301-1/+1
| | | Missed in #8671.
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-292-9/+56
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-282-9/+7
|\ | | | | Fix serialisation errors when using third-party event rules.
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-282-9/+7
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Add type hints to application services. (#8655)Patrick Cloke2020-10-281-38/+60
| |
* | Don't pull event from DB when handling replication traffic. (#8669)Erik Johnston2020-10-281-2/+6
| | | | | | | | | | I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-2/+2
| | | | | | | | | | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* | Abstract code for stripping room state into a separate method (#8671)Andrew Morgan2020-10-271-1/+53
| | | | | | | | | | | | | | | | | | | | | | This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change. For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included. The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event. This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem. In essence this is just moving some inline code to a reusable store method.
* | Add admin API to list users' local media (#8647)Dirk Klimpel2020-10-273-0/+60
| | | | | | Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
* | Add type hints for account validity handler (#8620)Patrick Cloke2020-10-262-4/+4
| | | | | | This also fixes a bug by fixing handling of an account which doesn't expire.
* | Split admin API for reported events into a detail and a list view (#8539)Dirk Klimpel2020-10-261-17/+87
| | | | | | | | | | | | | | | | | | | | | | | | Split admin API for reported events in detail und list view. API was introduced with #8217 in synapse v.1.21.0. It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview. The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`. It is similar to room and users API. It is a kind of regression in `GET /_synapse/admin/v1/event_reports`. `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec). Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Add an admin api to delete local media. (#8519)Dirk Klimpel2020-10-261-0/+53
| | | | | | | | | | | | Related to: #6459, #3479 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server.
* | Merge branch 'release-v1.22.0' into developErik Johnston2020-10-261-4/+5
|\|
| * Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt2020-10-261-7/+22
| |
| * Fix user_daily_visits to not have duplicate rows for UA. (#8654)Erik Johnston2020-10-261-4/+5
| | | | | | | | | | | | | | | | | | | | | | * Fix user_daily_visits to not have duplicate rows for UA. Fixes #8641. * Newsfile * Fix typo. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt2020-10-261-7/+22
| |
* | Type hints for RegistrationStore (#8615)Erik Johnston2020-10-222-74/+83
|/
* Add type hints to profile and base handlers. (#8609)Patrick Cloke2020-10-211-3/+3
|
* Consistently use wrap_as_background_task in more places (#8599)Patrick Cloke2020-10-202-14/+13
|
* Move schema file for as_device_stream (#8590)Will Hunt2020-10-201-3/+2
| | | | | | | | | * Move schema file * Add a . * Add matching changelog entry * Fix sqlite
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-12 20:38:03 +0000