summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Only send out device list updates for our own users (#12465)Erik Johnston2022-04-141-1/+3
| | | Broke in #12365
* Fix missing sync events during historical batch imports (#12319)Nick Mills-Barrett2022-04-131-0/+26
| | | | | | | | Discovered after much in-depth investigation in #12281. Closes: #12281 Closes: #3305 Signed off by: Nick Mills-Barrett nick@beeper.com
* Process device list updates asynchronously (#12365)Erik Johnston2022-04-122-55/+12
|
* Resync state after partial-state join (#12394)Richard van der Hoff2022-04-125-0/+174
| | | | | We work through all the events with partial state, updating the state at each of them. Once it's done, we recalculate the state for the whole room, and then mark the room as having complete state.
* Remove references to unstable identifiers from MSC3440. (#12382)Patrick Cloke2022-04-122-66/+17
| | | | | Removes references to unstable thread relation, unstable identifiers for filtering parameters, and the experimental config flag.
* Add some type hints to datastore (#12423)Dirk Klimpel2022-04-127-75/+120
| | | | | | | | | | | | | | | | | | | * Add some type hints to datastore * newsfile * change `Collection` to `List` * refactor return type of `select_users_txn` * correct type hint in `stream.py` * Remove `Optional` in `select_users_txn` * remove not needed return type in `__init__` * Revert change in `get_stream_id_for_event_txn` * Remove import from `Literal`
* Do not consider events by ignored users for bundled aggregations (#12235)Patrick Cloke2022-04-111-9/+143
| | | | | | | Consider the requester's ignored users when calculating the bundled aggregations. See #12285 / 4df10d32148ae29f792afc68ff774bcbd1915cea for corresponding changes for the `/relations` endpoint.
* Disallow untyped defs in synapse._scripts (#12422)David Robertson2022-04-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Of note: * No untyped defs in `register_new_matrix_user` This one might be contraversial. `request_registration` has three dependency-injection arguments used for testing. I'm removing the injection of the `requests` module and using `unitest.mock.patch` in the test cases instead. Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the change: ``` synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any" synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any" ``` And after: ``` synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType" synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response" ``` * Drive-by comment in `synapse.storage.types` * No untyped defs in `synapse_port_db` This was by far the most painful. I'm happy to break this up into smaller pieces for review if it's not managable as-is.
* Optimise `_update_client_ips_batch_txn` to batch together database ↵reivilibre2022-04-082-35/+132
| | | | | operations. (#12252) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix `synapse_event_persisted_position` metric (#12390)Richard van der Hoff2022-04-061-3/+3
| | | | Fixes a bug introduced in #11417 where we would only included backfilled events in `synapse_event_persisted_position`
* Update type annotations for compatiblity with prometheus_client 0.14 (#12389)Richard van der Hoff2022-04-061-3/+1
| | | | | | | Principally, `prometheus_client.REGISTRY.register` now requires its argument to extend `prometheus_client.Collector`. Additionally, `Gauge.set` is now annotated so that passing `Optional[int]` causes an error.
* Generate historic pagination token for `/messages` when no `?from` token ↵Eric Eastwood2022-04-061-9/+11
| | | | provided (#12370)
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-1/+1
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Prefill more stream change caches. (#12372)Erik Johnston2022-04-054-38/+89
|
* Prefill the device_list_stream_cache (#12367)Erik Johnston2022-04-041-1/+11
| | | | | | | * Prefill the device_list_stream_cache * Newsfile * Newsfile
* Track device list updates per room. (#12321)Erik Johnston2022-04-044-25/+232
| | | | | | | | | | | | | | This is a first step in dealing with #7721. The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things: 1. we can defer calculating the set of remote servers that need to be poked about the change; and 2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed. However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled. There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly). Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
* Use a sequence to generate AS transaction IDs, drop `last_txn` AS state (#12209)Nick Mills-Barrett2022-04-013-44/+67
| | | | | | | | Switching to a sequence means there's no need to track `last_txn` on the AS state table to generate new TXN IDs. This also means that there is no longer contention between the AS scheduler and AS handler on updates to the `application_services_state` table, which will prevent serialization errors during the complete AS txn transaction.
* Move `update_client_ip` background job from the main process to the ↵reivilibre2022-04-014-76/+117
| | | | background worker. (#12251)
* Raise an exception when getting state at an outlier (#12191)Richard van der Hoff2022-04-012-4/+32
| | | | | | It seems like calling `_get_state_group_for_events` for an event where the state is unknown is an error. Accordingly, let's raise an exception rather than silently returning an empty result.
* Optimise `_get_state_after_missing_prev_event`: use `/state` (#12040)Richard van der Hoff2022-04-011-5/+3
| | | If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
* Add more type hints to the main state store. (#12267)Patrick Cloke2022-03-311-7/+11
|
* Remove an unnecessary class from the relations code. (#12338)Patrick Cloke2022-03-312-62/+8
| | | | | The PaginationChunk class attempted to bundle some properties together, but really just caused callers to jump through hoops and hid implementation details.
* Remove the unused and unstable `/aggregations` endpoint. (#12293)Patrick Cloke2022-03-302-97/+14
| | | | | | | | | This endpoint was removed from MSC2675 before it was approved. It is currently unspecified (even in any MSCs) and therefore subject to removal. It is not implemented by any known clients. This also changes the bundled aggregation format for `m.annotation`, which previously included pagination tokens for the `/aggregations` endpoint, which are no longer useful.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-303-18/+67
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Start application service stream token tracking from 1 (#12193)Andrew Morgan2022-03-301-1/+2
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Add a configuration to exclude rooms from sync response (#12310)Brendan Abolivier2022-03-302-15/+36
|
* Update `LoggingTransaction.call_after` and `call_on_exception` docstrings ↵Sean Quah2022-03-291-3/+20
| | | | | | | | (#12315) Document the behaviour of `LoggingTransaction.call_after` and `LoggingTransaction.call_on_exception` when transactions are retried. Signed-off-by: Sean Quah <seanq@element.io>
* Bump `black` and `click` versions (#12320)David Robertson2022-03-291-1/+1
|
* Exclude outliers in `on_backfill_request` (#12314)Richard van der Hoff2022-03-281-3/+9
| | | | | | When we are processing a `/backfill` request from a remote server, exclude any outliers from consideration early on. We can't return outliers anyway (since we don't know the state at the outlier), and filtering them out earlier means that we won't attempt to calulate the state for them.
* Add some type hints to datastore. (#12255)Dirk Klimpel2022-03-288-39/+60
|
* Improve type annotations for `execute_values`. (#12311)reivilibre2022-03-281-10/+7
|
* Add cache for `get_membership_from_event_ids` (#12272)Erik Johnston2022-03-254-8/+55
| | | | | This should speed up push rule calculations for rooms with large numbers of local users when the main push rule cache fails. Co-authored-by: reivilibre <oliverw@matrix.org>
* Refuse to start if DB has an unsafe locale (#12262)Shay2022-03-231-15/+30
|
* Use psycopg2 type stubs (#12269)David Robertson2022-03-232-4/+12
|
* Rename shared_rooms to mutual_rooms (#12036)Jonathan de Jong2022-03-231-3/+3
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Move get_bundled_aggregations to relations handler. (#12237)Patrick Cloke2022-03-181-146/+5
| | | | | The get_bundled_aggregations code is fairly high-level and uses a lot of store methods, we move it into the handler as that seems like a better fit.
* Only fetch thread participation for events with threads. (#12228)Patrick Cloke2022-03-181-1/+3
| | | | | | | | | We fetch the thread summary in two phases: 1. The summary that is shared by all users (count of messages and latest event). 2. Whether the requesting user has participated in the thread. There's no use in attempting step 2 for events which did not return a summary from step 1.
* Add some type hints to datastore (#12248)Dirk Klimpel2022-03-182-78/+116
| | | | | * inherit `MonthlyActiveUsersStore` from `RegistrationWorkerStore` Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correct `check_username_for_spam` annotations and docs (#12246)David Robertson2022-03-181-4/+19
| | | | | | | * Formally type the UserProfile in user searches * export UserProfile in synapse.module_api * Update docs Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Handle cancellation in `DatabasePool.runInteraction()` (#12199)Sean Quah2022-03-161-24/+37
| | | | | | | | | | | | To handle cancellation, we ensure that `after_callback`s and `exception_callback`s are always run, since the transaction will complete on another thread regardless of cancellation. We also wait until everything is done before releasing the `CancelledError`, so that logging contexts won't get used after they have been finished. Signed-off-by: Sean Quah <seanq@element.io>
* Add some missing type hints to cache datastore. (#12216)Patrick Cloke2022-03-161-21/+36
|
* Use the ignored_users table to test event visibility & sync. (#12225)Patrick Cloke2022-03-151-3/+38
| | | | | Instead of fetching the raw account data and re-parsing it. The ignored_users table is a denormalised version of the account data for quick searching.
* Fix broken background updates when using sqlite with `enable_search` off ↵Sean Quah2022-03-141-6/+7
| | | | | (#12215) Signed-off-by: Sean Quah <seanq@element.io>
* Add config settings for background update parameters (#11980)Shay2022-03-111-14/+25
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-112-3/+0
|
* Support stable identifiers for MSC3440: Threading (#12151)Patrick Cloke2022-03-103-39/+61
| | | | The unstable identifiers are still supported if the experimental configuration flag is enabled. The unstable identifiers will be removed in a future release.
* Allow retrieving the relations of a redacted event. (#12130)Patrick Cloke2022-03-103-33/+42
| | | | | | | | | This is allowed per MSC2675, although the original implementation did not allow for it and would return an empty chunk / not bundle aggregations. The main thing to improve is that the various caches get cleared properly when an event is redacted, and that edits must not leak if the original event is redacted (as that would presumably leak something similar to the original event content).
* Allow for ignoring some arguments when caching. (#12189)Patrick Cloke2022-03-091-2/+2
| | | | | * `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key. * Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity. * Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
* Remove some unused variables/parameters. (#12187)Patrick Cloke2022-03-091-9/+5
|
* Fix a bug in background updates wherein background updates are never run ↵Shay2022-03-071-3/+5
| | | | using the default batch size (#12157)
* Invalidate caches when an event with a relation is redacted. (#12121)Patrick Cloke2022-03-072-5/+35
| | | | | The caches for the target of the relation must be cleared so that the bundled aggregations are re-calculated after the redaction is processed.
* Reduce to-device queries for /sync. (#12163)Erik Johnston2022-03-041-0/+3
|
* Remove backwards compatibility with RelationPaginationToken. (#12138)Patrick Cloke2022-03-041-31/+0
|
* Fix type of `events` in `StateGroupStorage` and `StateHandler` (#12156)Richard van der Hoff2022-03-041-4/+4
| | | We make multiple passes over this, so a regular iterable won't do.
* Back out in-flight state caching changes. (#12126)reivilibre2022-03-021-218/+25
|
* Make get_room_version use cached get_room_version_id. (#11808)lukasdenk2022-03-021-14/+13
|
* Order in-flight state group queries in biggest-first order (#11610)reivilibre2022-03-011-3/+27
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Faster joins: persist to database (#12012)Richard van der Hoff2022-03-015-0/+203
| | | | | | | | | | | | When we get a partial_state response from send_join, store information in the database about it: * store a record about the room as a whole having partial state, and stash the list of member servers too. * flag the join event itself as having partial state * also, for any new events whose prev-events are partial-stated, note that they will *also* be partial-stated. We don't yet make any attempt to interpret this data, so API calls (and a bunch of other things) are just going to get incorrect data.
* Ensure that `get_datastores().main` is typed (#12070)Sean Quah2022-02-251-1/+2
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Minor typing fixes for `synapse/storage/persist_events.py` (#12069)Sean Quah2022-02-252-23/+25
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Add support for MSC3202: sending one-time key counts and fallback key usage ↵reivilibre2022-02-242-2/+143
| | | | | states to Application Services. (#11617) Co-authored-by: Erik Johnston <erik@matrix.org>
* Fix non-strings in the `event_search` table (#12037)Sean Quah2022-02-243-9/+57
| | | | | | | Don't attempt to add non-string `value`s to `event_search` and add a background update to clear out bad rows from `event_search` when using sqlite. Signed-off-by: Sean Quah <seanq@element.io>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Cap the number of in-flight requests for state from a single group (#11608)reivilibre2022-02-221-0/+16
|
* Fix slow performance of `/logout` in some cases where refresh tokens are in ↵reivilibre2022-02-222-2/+44
| | | | use. The slowness existed since the initial implementation of refresh tokens. (#12056)
* remote join processing: get create event from state, not auth_chain (#12039)Richard van der Hoff2022-02-211-2/+2
| | | A follow-up to #12005, in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
* Add type hints to `synapse/storage/databases/main` (#11984)Dirk Klimpel2022-02-213-35/+61
|
* Document why auth providers aren't validated in the admin API. (#12004)Patrick Cloke2022-02-181-0/+21
| | | | Since it is reasonable to give a future or past auth provider, which might not be in the current configuration.
* Track and deduplicate in-flight requests to `_get_state_for_groups`. (#10870)reivilibre2022-02-181-25/+178
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix bug in `StateFilter.return_expanded()` and add some tests. (#12016)reivilibre2022-02-181-1/+7
|
* Fix 500 error with Postgres when looking backwards with the MSC3030 ↵Eric Eastwood2022-02-181-1/+1
| | | | `/timestamp_to_event` endpoint (#12024)
* Fix a typo in a comment.Patrick Cloke2022-02-161-1/+1
|
* Optimise calculating device_list changes in `/sync`. (#11974)Erik Johnston2022-02-152-0/+72
| | | | | | For users with large accounts it is inefficient to calculate the set of users they share a room with (and takes a lot of space in the cache). Instead we can look at users whose devices have changed since the last sync and check if they share a room with the syncing user.
* Fix incorrect `get_rooms_for_user` for remote user (#11999)Erik Johnston2022-02-151-11/+16
| | | | | | | When the server leaves a room the `get_rooms_for_user` cache is not correctly invalidated for the remote users in the room. This means that subsequent calls to `get_rooms_for_user` for the remote users would incorrectly include the room (it shouldn't be included because the server no longer knows anything about the room).
* Refactor search code to reduce function size. (#11991)Patrick Cloke2022-02-151-7/+10
| | | | | | | | | Splits the search code into a few logical functions instead of a single unreadable function. There are also a few additional changes for readability. After refactoring it was clear to see there were some unused and unnecessary variables, which were simplified.
* Fix incorrect thread summaries when the latest event is edited. (#11992)Patrick Cloke2022-02-152-7/+19
| | | | | If the latest event in a thread was edited than the original event content was included in bundled aggregation for threads instead of the edited event content.
* Implement MSC3706: partial state in `/send_join` response (#11967)Richard van der Hoff2022-02-121-6/+6
| | | | | | | | | | | | * Make `get_auth_chain_ids` return a Set It has a set internally, and a set is often useful where it gets used, so let's avoid converting to an intermediate list. * Minor refactors in `on_send_join_request` A little bit of non-functional groundwork * Implement MSC3706: partial state in /send_join response
* Fetch thread summaries for multiple events in a single query (#11752)Patrick Cloke2022-02-112-74/+150
| | | | | This should reduce database usage when fetching bundled aggregations as the number of individual queries (and round trips to the database) are reduced.
* Fix to-device being dropped in limited sync in SQLite. (#11966)Erik Johnston2022-02-111-1/+4
| | | | | | | | | If ther are more than 100 to-device messages pending for a device `/sync` will only return the first 100, however the next batch token was incorrectly calculated and so all other pending messages would be dropped. This is due to `txn.rowcount` only returning the number of rows that *changed*, rather than the number *selected* in SQLite.
* Support pagination tokens from /sync and /messages in the relations API. ↵Patrick Cloke2022-02-102-21/+40
| | | | (#11952)
* Experimental support to include bundled aggregations in search results ↵Patrick Cloke2022-02-081-2/+11
| | | | (MSC3666) (#11837)
* Fetch edits for multiple events in a single query. (#11660)Patrick Cloke2022-02-082-54/+100
| | | | | This should reduce database usage when fetching bundled aggregations as the number of individual queries (and round trips to the database) are reduced.
* Add a docstring to `add_device_change_to_streams` and fix some nearby types ↵Andrew Morgan2022-02-081-6/+16
| | | | (#11912)
* Fix historical messages backfilling in random order on remote homeservers ↵Eric Eastwood2022-02-072-103/+223
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#11114) Fix https://github.com/matrix-org/synapse/issues/11091 Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`) 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`) - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)). - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort. - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date. - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order. 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls. - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793 Before | After --- | --- ![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png) #### Why aren't we sorting topologically when receiving backfill events? > The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway. > > As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering. > > -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138 See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
* Invalidate the get_users_in_room{_with_profile} caches only when necessary. ↵Patrick Cloke2022-02-022-8/+19
| | | | | | | (#11878) The get_users_in_room and get_users_in_room_with_profiles are now only invalidated when the membership of a room changes, instead of during any state change in the room.
* Revert experimental push rules from #7997. (#11884)Patrick Cloke2022-02-021-16/+4
| | | Manually reverts the merge from cdbb8e6d6e36e0b6bc36e676d8fe66c96986b399.
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-022-55/+129
| | | | users. (#11655)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-013-46/+275
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove the obsolete MSC1849 configuration flag. (#11843)Patrick Cloke2022-01-311-4/+0
| | | | | MSC1849 was replaced by MSC2675, which was merged. The configuration flag, which defaulted to true, is no longer useful.
* Pass `isolation_level` to `runWithConnection` (#11847)Brendan Abolivier2022-01-271-0/+1
| | | This was missed in https://github.com/matrix-org/synapse/pull/11799
* Create singletons for `StateFilter.{all,none}()` (#11836)Richard van der Hoff2022-01-271-5/+9
| | | No point recreating these for each call, since they are frozen
* Include `prev_content` field in AS events (#11798)Vaishnav Nair2022-01-261-1/+1
| | | | | | * Include 'prev_content' field in AS events Signed-off-by: Vaishnav Nair <nairvaishnav007@icloud.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Improvements to bundling aggregations. (#11815)Patrick Cloke2022-01-262-30/+53
| | | | | | | | | | | This is some odds and ends found during the review of #11791 and while continuing to work in this code: * Return attrs classes instead of dictionaries from some methods to improve type safety. * Call `get_bundled_aggregations` fewer times. * Adds a missing assertion in the tests. * Do not return empty bundled aggregations for an event (preferring to not include the bundle at all, as the docstring states).
* Add admin API to get a list of federated rooms (#11658)Dirk Klimpel2022-01-251-0/+48
|
* Db txn set isolation level (#11799)Nick Barrett2022-01-254-5/+60
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Minor updates, and docs, for schema delta files (#11823)Richard van der Hoff2022-01-251-3/+6
| | | | | | | | | * Make functions in python deltas optional It's annoying to always have to write stubs for these. * Documentation for delta files * changelog
* Merge tag 'v1.51.0rc2' into developAndrew Morgan2022-01-241-1/+4
|\ | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc2 (2022-01-24) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
| * Fix logic for dropping old events in fed queue (#11806)Andrew Morgan2022-01-241-1/+4
| | | | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
* | Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-241-2/+71
| | | | | | | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Drop support for and remove references to EOL Python 3.6 (#11683)Shay2022-01-211-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
* | Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-213-4/+37
| | | | | | | | ... and start populating them for new events
* | Drop unused table `public_room_list_stream`. (#11795)Richard van der Hoff2022-01-213-2/+21
| | | | | | This is a follow-up to #10565.
* | Stop reading from `event_reference_hashes` (#11794)Richard van der Hoff2022-01-213-32/+29
| | | | | | | | Preparation for dropping this table altogether. Part of #6574.
* | Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre2022-01-211-4/+4
| | | | | | | | whose key is prefixed with the user ID (#11788)
* | Make `get_account_data_for_room_and_type` a tree cache (#11789)reivilibre2022-01-211-1/+1
|/
* Debug for device lists updates (#11760)David Robertson2022-01-201-0/+18
| | | | | | | | | | | | | | | | | | Debug for #8631. I'm having a hard time tracking down what's going wrong in that issue. In the reported example, I could see server A sending federation traffic to server B and all was well. Yet B reports out-of-sync device updates from A. I couldn't see what was _in_ the events being sent from A to B. So I have added some crude logging to track - when we have updates to send to a remote HS - the edus we actually accumulate to send - when a federation transaction includes a device list update edu - when such an EDU is received This is a bit of a sledgehammer.
* Comments and typing for `_update_outliers_txn` (#11776)Richard van der Hoff2022-01-191-12/+23
| | | | A couple of surprises for me here, so thought I'd document them
* Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke2022-01-182-11/+62
| | | | | | Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
* Remove `log_function` and its uses (#11761)Richard van der Hoff2022-01-181-2/+0
| | | | | | | I've never found this terribly useful. I think it was added in the early days of Synapse, without much thought as to what would actually be useful to log, and has just been cargo-culted ever since. Rather, it tends to clutter up debug logs with useless information.
* Make pagination of rooms in admin api stable (#11737)Daniel Sonck2022-01-171-9/+9
| | | | | | | | | | | | | | Always add state.room_id after the configurable ORDER BY. Otherwise, for any sort, certain pages can contain results from other pages. (Especially when sorting by creator, since there may be many rooms by the same creator) * Document different order direction of numerical fields "joined_members", "joined_local_members", "version" and "state_events" are ordered in descending direction by default (dir=f). Added a note in tests to explain the differences in ordering. Signed-off-by: Daniël Sonck <daniel@sonck.nl>
* Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-141-17/+85
|\
| * Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could ↵reivilibre2022-01-131-16/+78
| | | | | | | | | | fail because too many EDUs were produced for device updates. (#11730) Co-authored-by: David Robertson <davidr@element.io>
| * Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not ↵reivilibre2022-01-121-1/+7
| | | | | | | | | | be sent to remote homeservers if there were too many to send at once. (#11729) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-1315-251/+224
| | | | | | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* | Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-1312-61/+61
| |
* | Fix docstring on `add_account_data_for_user`. (#11716)reivilibre2022-01-101-1/+1
| |
* | Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke2022-01-071-3/+125
| | | | | | | | | | | | | | | | This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
* | Fix get federation status of destination if no error occured (#11593)Dirk Klimpel2022-01-051-0/+11
| |
* | Run `pyupgrade --py37-plus --keep-percent-format` on Synapse (#11685)Shay2022-01-051-1/+0
| | | | | | | | | | | | | | | | | | * newsfragment * fix newsfragment number * update changelog * remove extra space
* | Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-051-2/+6
|/ | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Remove redundant `get_current_events_token` (#11643)Richard van der Hoff2022-01-043-25/+17
| | | | | | | | | | | | | | | | | * Push `get_room_{min,max_stream_ordering}` into StreamStore Both implementations of this are identical, so we may as well push it down and get rid of the abstract base class nonsense. * Remove redundant `StreamStore` class This is empty now * Remove redundant `get_current_events_token` This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid of it. * newsfile
* Better error messages from `get_create_event_for_room` (#11638)Richard van der Hoff2022-01-041-1/+5
| | | | "Unknown room" can mean a multitude of things here. To help with debugging, add some more words to the exception text.
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-306-37/+54
| | | To improve type hints throughout the code.
* Add type hints to `synapse/storage/databases/main/events_bg_updates.py` (#11654)Dirk Klimpel2021-12-301-29/+40
|
* Improve type hints in storage classes. (#11652)Dirk Klimpel2021-12-299-34/+43
| | | By using cast and making ignores more specific.
* Add type hints to `synapse/storage/databases/main/stats.py` (#11653)Dirk Klimpel2021-12-291-41/+53
|
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-211-91/+158
|
* Add type hints to `synapse/storage/databases/main/transactions.py` (#11589)Dirk Klimpel2021-12-161-24/+25
|
* Add type hints to `synapse/storage/databases/main/room.py` (#11575)Sean Quah2021-12-152-74/+100
|
* Require Collections as the parameters for simple_* methods. (#11580)Patrick Cloke2021-12-152-40/+11
| | | | Instead of Iterable since the generators are not allowed due to the potential for their re-use.
* Add experimental support for MSC3202: allowing application services to ↵reivilibre2021-12-151-0/+20
| | | | masquerade as specific devices. (#11538)
* Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-141-70/+167
|
* Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-147-20/+18
| | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* checks for generators in database functions (#11564)Richard van der Hoff2021-12-132-7/+48
| | | | | A couple of safety-checks to hopefully stop people doing what I just did, and create a storage function which only works the first time it is called (and not when it is re-run due to a database concurrency error or similar).
* Type hint the constructors of the data store classes (#11555)Sean Quah2021-12-1328-75/+308
|
* Add type hints to `synapse/storage/databases/main/end_to_end_keys.py` (#11551)Sean Quah2021-12-132-68/+146
|
* Add type hints to `synapse/storage/databases/main/account_data.py` (#11546)Sean Quah2021-12-132-32/+83
|
* Make `get_device` return None if the device doesn't exist rather than ↵reivilibre2021-12-131-4/+6
| | | | | raising an exception. (#11565) Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* skip some dict munging in event persistence (#11560)Richard van der Hoff2021-12-102-60/+113
| | | | | | Create a new dict helper method `simple_insert_many_values_txn`, which takes raw row values, rather than {key=>value} dicts. This saves us a bunch of dict munging, and makes it easier to use generators rather than creating intermediate lists and dicts.
* Stop populating `state_events.prev_state` (#11558)Richard van der Hoff2021-12-102-5/+4
| | | this field is never read, so we may as well stop populating it.
* Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke2021-12-102-2/+4
| | | | This should fix pushers with an email in non-canonical form is used as the pushkey.
* Do not allow cross-room relations, per MSC2674. (#11516)Patrick Cloke2021-12-092-12/+32
|
* Add a constant for receipt types (m.read). (#11531)Patrick Cloke2021-12-081-33/+68
| | | And expand some type hints in the receipts storage module.
* Improve log messages for stream ids (#11536)Richard van der Hoff2021-12-082-4/+6
| | | | Somehow I'd managed to get my database in a pickle with stream ids. These changes were useful to debug.
* Send and handle cross-signing messages using the stable prefix. (#10520)Hubert Chathi2021-12-081-1/+3
|
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-062-1/+76
| | | As a step towards allowing back-channel logout for OIDC.
* Add admin API to get some information about federation status (#11407)Dirk Klimpel2021-12-061-0/+70
|
* Disambiguate queries on `state_key` (#11497)Richard van der Hoff2021-12-026-16/+20
| | | | | We're going to add a `state_key` column to the `events` table, so we need to add some disambiguation to queries which use it.
* Comments on the /sync tentacles (#11494)David Robertson2021-12-021-4/+11
| | | | | This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add type annotations to `tests.storage.test_appservice`. (#11488)reivilibre2021-12-021-3/+3
|
* Add MSC3030 experimental client and federation API endpoints to get the ↵Eric Eastwood2021-12-021-0/+195
| | | | | | | | | | | | | | | | | | | | | | | | | closest event to a given timestamp (#9445) MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030 Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about. ``` GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction> { "event_id": ... "origin_server_ts": ... } ``` Federation API endpoint: ``` GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction> { "event_id": ... "origin_server_ts": ... } ``` Co-authored-by: Erik Johnston <erik@matrix.org>
* Improved push typing (#11409)Marcus2021-11-301-2/+17
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Refactor `backfilled` into specific behavior function arguments ↵Eric Eastwood2021-11-292-21/+56
| | | | | | | | | | | | | | | (`_persist_events_and_state_updates`) (#11417) Part of https://github.com/matrix-org/synapse/issues/11300 Call stack: - `_persist_events_and_state_updates` (added `use_negative_stream_ordering`) - `_persist_events_txn` - `_update_room_depths_txn` (added `update_room_forward_stream_ordering`) - `_update_metadata_tables_txn` - `_store_room_members_txn` (added `inhibit_local_membership_updates`) Using keyword-only arguments (`*`) to reduce the mistakes from `backfilled` being left as a positional argument somewhere and being interpreted wrong by our new arguments.
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-291-23/+169
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-265-148/+230
| | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* Support expiry of refresh tokens and expiry of the overall session when ↵reivilibre2021-11-262-2/+54
| | | | refresh tokens are in use. (#11425)
* Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-42/+112
| | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-252-146/+81
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-241-1/+1
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add missing type hints to config base classes (#11377)Patrick Cloke2021-11-231-1/+2
|
* Remove code invalidated by deprecated config flag ↵Shay2021-11-231-32/+3
| | | | | | | | | | | | | | | | | | | | | 'trust_identity_servers_for_password_resets' (#11395) * remove background update code related to deprecated config flag * changelog entry * update changelog * Delete 11394.removal Duplicate, wrong number * add no-op background update and change newfragment so it will be consolidated with associated work * remove unused code * Remove code associated with deprecated flag from legacy docker dynamic config file Co-authored-by: reivilibre <oliverw@matrix.org>
* Store arbitrary relations from events. (#11391)Patrick Cloke2021-11-223-44/+75
| | | | | Instead of only known relation types. This also reworks the background update for thread relations to crawl events and search for any relation type, not just threaded relations.
* Add an admin API to run background jobs. (#11352)Dirk Klimpel2021-11-191-0/+2
| | | | | | Instead of having admins poke into the database directly. Can currently run jobs to populate stats and to populate the user directory.
* Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-11/+40
|
* Add dedicated admin API for blocking a room (#11324)Dirk Klimpel2021-11-181-0/+32
|
* Use auto_attribs for RefreshTokenLookupResult (#11386)reivilibre2021-11-181-7/+7
|
* Do not allow MSC3440 threads to fork threads (#11161)Patrick Cloke2021-11-181-2/+65
| | | | | | | | | | | Adds validation to the Client-Server API to ensure that the potential thread head does not relate to another event already. This results in not allowing a thread to "fork" into other threads. If the target event is unknown for some reason (maybe it isn't visible to your homeserver), but is the target of other events it is assumed that the thread can be created from it. Otherwise, it is rejected as an unknown event.
* Add type annotations to `synapse.metrics` (#10847)Sean Quah2021-11-171-3/+3
|
* Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-162-19/+35
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Rename `remove_deleted_devices_from_device_inbox` to ensure it is always run ↵Andrew Morgan2021-11-161-1/+13
| | | | | | | | | | (#11353) Co-authored-by: reivilibre <oliverw@matrix.org>
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-18/+22
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-151-1/+1
| |
* | Add ability to un-shadow-ban via the admin API. (#11347)Patrick Cloke2021-11-161-1/+1
| |
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-4/+8
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Get directory db file to pass mypy (#11339)David Robertson2021-11-152-6/+7
| |
* | Remove unused tables `room_stats_historical` and `user_stats_historical` ↵Shay2021-11-122-1/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#11280) * remove unused tables room_stats_historical and user_stats_historical * update changelog number * Bump schema compat version comment * make linter happy * Update comment to give more info Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* | Annotations for state_deltas.py (#11316)David Robertson2021-11-121-3/+13
| | | | | | | | | | I was sad that I couldn't do better for `_curr_state_delta_stream_cache`. At least it's explicitly called out in a comment with #TODO.
* | Test room alias deletion (#11327)David Robertson2021-11-121-2/+5
| | | | | | | | | | | | | | | | | | | | * Prefer `HTTPStatus` over plain `int` This is an Opinion that no-one has seemed to object to yet. * `--disallow-untyped-defs` for `tests.rest.client.test_directory` * Improve synapse's annotations for deleting aliases * Test case for deleting a room alias * Changelog
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Add type hints to media repository storage module (#11311)Patrick Cloke2021-11-121-57/+84
| |
* | Attempt to annotate events_forward_extremities (#11314)David Robertson2021-11-122-7/+16
| | | | | | | | | | | | * Make DataStore inherit from EventForwardExtremitiesStore before CacheInvalidationWorkerStore the former implicitly inherits from the latter, so they should be ordered like this when used.
* | Annotations for user_erasure_store (#11313)David Robertson2021-11-111-4/+5
| | | | | | | | I'm not sure why this was excluded---it seemed to be passing for me. But it's easy enough to fixup.
* | Get db signatures file to pass mypy (#11312)David Robertson2021-11-111-2/+2
| |
* | Correct type hint for room_batch.py (#11310)David Robertson2021-11-111-10/+8
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-111-1/+1
| |
* | Add type hints to some storage classes (#11307)Patrick Cloke2021-11-117-47/+115
| |
* | Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-092-7/+24
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Allow admins to proactively block rooms (#11228)David Robertson2021-11-091-1/+6
| | | | | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-092-23/+121
|/ | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Rename to more clear `get_insertion_event_id_by_batch_id` (MSC2716) (#11244)Eric Eastwood2021-11-081-1/+1
| | | | | `get_insertion_event_by_batch_id` -> `get_insertion_event_id_by_batch_id` Split out from https://github.com/matrix-org/synapse/pull/11114
* Add some background update admin APIs (#11263)Erik Johnston2021-11-082-18/+51
| | | Fixes #11259
* Fix typo in comment from #11255. (#11276)Patrick Cloke2021-11-081-1/+1
|
* Handle federation inbound instances being killed more gracefully (#11262)Erik Johnston2021-11-081-10/+21
| | | | | | | | | | | | | | | | | * Make lock better handle process being killed If the process gets killed and restarted (so that it didn't have a chance to drop its locks gracefully) then there may still be locks in the DB that are for the same instance that haven't yet timed out but are safe to delete. We handle this case by a) checking if the current instance already has taken out the lock, and b) if not then ignoring locks that are for the same instance. * Periodically check for old staged events This is to protect against other instances dying and their locks timing out.
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-11/+12
| | | Fixes #11252
* Track ongoing event fetches correctly in the presence of failure (#11240)Sean Quah2021-11-041-22/+34
| | | | | | When an event fetcher aborts due to an exception, `_event_fetch_ongoing` must be decremented, otherwise the event fetcher would never be replaced. If enough event fetchers were to fail, no more events would be fetched and requests would get stuck waiting for events.
* Add index to `local_group_updates.stream_id` (#11231)Erik Johnston2021-11-022-1/+34
| | | | This should speed up startup times and generally increase performance of groups.
* Add remaining type hints to `synapse.events`. (#11098)Patrick Cloke2021-11-022-6/+9
|
* Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-022-0/+111
|
* Fix providing a `RoomStreamToken` instance to ↵Andrew Morgan2021-11-022-3/+3
| | | | | `_notify_app_services_ephemeral` (#11137) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add search by room ID and room alias to List Room admin API (#11099)Dirk Klimpel2021-11-021-11/+18
| | | | Fixes: #10874 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Add metrics to the threadpools (#11178)Erik Johnston2021-11-011-1/+6
|
* Stop synapse from saving messages in device_inbox for hidden devices. (#10097)JohannesKleine2021-11-011-2/+6
| | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fix comments referencing v1.46.0 from PR #10969. (#11212)Dirk Klimpel2021-10-292-2/+2
| | | | #10969 was merged after 1.46.0rc1 was cut and will be included in v1.47.0rc1 instead.
* Additional type hints for relations database class. (#11205)Patrick Cloke2021-10-281-15/+23
|
* Annotate `log_function` decorator (#10943)reivilibre2021-10-271-1/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-273-15/+134
| | | Fixes: #9346
* Fix thread BG update to not seq scan event_json (#11192)Erik Johnston2021-10-271-1/+1
| | | | For some reason the query optimiser decided to seq scan both tables, rather than index scanning `event_json`.
* Add a background update for updating MSC3440 relation threads. (#11181)Patrick Cloke2021-10-262-2/+101
|
* Enable changing user type via users admin API (#11174)Jason Robinson2021-10-261-0/+18
| | | | | | | | Users admin API can now also modify user type in addition to allowing it to be set on user creation. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Fix module API's `get_user_ip_and_agents` function when run on workers (#11112)Sean Quah2021-10-251-39/+85
|
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-2222-59/+148
|
* Add a thread relation type per MSC3440. (#11088)Patrick Cloke2021-10-212-1/+62
| | | | Adds experimental support for MSC3440's `io.element.thread` relation type (and the aggregation for it).
* Fix setting a user's external_id via the admin API returns 500 and deletes ↵Dirk Klimpel2021-10-211-5/+90
| | | | | users existing external mappings if that external ID is already mapped (#11051) Fixes #10846
* Merge branch 'master' into developSean Quah2021-10-201-24/+0
|\
| * Revert change to counting of deactivated users towards the monthly active ↵Sean Quah2021-10-201-24/+0
| | | | | | | | | | | | | | | | users limit (#11127) Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)". This reverts commit eda8c88b84ee7506379a71ac2a7a88c08b759d43.
* | Add missing type hints to event fetching. (#11121)Patrick Cloke2021-10-191-61/+81
| | | | | | | | Updates the event rows returned from the database to be attrs classes instead of dictionaries.
* | Add missing type hints to synapse.api. (#11109)Patrick Cloke2021-10-181-4/+4
| | | | | | | | * Convert UserPresenceState to attrs. * Remove args/kwargs from error classes and explicitly pass msg/errorcode.
* | `_run_push_actions_and_persist_event`: handle no min_depth (#11014)Richard van der Hoff2021-10-181-1/+1
| | | | | | | | | | Make sure that we correctly handle rooms where we do not yet have a `min_depth`, and also add some comments and logging.
* | Move experimental & retention config out of the server module. (#11070)Patrick Cloke2021-10-151-4/+4
| |
* | Resolve and share `state_groups` for all historical events in batch ↵Eric Eastwood2021-10-134-5/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#10975) Resolve and share `state_groups` for all historical events in batch. This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch. This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room. For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch. --- How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example: **Before** (new `state_group` for every event 😬, very inefficient): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10 create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11 create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12 create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13 create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14 ``` **After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10 create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10 create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10 create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10 create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10 ```
* | Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-131-2/+22
|\|
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-133-7/+202
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-0/+2
| | | | | | | | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Merge remote-tracking branch 'origin/release-v1.45' into developErik Johnston2021-10-121-15/+67
|\|
| * Fix race in `MultiWriterIdGenerator` (#11045)Erik Johnston2021-10-121-15/+67
| | | | | | | | | | | | | | | | | | | | The race allowed the current position to advance too far when stream IDs are still being persisted. This happened when it received a new stream ID from a remote write between a new stream ID being allocated and it being added to the set of unpersisted stream IDs. Fixes #9424.
* | Add type hints to `synapse.storage.databases.main.client_ips` (#10972)Sean Quah2021-10-121-40/+100
| |
* | Fix inconsistent behavior of `get_last_client_by_ip` (#10970)Sean Quah2021-10-121-4/+9
| | | | | | | | | | | | | | Make `get_last_client_by_ip` return the same dictionary structure regardless of whether the data has been persisted to the database. This change will allow slightly cleaner type hints to be applied later on.
* | Add an approximate difference method to StateFilters (#10825)reivilibre2021-10-121-1/+171
|/
* disallow-untyped-defs for synapse.push (#11023)David Robertson2021-10-111-2/+2
|
* Annotate synapse.storage.util (#10892)David Robertson2021-10-084-60/+108
| | | | | Also mark `synapse.streams` as having has no untyped defs Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-081-0/+4
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-062-2/+6
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-13/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix logic flaw preventing tracking of MSC2716 events in existing room ↵Eric Eastwood2021-10-051-6/+4
| | | | | | | | | | | | | | | versions (#10962) We correctly allowed using the MSC2716 batch endpoint for the room creator in existing room versions but accidentally didn't track the events because of a logic flaw. This prevented you from connecting subsequent chunks together because it would throw the unknown batch ID error. We only want to process MSC2716 events when: - The room version supports MSC2716 - Any room where the homeserver has the `msc2716_enabled` experimental feature enabled and the event is from the room creator
* Add functionality to remove deactivated users from the monthly_active_users ↵Hillery Shay2021-10-041-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | table (#10947) * add test * add function to remove user from monthly active table in deactivate code * add function to remove user from monthly active table * add changelog entry * update changelog number * requested changes * update docstring on new function * fix lint error * Update synapse/storage/databases/main/monthly_active_users.py Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-041-2/+2
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-2/+2
| |
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-041-13/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-041-1/+1
| |
* | Add type hints to filtering classes. (#10958)Patrick Cloke2021-10-011-2/+6
| |
* | Refactor user directory tests (#10935)David Robertson2021-09-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-298-19/+21
| |
* | Ensure `(room_id, next_batch_id)` is unique to avoid cross-talk/conflicts ↵Eric Eastwood2021-09-281-2/+4
| | | | | | | | | | | | | | | | between batches (MSC2716) (#10877) Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) Part of https://github.com/matrix-org/synapse/issues/10737
* | Drop backwards-compatibility support for "outlier" (#10903)Richard van der Hoff2021-09-282-25/+3
|/ | | | | Before Synapse 1.31 (#9411), we relied on `outlier` being stored in the `internal_metadata` column. We can now assume nobody will roll back their deployment that far and drop the legacy support.
* Document changes to schema version 61 - 64 (#10917)Eric Eastwood2021-09-241-0/+11
| | | | | As pointed out by @richvdh, https://github.com/matrix-org/synapse/pull/10838#discussion_r715424244 Retroactively summarize `61` - `64`
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-243-4/+4
|
* Improve typing in user_directory files (#10891)David Robertson2021-09-241-35/+89
| | | | | | | | | | | * Improve typing in user_directory files This makes the user_directory.py in storage pass most of mypy's checks (including `no-untyped-defs`). Unfortunately that file is in the tangled web of Store class inheritance so doesn't pass mypy at the moment. The handlers directory has already been mypyed. Co-authored-by: reivilibre <olivier@librepush.net>
* In `_purge_history_txn`, ensure that txn.fetchall has elements before ↵Kokokokoka2021-09-241-9/+13
| | | | | | accessing rows (#10690) This change adds a check for row existence before accessing row element, this should fix issue #10669 Signed-off-by: Vasya Boytsov vasiliy.boytsov@phystech.edu
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+1
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-235-5/+5
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.