summary refs log tree commit diff
path: root/synapse/handlers/sync.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'release-v1.76' into developDavid Robertson2023-01-271-7/+17
|\
| * Batch look-ups to see if rooms are partial stated. (#14917)Patrick Cloke2023-01-261-7/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Batch look-ups to see if rooms are partial stated. * Fix issues found in linting. * Fix typo. * Apply suggestions from code review Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Clarify comments. Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Also improve the cache size while we're at it * is_partial_state_rooms -> is_partial_state_room_batched * Run `black` * Improve annotation for `simple_select_many_batch` * Fix is_partial_state_room_batched impl * Okay, _actually_ fix impl * Update description. * Update synapse/storage/databases/main/room.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Run black. Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: David Robertson <davidr@element.io>
* | Use StrCollection in place of Collection[str] in (most) handlers code. (#14922)Patrick Cloke2023-01-261-2/+2
|/ | | | Due to the increased safety of StrCollection over Collection[str] and Sequence[str].
* Faster joins: omit partial rooms from eager syncs until the resync completes ↵David Robertson2023-01-231-10/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14870) * Allow `AbstractSet` in `StrCollection` Or else frozensets are excluded. This will be useful in an upcoming commit where I plan to change a function that accepts `List[str]` to accept `StrCollection` instead. * `rooms_to_exclude` -> `rooms_to_exclude_globally` I am about to make use of this exclusion mechanism to exclude rooms for a specific user and a specific sync. This rename helps to clarify the distinction between the global config and the rooms to exclude for a specific sync. * Better function names for internal sync methods * Track a list of excluded rooms on SyncResultBuilder I plan to feed a list of partially stated rooms for this sync to ignore * Exclude partial state rooms during eager sync using the mechanism established in the previous commit * Track un-partial-state stream in sync tokens So that we can work out which rooms have become fully-stated during a given sync period. * Fix mutation of `@cached` return value This was fouling up a complement test added alongside this PR. Excluding a room would mean the set of forgotten rooms in the cache would be extended. This means that room could be erroneously considered forgotten in the future. Introduced in #12310, Synapse 1.57.0. I don't think this had any user-visible side effects (until now). * SyncResultBuilder: track rooms to force as newly joined Similar plan as before. We've omitted rooms from certain sync responses; now we establish the mechanism to reintroduce them into future syncs. * Read new field, to present rooms as newly joined * Force un-partial-stated rooms to be newly-joined for eager incremental syncs only, provided they're still fully stated * Notify user stream listeners to wake up long polling syncs * Changelog * Typo fix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Unnecessary list cast Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Rephrase comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Another comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Fixup merge(?) * Poke notifier when receiving un-partial-stated msg over replication * Fixup merge whoops Thanks MV :) Co-authored-by: Mathieu Velen <mathieuv@matrix.org> Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Merge account data streams (#14826)Erik Johnston2023-01-131-2/+9
|
* Calculate rooms changed for device lists to work. (#14810)Patrick Cloke2023-01-111-4/+0
| | | | Back-out some changes from 7e582a25f8f350df29d7d83ca902bdb522d1bbaf (#14786) which skipped necessary logic to calculate device lists properly.
* Improve /sync performance of when passing filters with empty arrays. (#14786)Patrick Cloke2023-01-091-3/+11
| | | | | | | | | | | | | | | | This has two related changes: * It enables fast-path processing for an empty filter (`[]`) which was previously only used for wildcard not-filters (`["*"]`). * It special cases a `/sync` filter with no-rooms to skip all room processing, previously we would partially skip processing, but would generally still calculate intermediate values for each room which were then unused. Future changes might consider further optimizations: * Skip calculating per-room account data when all rooms are filtered (currently this is thrown away). * Make similar improvements to other endpoints which support filters.
* Log to-device msgids when we return them over /sync (#14724)Richard van der Hoff2022-12-231-7/+13
|
* Move `StateFilter` to `synapse.types` (#14668)David Robertson2022-12-121-1/+1
| | | | | * Move `StateFilter` to `synapse.types` * Changelog
* Improve logging and opentracing for to-device message handling (#14598)Richard van der Hoff2022-12-061-7/+19
| | | | | | | A batch of changes intended to make it easier to trace to-device messages through the system. The intention here is that a client can set a property org.matrix.msgid in any to-device message it sends. That ID is then included in any tracing or logging related to the message. (Suggestions as to where this field should be documented welcome. I'm not enthusiastic about speccing it - it's very much an optional extra to help with debugging.) I've also generally improved the data we send to opentracing for these messages.
* Better return type for `get_all_entities_changed` (#14604)Erik Johnston2022-12-051-2/+4
| | | | Help callers from using the return value incorrectly by ensuring that callers explicitly check if there was a cache hit or not.
* Use `device_one_time_keys_count` to match MSC3202 (#14565)Andrew Ferrazzutti2022-11-281-3/+3
| | | | | | | | | | | | | | | | | | | * Use `device_one_time_keys_count` to match MSC3202 Rename the `device_one_time_key_counts` key in responses to `device_one_time_keys_count` to match the name specified by MSC3202. Also change related variable/class names for consistency. Signed-off-by: Andrew Ferrazzutti <andrewf@element.io> * Update changelog.d/14565.misc * Revert name change for `one_time_key_counts` key as this is a different key altogether from `device_one_time_keys_count`, which is used for `/sync` instead of appservice transactions. Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
* Include heroes in partial join responses' state (#14442)David Robertson2022-11-151-17/+3
| | | | | | | | | | | * Pull out hero selection logic * Include heroes in partial join response's state * Changelog * Fixup trial test * Remove TODO
* Use stable identifiers for MSC3771 & MSC3773. (#14050)Patrick Cloke2022-10-071-6/+1
| | | | | These are both part of Matrix 1.4 which has now been released. For now, support both the unstable and stable identifiers.
* Remove get rooms for user with stream ordering (#13991)Nick Mills-Barrett2022-10-041-80/+69
| | | | | | By getting the joined rooms before the current token we avoid any reading history to confirm a user *was* in a room. We can then use any membership change events, which we already fetch during sync, to determine the final list of joined room IDs.
* Track notification counts per thread (implement MSC3773). (#13776)Patrick Cloke2022-10-041-5/+35
| | | | | | | | When retrieving counts of notifications segment the results based on the thread ID, but choose whether to return them as individual threads or as a single summed field by letting the client opt-in via a sync flag. The summarization code is also updated to be per thread, instead of per room.
* Optimise get_rooms_for_user (drop with_stream_ordering) (#13787)Nick Mills-Barrett2022-09-291-10/+4
|
* Faster room joins: Fix spurious error when joining a room (#13872)Sean Quah2022-09-271-3/+19
| | | | | | | | | | | | | | | | During a `lazy_load_members` `/sync`, we look through auth events in rooms with partial state to find prior membership events. When such a membership is not found, an error is logged. Since the first join event for a user never has a prior membership event to cite, the error would always be logged when one appeared in the room timeline. Avoid logging errors for such events. Introduced in #13477. Signed-off-by: Sean Quah <seanq@matrix.org>
* Re-type hint some collections in `/sync` code as read-only (#13754)Sean Quah2022-09-081-10/+10
| | | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Cache user IDs instead of profile objects (#13573)Nick Mills-Barrett2022-08-231-2/+2
| | | The profile objects are never used and increase cache size significantly.
* Avoid blocking lazy-loading `/sync`s during partial joins (#13477)Sean Quah2022-08-181-30/+223
| | | | | | | | | | | | | | | | | Use a state filter or accept partial state in a few places where we request state, to avoid blocking. To make lazy-loading `/sync`s work, we need to provide the memberships of event senders, which are not guaranteed to be in the room state. Instead we dig through auth events for memberships to present to clients. The auth events of an event are guaranteed to contain a passable membership event, otherwise the event would have been rejected. Note that this only covers the common code paths encountered during testing. There has been no exhaustive checking of all sync code paths. Fixes #13146. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add some miscellaneous comments around sync (#13474)Sean Quah2022-08-101-38/+78
| | | | | | | | Add some miscellaneous comments to document sync, especially around `compute_state_delta`. Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix rooms not being properly excluded from incremental sync (#13408)Brendan Abolivier2022-08-041-10/+15
|
* Use new `device_list_changes_in_room` table when getting device list changes ↵Erik Johnston2022-06-171-14/+5
| | | | (#13045)
* Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston2022-06-151-7/+3
| | | | | | | | | | | Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
* Decouple `synapse.api.auth_blocking.AuthBlocking` from ↵Quentin Gliech2022-06-141-2/+2
| | | | `synapse.api.auth.Auth`. (#13021)
* Wait for lazy join to complete when getting current state (#12872)Erik Johnston2022-06-011-4/+9
|
* Fix 404 on `/sync` when the last event is a redaction of an unknown/purged ↵Richard van der Hoff2022-06-011-8/+19
| | | | | | | | | | | event (#12905) Currently, we try to pull the event corresponding to a sync token from the database. However, when we fetch redaction events, we check the target of that redaction (because we aren't allowed to send redactions to clients without validating them). So, if the sync token points to a redaction of an event that we don't have, we have a problem. It turns out we don't really need that event, and can just work with its ID and metadata, which sidesteps the whole problem.
* Rename storage classes (#12913)Erik Johnston2022-05-311-12/+14
|
* Fix up `state_store` naming (#12871)Erik Johnston2022-05-251-10/+14
|
* Remove user-visible groups/communities code (#12553)Patrick Cloke2022-05-251-65/+0
| | | | | | | | | Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove: * All API endpoints (including Client-Server, Server-Server, and admin). * Documented configuration options (and the experimental flag, which is now unused). * Special handling during room upgrades. * The `groups` section of the `/sync` response.
* Add some type hints to datastore (#12717)Dirk Klimpel2022-05-171-3/+3
|
* Add `StreamKeyType` class and replace string literals with constants (#12567)Andrew Morgan2022-05-161-9/+14
|
* Implement changes to MSC2285 (hidden read receipts) (#12168)Šimon Brandner2022-05-041-1/+1
| | | | | * Changes hidden read receipts to be a separate receipt type (instead of a field on `m.read`). * Updates the `/receipts` endpoint to accept `m.fully_read`.
* Fix missing sync events during historical batch imports (#12319)Nick Mills-Barrett2022-04-131-8/+7
| | | | | | | | Discovered after much in-depth investigation in #12281. Closes: #12281 Closes: #3305 Signed off by: Nick Mills-Barrett nick@beeper.com
* Fix rejecting invites over federation (#12409)Erik Johnston2022-04-071-20/+33
| | | | | Currently causes future incremental syncs to fail. Broke by #12191
* Do not add groups to sync results if disabled. (#12408)Patrick Cloke2022-04-071-2/+3
|
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-31/+9
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add a configuration to exclude rooms from sync response (#12310)Brendan Abolivier2022-03-301-6/+17
|
* Move get_bundled_aggregations to relations handler. (#12237)Patrick Cloke2022-03-181-3/+6
| | | | | The get_bundled_aggregations code is fairly high-level and uses a lot of store methods, we move it into the handler as that seems like a better fit.
* Use the ignored_users table to test event visibility & sync. (#12225)Patrick Cloke2022-03-151-28/+2
| | | | | Instead of fetching the raw account data and re-parsing it. The ignored_users table is a denormalised version of the account data for quick searching.
* Add logging to `/sync` for debugging #11916 (#12068)Erik Johnston2022-02-231-0/+9
|
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Optimise calculating device_list changes in `/sync`. (#11974)Erik Johnston2022-02-151-15/+53
| | | | | | For users with large accounts it is inefficient to calculate the set of users they share a room with (and takes a lot of space in the cache). Instead we can look at users whose devices have changed since the last sync and check if they share a room with the syncing user.
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-2/+2
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Improvements to bundling aggregations. (#11815)Patrick Cloke2022-01-261-1/+2
| | | | | | | | | | | This is some odds and ends found during the review of #11791 and while continuing to work in this code: * Return attrs classes instead of dictionaries from some methods to improve type safety. * Call `get_bundled_aggregations` fewer times. * Adds a missing assertion in the tests. * Do not return empty bundled aggregations for an event (preferring to not include the bundle at all, as the docstring states).
* Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre2022-01-211-1/+1
| | | | whose key is prefixed with the user ID (#11788)
* Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke2022-01-181-1/+3
| | | | | | Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
* Include bundled aggregations in the sync response cache. (#11659)Patrick Cloke2022-01-131-0/+10
|
* Remove debug logging for #4422 (#11693)David Robertson2022-01-061-37/+1
| | | | as per https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-211-6/+6
|
* Various opentracing enhancements (#11619)Richard van der Hoff2021-12-211-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | * Wrap `auth.get_user_by_req` in an opentracing span give `get_user_by_req` its own opentracing span, since it can result in a non-trivial number of sub-spans which it is useful to group together. This requires a bit of reorganisation because it also sets some tags (and may force tracing) on the servlet span. * Emit opentracing span for encoding json responses This can be a significant time sink. * Rename all sync spans with a prefix * Write an opentracing span for encoding sync response * opentracing span to group generate_room_entries * opentracing spans within sync.encode_response * changelog * Use the `trace` decorator instead of context managers
* Adjust _get_rooms_changed comments (#11550)David Robertson2021-12-101-22/+29
| | | C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
* Add a constant for receipt types (m.read). (#11531)Patrick Cloke2021-12-081-2/+2
| | | And expand some type hints in the receipts storage module.
* Refactors in `_generate_sync_entry_for_rooms` (#11515)David Robertson2021-12-071-45/+77
| | | | | | | | * Move sync_token up to the top * Pull out _get_ignored_users * Try to signpost the body of `_generate_sync_entry_for_rooms` * Pull out _calculate_user_changes Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correctly ignore invites from ignored users (#11511)David Robertson2021-12-071-5/+6
|
* Comments on the /sync tentacles (#11494)David Robertson2021-12-021-39/+117
| | | | | This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-8/+10
| | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-211-3/+3
|
* Require type hints in the handlers module. (#10831)Patrick Cloke2021-09-201-5/+6
| | | | | | | Adds missing type hints to methods in the synapse.handlers module and requires all methods to have type hints there. This also removes the unused construct_auth_difference method from the FederationHandler.
* Merge branch 'release-v1.43' into developDavid Robertson2021-09-171-126/+141
|\
| * Cleanup opentracing logging for syncs (#10828)Erik Johnston2021-09-151-126/+141
| | | | | | | | | | We added a bunch of spans in #10704, but this ended up adding a lot of redundant spans for rooms where nothing changed, so instead we only start the span if there might be something interesting going on.
* | Use direct references for some configuration variables (part 2) (#10812)Patrick Cloke2021-09-151-1/+1
|/
* Add logging to help debug #9424 (#10704)Azrenbeth2021-09-061-13/+54
|
* Additional type hints for the sync REST servlet. (#10666)Patrick Cloke2021-08-231-10/+11
|
* Do not include rooms with an unknown room version in a sync response. (#10644)Patrick Cloke2021-08-191-2/+5
| | | | A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.
* Convert room member storage tuples to attrs. (#10629)Patrick Cloke2021-08-181-8/+10
| | | | Instead of using namedtuples. This helps with asserting type hints and code completion.
* Use auto-attribs for attrs classes for sync. (#10630)Patrick Cloke2021-08-181-78/+78
|
* Make sync response cache time configurable. (#10513)Richard van der Hoff2021-08-031-3/+11
|
* Always communicate device OTK counts to clients (#10485)Andrew Morgan2021-07-271-0/+4
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-161-15/+17
|
* Ensure that we do not cache empty sync responses after a timeout (#10158)Richard van der Hoff2021-06-171-10/+26
| | | Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
* Implement knock feature (#6739)Sorunome2021-06-091-20/+67
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Delete completes to-device messages earlier in /sync (#10124)Richard van der Hoff2021-06-071-10/+11
| | | | I hope this will improve https://github.com/matrix-org/synapse/issues/9564.
* Make /sync do less state res (#10102)Erik Johnston2021-06-021-2/+2
|
* Use get_current_users_in_room from store and not StateHandler (#9910)Erik Johnston2021-05-051-3/+3
|
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-221-2/+11
| | | This is no longer required, since we have dropped support for Python 3.5.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-4/+4
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-4/+6
|
* Improve tracing for to device messages (#9686)Erik Johnston2021-04-011-1/+17
|
* Add a type hints for service notices to the HomeServer object. (#9675)Patrick Cloke2021-03-241-2/+4
|
* Fix bad naming of storage function (#9637)Erik Johnston2021-03-171-2/+4
| | | | | | We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-1/+1
|
* Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke2021-03-021-2/+1
| | | | | | | This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
* Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong2021-02-241-1/+2
| | | | | | | This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-17/+20
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add additional type hints to the storage module. (#8980)Patrick Cloke2020-12-301-1/+1
|
* Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-1/+3
|
* Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-231-2/+2
|
* Send some ephemeral events to appservices (#8437)Will Hunt2020-10-151-1/+0
| | | Optionally sends typing, presence, and read receipt information to appservices.
* Add type hints to response cache. (#8507)Patrick Cloke2020-10-091-1/+3
|
* Fix returning incorrect prev_batch token in incremental sync (#8486)Erik Johnston2020-10-071-1/+6
|
* Add support for MSC2732: olm fallback keys (#8312)Hubert Chathi2020-10-061-0/+8
|
* Do not assume that account data is of the correct form. (#8454)Patrick Cloke2020-10-051-8/+11
| | | | This fixes a bug where `m.ignored_user_list` was assumed to be a dict, leading to odd behavior for users who set it to something else.
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-1/+1
|
* Various clean ups to room stream tokens. (#8423)Erik Johnston2020-09-291-6/+14
|
* Add EventStreamPosition type (#8388)Erik Johnston2020-09-241-5/+5
| | | | | | | | | | | | | | The idea is to remove some of the places we pass around `int`, where it can represent one of two things: 1. the position of an event in the stream; or 2. a token that partitions the stream, used as part of the stream tokens. The valid operations are then: 1. did a position happen before or after a token; 2. get all events that happened before or after a token; and 3. get all events between two tokens. (Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-24/+10
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)Erik Johnston2020-09-111-5/+6
|
* Add types to StreamToken and RoomStreamToken (#8279)Erik Johnston2020-09-081-3/+2
| | | The intention here is to change `StreamToken.room_key` to be a `RoomStreamToken` in a future PR, but that is a big enough change without this refactoring too.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-2/+2
|
* Fix typing for SyncHandler (#8237)Erik Johnston2020-09-031-5/+7
|
* Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-15/+18
|
* Define StateMap as immutable and add a MutableStateMap type. (#8183)Patrick Cloke2020-08-281-2/+3
|
* Revert #7736 (#8039)Brendan Abolivier2020-08-061-6/+0
|
* Convert streams to async. (#8014)Patrick Cloke2020-08-041-1/+1
|
* Re-implement unread counts (#7736)Brendan Abolivier2020-07-291-0/+6
|
* Fix up types and comments that refer to Deferreds. (#7945)Patrick Cloke2020-07-241-4/+0
|
* Add debugging to sync response generation (#7929)Richard van der Hoff2020-07-221-2/+15
|
* Back out MSC2625 implementation (#7761)Brendan Abolivier2020-07-011-3/+0
|
* Merge branch 'develop' into babolivier/mark_unreadBrendan Abolivier2020-06-151-18/+16
|\
| * Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-18/+16
| |
* | Appease mypyBrendan Abolivier2020-06-101-1/+1
| |
* | LintBrendan Abolivier2020-06-101-3/+3
| |
* | Use temporary prefixes as per the MSCBrendan Abolivier2020-06-101-1/+3
| |
* | Add a new unread_counter to sync responsesBrendan Abolivier2020-06-101-0/+1
|/
* Typo fixes.Patrick Cloke2020-06-051-1/+1
|
* Speed up fetching device lists changes in sync.Erik Johnston2020-05-051-4/+8
| | | | | Currently we copy `users_who_share_room` needlessly about three times, which is expensive when the set is large (which it can easily be).
* typosMatthew Hodgson2020-04-111-1/+1
|
* Add explanatory commentAndrew Morgan2020-03-301-0/+2
|
* Just add own user ID to the list we track device changes forDavid Baker2020-03-301-4/+6
|
* Always send the user updates to their own device listDavid Baker2020-03-271-3/+4
| | | | | This will allow clients to notify users about new devices even if the user isn't in any rooms (yet).
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-241-2/+2
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-12/+10
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Merge branch 'release-v1.10.0' into developRichard van der Hoff2020-02-111-0/+1
|\
| * filter out m.room.aliases from /sync state blocks (#6884)Matthew Hodgson2020-02-101-0/+1
| | | | | | We forgot to filter out aliases from /sync state blocks as well as the timeline.
* | Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-3/+3
| | | | | | | | | | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* | Add typing to SyncHandler (#6821)Erik Johnston2020-02-031-341/+364
|/ | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Fixup functions to consistently return deferredsErik Johnston2019-12-061-3/+3
|
* Port SyncHandler to async/awaitErik Johnston2019-12-051-134/+117
|
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-7/+9
| | | * update version of black and also fix the mypy config being overridden
* Port to use state storageErik Johnston2019-10-301-9/+11
|
* Merge branch 'develop' into cross-signing_keysHubert Chathi2019-09-041-5/+4
|\
| * Remove double return statements (#5962)Andrew Morgan2019-09-031-1/+0
| | | | | | | | | | | | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.
| * Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-4/+4
| | | | | | | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* | Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keysHubert Chathi2019-08-281-6/+25
|\|
| * Remove logging for #5407 and update commentsErik Johnston2019-08-151-17/+5
| |
| * Handle TimelineBatch being limited and empty.Erik Johnston2019-08-061-6/+37
| | | | | | | | | | | | This hopefully addresses #5407 by gracefully handling an empty but limited TimelineBatch. We also add some logging to figure out how this is happening.
* | Merge branch 'cross-signing_hidden' into cross-signing_keysHubert Chathi2019-08-011-61/+51
|\|
| * Replace returnValue with return (#5736)Amber Brown2019-07-231-61/+51
| |
* | allow uploading keys for cross-signingHubert Chathi2019-07-251-1/+6
|/
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Review commentsErik Johnston2019-06-271-4/+4
|
* Refactor and comment sync device list codeErik Johnston2019-06-261-17/+53
|
* Rename get_users_whose_devices_changedErik Johnston2019-06-261-1/+1
|
* Refactor get_user_ids_changed to pull less from DBErik Johnston2019-06-261-12/+10
| | | | | | | | | | | When a client asks for users whose devices have changed since a token we used to pull *all* users from the database since the token, which could easily be thousands of rows for old tokens. This PR changes this to only check for changes for users the client is actually interested in. Fixes #5553
* Run Black. (#5482)Amber Brown2019-06-201-343/+351
|
* Merge pull request #5089 from dnaf/m-heroes-empty-room-nameBrendan Abolivier2019-06-061-5/+4
|\ | | | | Make /sync return heroes if room name or canonical alias are empty
| * Simplify conditionBrendan Abolivier2019-06-061-2/+2
| |
| * Simplify conditionBrendan Abolivier2019-06-051-3/+2
| |
| * Merge branch 'develop' into m-heroes-empty-room-nameBrendan Abolivier2019-06-051-19/+25
| |\
| * | LintKatie Wolfe2019-04-241-1/+1
| | | | | | | | | | | | | | | | | | I probably should've just run autopep8 in the first place... Signed-off-by: Katie Wolfe <katie@dnaf.moe>
| * | Clean up codeKatie Wolfe2019-04-241-2/+3
| | | | | | | | | | | | Signed-off-by: Katie Wolfe <katie@dnaf.moe>
| * | Show heroes if room name or canonical alias are emptyKatie Wolfe2019-04-241-3/+4
| | | | | | | | | | | | | | | | | | Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>
| * | Show heroes if room name or canonical alias are emptyKatie Wolfe2019-04-241-5/+3
| | | | | | | | | | | | | | | | | | Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>
* | | Do user_id != me checks before deciding whether we should pick heroes from ↵Brendan Abolivier2019-06-051-15/+19
| |/ |/| | | | | the joined members or the parted ones
* | Make /sync attempt to return device updates for both joined and invited ↵Matthew Hodgson2019-05-161-19/+25
|/ | | | users (#3484)
* Fix grammar and document get_current_users_in_room (#4998)Andrew Morgan2019-04-031-4/+4
|
* Merge pull request #4815 from matrix-org/erikj/docstringsErik Johnston2019-03-061-7/+26
|\ | | | | Add docstrings from matrix-org-hotfixes
| * Add docstrings from matrix-org-hotfixesErik Johnston2019-03-061-7/+26
| |
* | Port #4422 debug logging from hotfixesErik Johnston2019-03-061-7/+46
|/
* Add some debug for membership syncing issues (#4538)Richard van der Hoff2019-01-311-2/+11
| | | I can't figure out what's going on with #4422 and #4436; perhaps this will help.
* Fix remote invite rejections not comming down syncErik Johnston2019-01-301-1/+13
| | | | | | | | | | This was broken in PR #4405, commit 886e5ac, where we changed remote rejections to be outliers. The fix is to explicitly add the leave event in when we know its an out of band invite. We can't always add the event as if the server is/was in the room there might be more events to send down the sync than just the leave.
* Fixed line lengthNathan Pennie2018-12-191-1/+5
|
* Fix the variable names used for account_dataNathan Pennie2018-12-191-3/+3
|
* Refactor state group lookup to reduce DB hits (#4011)Erik Johnston2018-10-251-54/+43
| | | | | | | | Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.
* Add description to counter metricErik Johnston2018-10-101-1/+5
|
* Track number of non-empty sync responses insteadErik Johnston2018-10-101-9/+15
|
* Append _total to metric and fix up spellingErik Johnston2018-10-101-2/+2
|
* Add metric to count lazy member sync requestsErik Johnston2018-10-091-6/+20
|
* Fix lazy loaded sync with rejected state eventsErik Johnston2018-10-011-2/+2
| | | | | | | | In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.
* Only lazy load self-members on initial syncMatthew Hodgson2018-09-251-5/+8
| | | Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
* Always LL ourselves if we're in a room to simplify clients (#3916)Matthew Hodgson2018-09-201-0/+4
| | | | Should fix https://github.com/vector-im/riot-web/issues/7209
* show heroes if a room has a 'deleted' name/canonical_alias (#3851)Matthew Hodgson2018-09-121-2/+15
|
* argh pep8Matthew Hodgson2018-09-121-1/+1
|
* correctly log gappy sync metricsMatthew Hodgson2018-09-121-1/+1
|
* fix loglineMatthew Hodgson2018-09-121-1/+1
|
* Speed up lazy loading (#3827)Matthew Hodgson2018-09-121-30/+88
| | | | | * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)
* Don't send empty tags list down syncErik Johnston2018-09-061-0/+5
|
* Send existing room tags down sync on joinErik Johnston2018-09-061-0/+8
| | | | | | | | When a user joined a room any existing tags were not sent down the sync stream. Ordinarily this isn't a problem because the user needs to be in the room to have set tags in it, however synapse will sometimes add tags for a user to a room, e.g. for server notices, which need to come down sync.
* Merge pull request #3795 from matrix-org/erikj/faster_sync_stateAmber Brown2018-09-071-8/+8
|\ | | | | User iter* during sync state calculations
| * User iter* during sync state calculationsErik Johnston2018-09-051-8/+8
| |
* | Port handlers/ to Python 3 (#3803)Amber Brown2018-09-071-3/+3
|/
* don't return non-LL-member state in incremental sync state blocks (#3760)Matthew Hodgson2018-08-281-1/+8
| | | don't return non-LL-member state in incremental sync state blocks
* Integrate presence from hotfixes (#3694)Amber Brown2018-08-181-1/+2
|
* initial cut at a room summary API (#3574)Matthew Hodgson2018-08-161-10/+149
|
* Merge pull request #3670 from matrix-org/neilj/mau_sync_blockNeil Johnson2018-08-141-2/+11
|\ | | | | Block ability to read via sync if mau limit exceeded
| * only block on sync where user is not part of the mau cohortNeil Johnson2018-08-091-1/+6
| |
| * block sync if auth checks failNeil Johnson2018-08-091-7/+5
| |
| * sync auth blockingNeil Johnson2018-08-081-5/+11
| |
* | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
|/
* Deduplicate redundant lazy-loaded members (#3331)Matthew Hodgson2018-07-261-25/+62
| | | | | | | | | | | | | | | | | | | | * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review
* clarify comment on p_idsMatthew Hodgson2018-07-241-1/+2
|
* rather than adding ll_ids, remove them from p_idsMatthew Hodgson2018-07-241-5/+6
|
* incorporate more reviewMatthew Hodgson2018-07-241-3/+9
|
* incorporate reviewMatthew Hodgson2018-07-231-40/+27
|
* incorporate reviewMatthew Hodgson2018-07-191-17/+27
|
* add a filtered_types param to limit filtering to specific typesMatthew Hodgson2018-07-191-29/+36
|
* merge develop pydoc for _get_state_for_groupsMatthew Hodgson2018-07-191-13/+13
|\
| * run isortAmber Brown2018-07-091-12/+12
| |
| * typoMatthew Hodgson2018-06-251-1/+1
| |
* | Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-06-041-4/+14
|\|
| * Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-311-3/+3
| | | | | | | | they're not meant to be lazy (#3307)
| * Update some comments and docstrings in SyncHandlerRichard van der Hoff2018-05-291-1/+11
| |
* | more commentsMatthew Hodgson2018-06-041-0/+7
| |
* | add pydocMatthew Hodgson2018-05-291-5/+13
| |
* | merge developMatthew Hodgson2018-05-291-24/+41
|\|
| * replace some iteritems with sixAdrian Tschira2018-05-191-6/+8
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * Add comment to sync as to why code path is splitErik Johnston2018-05-091-0/+5
| |
| * Refactor sync APIs to reuse pagination APIErik Johnston2018-05-091-6/+13
| | | | | | | | | | | | The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.
| * Refactor get_recent_events_for_room return typeErik Johnston2018-05-091-1/+1
| | | | | | | | | | | | There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs
| * Merge pull request #3107 from NotAFile/py3-bool-nonzeroRichard van der Hoff2018-04-201-0/+7
| |\ | | | | | | add __bool__ alias to __nonzero__ methods
| | * add __bool__ alias to __nonzero__ methodsAdrian Tschira2018-04-151-0/+7
| | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | Refactor ResponseCache usageRichard van der Hoff2018-04-121-10/+6
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.
| * Add metrics for ResponseCacheRichard van der Hoff2018-04-101-1/+1
| |
* | make lazy_load_members configurable in filtersMatthew Hodgson2018-03-191-9/+9
| |
* | reshuffle todo & commentsMatthew Hodgson2018-03-191-2/+10
| |
* | only get member state IDs for incremental syncs if we're filteringMatthew Hodgson2018-03-181-5/+5
| |
* | remove debugMatthew Hodgson2018-03-161-4/+0
| |
* | fix tsm search againMatthew Hodgson2018-03-161-5/+4
| |
* | add state_ids for timeline entriesMatthew Hodgson2018-03-161-4/+14
| |
* | simplify timeline_start_membersMatthew Hodgson2018-03-161-14/+13
| |
* | make incr syncs workMatthew Hodgson2018-03-161-3/+4
| |
* | add copyright to nudge CIMatthew Hodgson2018-03-141-1/+2
| |
* | oopsMatthew Hodgson2018-03-131-1/+1
| |
* | make it workMatthew Hodgson2018-03-131-3/+3
| |
* | ensure we always include the members for a given timeline blockMatthew Hodgson2018-03-131-4/+32
| |
* | typosMatthew Hodgson2018-03-111-2/+2
| |
* | WIP experiment in lazyloading room membersMatthew Hodgson2018-03-111-12/+31
|/
* Fix up log messageErik Johnston2018-03-071-1/+1
|
* Fix typoErik Johnston2018-03-071-2/+2
|
* Fix race in sync when joining roomErik Johnston2018-03-071-28/+75
| | | | | | | | | | | The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.
* Remove ability for AS users to call /events and /syncErik Johnston2018-03-051-4/+6
| | | | | | | This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144
* Do logcontexts outside ResponseCacheErik Johnston2017-10-251-3/+3
|
* Merge branch 'develop' into erikj/groups_mergedDavid Baker2017-10-021-7/+32
|\
| * Refactor to speed up incremental syncsErik Johnston2017-09-251-5/+18
| |
| * Fix device list when rejoining room (#2461)Erik Johnston2017-09-221-2/+14
| |
* | Ensure that creator of group sees group down /syncErik Johnston2017-09-191-0/+1
| |
* | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_mergedErik Johnston2017-09-191-21/+105
|\|
| * Don't filter out current state events from timelineErik Johnston2017-09-181-0/+7
| |
| * Actually hook leave notifs upErik Johnston2017-09-141-2/+3
| |
| * Get left roomsErik Johnston2017-09-131-3/+32
| |
| * In sync handle device lists for newly joined/left roomsErik Johnston2017-09-121-2/+10
| |
| * Add left section to /keys/changesErik Johnston2017-09-081-1/+1
| |
| * Send down device list change notif when member leaves/rejoins roomErik Johnston2017-09-071-12/+52
| |
| * Use less DB for device list handling in syncErik Johnston2017-07-121-8/+7
| |
* | Correctly add joins to correct segmentErik Johnston2017-07-241-1/+1
| |