summary refs log tree commit diff
path: root/synapse/storage/databases/main/stream.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Bump ruff from 0.0.252 to 0.0.259 (#15328)dependabot[bot]2023-03-281-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Bump ruff from 0.0.252 to 0.0.259 Bumps [ruff](https://github.com/charliermarsh/ruff) from 0.0.252 to 0.0.259. - [Release notes](https://github.com/charliermarsh/ruff/releases) - [Changelog](https://github.com/charliermarsh/ruff/blob/main/BREAKING_CHANGES.md) - [Commits](https://github.com/charliermarsh/ruff/compare/v0.0.252...v0.0.259) --- updated-dependencies: - dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fix new warnings * Mypy * Newsfile --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Use immutabledict instead of frozendict (#15113)David Robertson2023-03-221-2/+2
| | | | | | | | | | Additionally: * Consistently use `freeze()` in test --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: 6543 <6543@obermui.de>
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-221-0/+1
|
* Use an enum for direction. (#14927)Patrick Cloke2023-01-271-28/+31
| | | | For better type safety we use an enum instead of strings to configure direction (backwards or forwards).
* Fix paginating /relations with a live token (#14866)Patrick Cloke2023-01-261-49/+105
| | | | | The `/relations` endpoint was not properly handle "live tokens" (i.e sync tokens), to do this properly we abstract the code that `/messages` has and re-use it.
* Request partial joins by default (#14905)David Robertson2023-01-241-6/+34
| | | | | | | | | | | | | | | | | | | | | | | | | * Request partial joins by default This is a little sloppy, but we are trying to gain confidence in faster joins in the upcoming RC. Admins can still opt out by adding the following to their Synapse config: ```yaml experimental: faster_joins: false ``` We may revert this change before the release proper, depending on how testing in the wild goes. * Changelog * Try to fix the backfill test failures * Upgrade notes * Postgres compat?
* Fix race calling `/members?at=` (#14817)Erik Johnston2023-01-121-6/+59
| | | Fixes #14814
* Merge/remove `Slaved*` stores into `WorkerStores` (#14375)Nick Mills-Barrett2022-11-111-0/+1
|
* Support filtering the /messages API by relation type (MSC3874). (#14148)Patrick Cloke2022-10-171-2/+27
| | | Gated behind an experimental configuration flag.
* Do not allow a None-limit on PaginationConfig. (#14146)Patrick Cloke2022-10-141-2/+0
| | | | | | | The callers either set a default limit or manually handle a None-limit later on (by setting a default value). Update the callers to always instantiate PaginationConfig with a default limit and then assume the limit is non-None.
* Fix a bug where redactions were not being sent over federation if we did not ↵Shay2022-10-111-15/+13
| | | | have the original event. (#13813)
* Properly paginate forward in the /relations API. (#13840)Patrick Cloke2022-09-221-3/+3
| | | | | This fixes a bug where the `/relations` API with `dir=f` would skip the first item of each page (except the first page), causing incomplete data to be returned to the client.
* Cancel the processing of key query requests when they time out. (#13680)reivilibre2022-09-071-0/+2
|
* Instrument `/messages` for understandable traces in Jaeger (#13368)Eric Eastwood2022-08-031-0/+2
| | | | | | In Jaeger: - Before: huge list of uncategorized database calls - After: nice and collapsible into units of work
* Federation Sender & Appservice Pusher Stream Optimisations (#13251)Nick Mills-Barrett2022-07-151-12/+20
| | | | | | | | | | | | | * Replace `get_new_events_for_appservice` with `get_all_new_events_stream` The functions were near identical and this brings the AS worker closer to the way federation senders work which can allow for multiple workers to handle AS traffic. * Pull received TS alongside events when processing the stream This avoids an extra query -per event- when both federation sender and appservice pusher process events.
* Improve performance of getting unread counts in rooms (#13119)Erik Johnston2022-06-291-0/+20
|
* Fix 404 on `/sync` when the last event is a redaction of an unknown/purged ↵Richard van der Hoff2022-06-011-7/+5
| | | | | | | | | | | event (#12905) Currently, we try to pull the event corresponding to a sync token from the database. However, when we fetch redaction events, we check the target of that redaction (because we aren't allowed to send redactions to clients without validating them). So, if the sync token points to a redaction of an event that we don't have, we have a problem. It turns out we don't really need that event, and can just work with its ID and metadata, which sidesteps the whole problem.
* Merge branch 'master' into developDavid Robertson2022-05-171-1/+1
|\
| * Fix query performance for /sync (#12745)Erik Johnston2022-05-161-1/+1
| |
* | Merge tag 'v1.59.0rc2' into developDavid Robertson2022-05-161-8/+11
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.59.0rc2 (2022-05-16) ============================== Synapse 1.59 makes several changes that server administrators should be aware of: - Device name lookup over federation is now disabled by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616)) - The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](https://github.com/matrix-org/synapse/issues/12452), [\#12654](https://github.com/matrix-org/synapse/issues/12654)) See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details. Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](https://github.com/matrix-org/synapse/issues/12597)) Bugfixes -------- - Fix a bug introduced in Synapse 1.58.0 where `/sync` would fail if the most recent event in a room was rejected. ([\#12729](https://github.com/matrix-org/synapse/issues/12729))
| * Fix bug /sync returning 404 (#12729)Erik Johnston2022-05-161-8/+11
| | | | | | | | | | * Fix bug /sync returning 404 Fixes #12571
* | Reduce the number of "untyped defs" (#12716)David Robertson2022-05-121-3/+5
| |
* | Fix `/messages` throwing a 500 when querying for non-existent room (#12683)Eric Eastwood2022-05-101-15/+11
|/ | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/12678 Complement test added: https://github.com/matrix-org/complement/pull/369 **Before:** 500 internal server error **After:** According to the [spec](https://spec.matrix.org/latest/client-server-api/#get_matrixclientv3roomsroomidmessages), calling `/messages` against a non-existent `room_id` should throw a 403 forbidden (since you're not part of the room). This also matches the behavior before https://github.com/matrix-org/synapse/pull/12370 which regressed Synapse to the 500 behavior. ```json { "errcode": "M_FORBIDDEN", "error": "User @test:my.synapse.server not in room !dne:my.synapse.server, and room previews are disabled" } ```
* Fix missing sync events during historical batch imports (#12319)Nick Mills-Barrett2022-04-131-0/+26
| | | | | | | | Discovered after much in-depth investigation in #12281. Closes: #12281 Closes: #3305 Signed off by: Nick Mills-Barrett nick@beeper.com
* Add some type hints to datastore (#12423)Dirk Klimpel2022-04-121-6/+18
| | | | | | | | | | | | | | | | | | | * Add some type hints to datastore * newsfile * change `Collection` to `List` * refactor return type of `select_users_txn` * correct type hint in `stream.py` * Remove `Optional` in `select_users_txn` * remove not needed return type in `__init__` * Revert change in `get_stream_id_for_event_txn` * Remove import from `Literal`
* Generate historic pagination token for `/messages` when no `?from` token ↵Eric Eastwood2022-04-061-9/+11
| | | | provided (#12370)
* Add a configuration to exclude rooms from sync response (#12310)Brendan Abolivier2022-03-301-10/+20
|
* Support stable identifiers for MSC3440: Threading (#12151)Patrick Cloke2022-03-101-8/+10
| | | | The unstable identifiers are still supported if the experimental configuration flag is enabled. The unstable identifiers will be removed in a future release.
* Improvements to bundling aggregations. (#11815)Patrick Cloke2022-01-261-7/+15
| | | | | | | | | | | This is some odds and ends found during the review of #11791 and while continuing to work in this code: * Return attrs classes instead of dictionaries from some methods to improve type safety. * Call `get_bundled_aggregations` fewer times. * Adds a missing assertion in the tests. * Do not return empty bundled aggregations for an event (preferring to not include the bundle at all, as the docstring states).
* Remove redundant `get_current_events_token` (#11643)Richard van der Hoff2022-01-041-19/+15
| | | | | | | | | | | | | | | | | * Push `get_room_{min,max_stream_ordering}` into StreamStore Both implementations of this are identical, so we may as well push it down and get rid of the abstract base class nonsense. * Remove redundant `StreamStore` class This is empty now * Remove redundant `get_current_events_token` This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid of it. * newsfile
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-5/+7
| | | To improve type hints throughout the code.
* Type hint the constructors of the data store classes (#11555)Sean Quah2021-12-131-1/+7
|
* Comments on the /sync tentacles (#11494)David Robertson2021-12-021-4/+11
| | | | | This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-22/+64
| | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-231-2/+2
|
* Fix /initialSync error due to unhashable `RoomStreamToken` (#10827)Sean Quah2021-09-221-1/+3
| | | | | | | | The deprecated /initialSync endpoint maintains a cache of responses, using parameter values as part of the cache key. When a `from` or `to` parameter is specified, it gets converted into a `StreamToken`, which contains a `RoomStreamToken` and forms part of the cache key. `RoomStreamToken`s need to be made hashable for this to work.
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-151-4/+2
|
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-221-2/+2
| | | This is no longer required, since we have dropped support for Python 3.5.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-17/+25
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Use vector clocks for room stream tokens. (#8439)Erik Johnston2020-10-071-37/+243
| | | | | | | | | | Currently when using multiple event persisters we (in the worst case) don't tell clients about events until all event persisters have persisted new events after the original event. This is a suboptimal, especially if one of the event persisters goes down. To handle this, we encode the position of each event persister in the room tokens so that we can send events to clients immediately. To reduce the size of the token we do two things: 1. We create a unique immutable persistent mapping between instance names and a generated small integer ID, which we can encode in the tokens instead of the instance name; and 2. We encode the "persisted upto position" of the room token and then only explicitly include instances that have positions strictly greater than that. The new tokens look something like: `m3478~1.3488~2.3489`, where the first number is the min position, and the subsequent `-` separated pairs are the instance ID to positions map. (We use `.` and `~` as separators as they're URL safe and not already used by `StreamToken`).
* Remove stream ordering from Metadata dict (#8452)Richard van der Hoff2020-10-051-13/+0
| | | | | | | | There's no need for it to be in the dict as well as the events table. Instead, we store it in a separate attribute in the EventInternalMetadata object, and populate that on load. This means that we can rely on it being correctly populated for any event which has been persited to the database.
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-1/+1
|
* Various clean ups to room stream tokens. (#8423)Erik Johnston2020-09-291-17/+21
|
* Merge tag 'v1.20.0rc5' into developPatrick Cloke2020-09-181-8/+5
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.20.0rc5 (2020-09-18) ============================== In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3. Features -------- - Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343)) Bugfixes -------- - Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342)) - Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349)) Internal Changes ---------------- - Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
| * Intelligently select extremities used in backfill. (#8349)Erik Johnston2020-09-181-8/+5
| | | | | | | | | | | | | | | | | | Instead of just using the most recent extremities let's pick the ones that will give us results that the pagination request cares about, i.e. pick extremities only if they have a smaller depth than the pagination token. This is useful when we fail to backfill an extremity, as we no longer get stuck requesting that same extremity repeatedly.
* | Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | | | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* | Switch metaclass initialization to python 3-compatible syntax (#8326)Jonathan de Jong2020-09-161-3/+1
| |
* | Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)Erik Johnston2020-09-111-43/+32
| |
* | Add types to StreamToken and RoomStreamToken (#8279)Erik Johnston2020-09-081-10/+11
|/ | | The intention here is to change `StreamToken.room_key` to be a `RoomStreamToken` in a future PR, but that is a big enough change without this refactoring too.
* Add StreamStore to mypy (#8232)Erik Johnston2020-09-021-18/+28
|
* Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-3/+18
|
* Convert state and stream stores and related code to async (#8194)Patrick Cloke2020-08-281-4/+7
|
* Convert events worker database to async/await. (#8071)Patrick Cloke2020-08-181-1/+0
|
* Fix the return type of send_nonmember_events. (#8112)Patrick Cloke2020-08-181-4/+15
|
* Convert stream database to async/await. (#8074)Patrick Cloke2020-08-171-187/+200
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-0/+1064