summary refs log tree commit diff
path: root/synapse/handlers/pagination.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Avoid backfill when we already have messages to return (#15737)Eric Eastwood2023-06-131-22/+115
| | | | | We now only block the client to backfill when we see a large gap in the events (more than 2 events missing in a row according to `depth`), more than 3 single-event holes, or not enough messages to fill the response. Otherwise, we return the messages directly to the client and backfill in the background for eventual consistency sake. Fix https://github.com/matrix-org/synapse/issues/15696
* Add stricter mypy options (#15694)Patrick Cloke2023-05-311-2/+2
| | | | Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any, and disallow_incomplete_defs.
* Include the room ID in more purge room log lines. (#15222)Patrick Cloke2023-03-081-1/+1
|
* Pass the requester during event serialization. (#15174)Quentin Gliech2023-03-061-1/+3
| | | | This allows Synapse to properly include the transaction ID in the unsigned data of events.
* Use an enum for direction. (#14927)Patrick Cloke2023-01-271-3/+3
| | | | For better type safety we use an enum instead of strings to configure direction (backwards or forwards).
* Use StrCollection in place of Collection[str] in (most) handlers code. (#14922)Patrick Cloke2023-01-261-3/+3
| | | | Due to the increased safety of StrCollection over Collection[str] and Sequence[str].
* Move `StateFilter` to `synapse.types` (#14668)David Robertson2022-12-121-1/+1
| | | | | * Move `StateFilter` to `synapse.types` * Changelog
* Implement message forward pagination from start when no from is given, fixes ↵Benjamin Kampmann2022-11-241-0/+6
| | | | | #12383 (#14149) Fixes https://github.com/matrix-org/synapse/issues/12383
* Do not allow a None-limit on PaginationConfig. (#14146)Patrick Cloke2022-10-141-5/+0
| | | | | | | The callers either set a default limit or manually handle a None-limit later on (by setting a default value). Update the callers to always instantiate PaginationConfig with a default limit and then assume the limit is non-None.
* Add Admin API to Fetch Messages Within a Particular Window (#13672)Connor Davis2022-09-071-13/+24
| | | This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
* Move the execution of the retention purge_jobs to the main worker (#13632)Brad Murray2022-08-261-4/+2
| | | | | Fixes #9927 Signed-off-by: Brad Murray brad@beeper.com
* `synapse.api.auth.Auth` cleanup: make permission-related methods use ↵Quentin Gliech2022-08-221-1/+1
| | | | | | | | | `Requester` instead of the `UserID` (#13024) Part of #13019 This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it. It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
* Instrument `/messages` for understandable traces in Jaeger (#13368)Eric Eastwood2022-08-031-0/+2
| | | | | | In Jaeger: - Before: huge list of uncategorized database calls - After: nice and collapsible into units of work
* Rename storage classes (#12913)Erik Johnston2022-05-311-7/+10
|
* Don't return `end` from `/messages` if there are no more events (#12903)Jacek Kuśnierz2022-05-301-6/+17
| | | | Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de>
* Fix up `state_store` naming (#12871)Erik Johnston2022-05-251-2/+2
|
* Prevent expired events from being filtered out when retention is disabled ↵Brendan Abolivier2022-05-231-1/+1
| | | | | | (#12611) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add `StreamKeyType` class and replace string literals with constants (#12567)Andrew Morgan2022-05-161-3/+3
|
* Fix `/messages` throwing a 500 when querying for non-existent room (#12683)Eric Eastwood2022-05-101-1/+1
| | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/12678 Complement test added: https://github.com/matrix-org/complement/pull/369 **Before:** 500 internal server error **After:** According to the [spec](https://spec.matrix.org/latest/client-server-api/#get_matrixclientv3roomsroomidmessages), calling `/messages` against a non-existent `room_id` should throw a 403 forbidden (since you're not part of the room). This also matches the behavior before https://github.com/matrix-org/synapse/pull/12370 which regressed Synapse to the 500 behavior. ```json { "errcode": "M_FORBIDDEN", "error": "User @test:my.synapse.server not in room !dne:my.synapse.server, and room previews are disabled" } ```
* Generate historic pagination token for `/messages` when no `?from` token ↵Eric Eastwood2022-04-061-1/+8
| | | | provided (#12370)
* Move get_bundled_aggregations to relations handler. (#12237)Patrick Cloke2022-03-181-1/+4
| | | | | The get_bundled_aggregations code is fairly high-level and uses a lot of store methods, we move it into the handler as that seems like a better fit.
* Add a relations handler to avoid duplication. (#12227)Patrick Cloke2022-03-161-2/+3
| | | Adds a handler layer between the REST and datastore layers for relations.
* Add cancellation support to `ReadWriteLock` (#12120)Sean Quah2022-03-141-4/+4
| | | | | | Also convert `ReadWriteLock` to use async context managers. Signed-off-by: Sean Quah <seanq@element.io>
* Use the proper serialization format when bundling aggregations. (#12090)Patrick Cloke2022-03-031-2/+5
| | | | This ensures that the `latest_event` field of the bundled aggregation for threads uses the same format as the other events in the response.
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke2022-01-181-1/+1
| | | | | | Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
* Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke2022-01-071-3/+5
| | | | | | | | This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
* Do not bundle aggregations for APIs which shouldn't include them. (#11592)Patrick Cloke2021-12-201-1/+4
| | | | | And make bundling aggregations opt-in, instead of opt-out to avoid having APIs to include extraneous data (and being much heavier than necessary).
* Fix 'delete room' admin api to work on incomplete rooms (#11523)Richard van der Hoff2021-12-071-3/+0
| | | | | If, for some reason, we don't have the create event, we should still be able to purge a room.
* Convert delete room admin API to async endpoint (#11223)Dirk Klimpel2021-11-121-7/+282
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-1/+1
| | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Refactor `Filter` to handle fields according to data being filtered. (#11194)Patrick Cloke2021-10-271-1/+1
| | | | This avoids filtering against fields which cannot exist on an event source. E.g. presence updates don't have a room.
* Move experimental & retention config out of the server module. (#11070)Patrick Cloke2021-10-151-5/+8
|
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-4/+10
|
* Clean-up type hints in server config (#10915)Patrick Cloke2021-09-281-4/+4
| | | | | | By using attrs instead of dicts to store configuration. Also updates some of the attrs classes to use proper type hints and auto_attribs.
* Require type hints in the handlers module. (#10831)Patrick Cloke2021-09-201-10/+9
| | | | | | | Adds missing type hints to methods in the synapse.handlers module and requires all methods to have type hints there. This also removes the unused construct_auth_difference method from the FederationHandler.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-1/+1
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-161-2/+2
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-1/+1
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-3/+11
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add `force_purge` option to delete-room admin api. (#8843)Richard van der Hoff2020-11-301-6/+11
|
* Move additional tasks to the background worker, part 4 (#8513)Patrick Cloke2020-10-131-1/+1
|
* Remove the deprecated Handlers object (#8494)Patrick Cloke2020-10-091-1/+1
| | | All handlers now available via get_*_handler() methods on the HomeServer.
* Make token serializing/deserializing async (#8427)Erik Johnston2020-09-301-4/+4
| | | The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
* Various clean ups to room stream tokens. (#8423)Erik Johnston2020-09-291-3/+2
|
* Fix a bad merge from release-v1.20.0. (#8354)Patrick Cloke2020-09-181-1/+1
|
* Merge tag 'v1.20.0rc5' into developPatrick Cloke2020-09-181-4/+4
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.20.0rc5 (2020-09-18) ============================== In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3. Features -------- - Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343)) Bugfixes -------- - Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342)) - Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349)) Internal Changes ---------------- - Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
| * Intelligently select extremities used in backfill. (#8349)Erik Johnston2020-09-181-4/+4
| | | | | | | | | | | | | | | | | | Instead of just using the most recent extremities let's pick the ones that will give us results that the pagination request cares about, i.e. pick extremities only if they have a smaller depth than the pagination token. This is useful when we fail to backfill an extremity, as we no longer get stuck requesting that same extremity repeatedly.
* | Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)Erik Johnston2020-09-111-2/+2
| |
* | Fix mypy error on develop (#8282)Erik Johnston2020-09-081-3/+6
| |
* | Clean up types for PaginationConfig (#8250)Erik Johnston2020-09-081-20/+22
|/ | | This removes `SourcePaginationConfig` and `get_pagination_rows`. The reasoning behind this is that these generic classes/functions erased the types of the IDs it used (i.e. instead of passing around `StreamToken` it'd pass in e.g. `token.room_key`, which don't have uniform types).
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-2/+2
|
* Add type hints to more handlers (#8244)Erik Johnston2020-09-031-24/+32
|
* Convert ReadWriteLock to async/await. (#8202)Patrick Cloke2020-08-281-23/+26
|
* Allow capping a room's retention policy (#8104)Brendan Abolivier2020-08-241-7/+29
|
* Convert streams to async. (#8014)Patrick Cloke2020-08-041-1/+1
|
* Convert the device message and pagination handlers to async/await. (#7678)Patrick Cloke2020-06-161-16/+8
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-3/+1
|
* Merge pull request #6949 from matrix-org/rav/list_room_aliases_peekableRichard van der Hoff2020-02-191-1/+3
|\ | | | | Make room alias lists peekable
| * Add `allow_departed_users` param to `check_in_room_or_world_readable`Richard van der Hoff2020-02-191-1/+3
| | | | | | | | | | | | | | ... and set it everywhere it's called. while we're here, rename it for consistency with `check_user_in_room` (and to help check that I haven't missed any instances)
* | Fix log in message retention purge jobsBrendan Abolivier2020-02-181-1/+1
|/
* s/get_room_version/get_room_version_id/Richard van der Hoff2020-01-311-1/+1
| | | | | ... to make way for a forthcoming get_room_version which returns a RoomVersion object.
* Add more logging around message retention policies support (#6717)Brendan Abolivier2020-01-171-0/+13
| | | So we can debug issues like #6683 more easily
* Fix instantiation of message retention purge jobsBrendan Abolivier2020-01-151-1/+1
| | | | | | | | | | | | | | | | When figuring out which topological token to start a purge job at, we need to do the following: 1. Figure out a timestamp before which events will be purged 2. Select the first stream ordering after that timestamp 3. Select info about the first event after that stream ordering 4. Build a topological token from that info In some situations (e.g. quiet rooms with a short max_lifetime), there might not be an event after the stream ordering at step 3, therefore we abort the purge with the error `No event found`. To mitigate that, this patch fetches the first event _before_ the stream ordering, instead of after.
* Convert federation backfill to asyncRichard van der Hoff2019-12-111-14/+13
| | | | | | | | | PaginationHandler.get_messages is only called by RoomMessageListRestServlet, which is async. Chase the code path down from there: - FederationHandler.maybe_backfill (and nested try_backfill) - FederationHandler.backfill
* Merge branch 'develop' into babolivier/message_retentionBrendan Abolivier2019-11-261-2/+4
|\
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-11-041-7/+6
| |\ | | | | | | | | | erikj/split_purge_history
| * \ Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-311-2/+3
| |\ \ | | | | | | | | | | | | erikj/split_purge_history
| * | | Split purge API into events vs stateErik Johnston2019-10-301-2/+5
| | | |
* | | | LintBrendan Abolivier2019-11-191-11/+6
| | | |
* | | | Implement per-room message retention policiesBrendan Abolivier2019-11-041-0/+111
| |_|/ |/| |
* | | Update black to 19.10b0 (#6304)Amber Brown2019-11-011-7/+6
| |/ |/| | | * update version of black and also fix the mypy config being overridden
* | Port to use state storageErik Johnston2019-10-301-2/+4
|/
* Servlet to purge old rooms (#5845)Richard van der Hoff2019-08-221-0/+17
|
* Replace returnValue with return (#5736)Amber Brown2019-07-231-8/+6
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Fix /messages on workers when no from param specified.Erik Johnston2019-06-241-3/+1
| | | | | | | If no `from` param is specified we calculate and use the "current token" that inlcuded typing, presence, etc. These are unused during pagination and are not available on workers, so we simply don't calculate them.
* Run Black. (#5482)Amber Brown2019-06-201-38/+30
|
* Allow client event serialization to be asyncErik Johnston2019-05-141-9/+13
|
* Log tracebacks correctlyErik Johnston2019-02-251-1/+5
|
* Merge pull request #4263 from rkfg/developErik Johnston2019-02-211-1/+1
|\ | | | | Prevent crash on pagination.
| * Prevent crash on pagination.rkfg2018-12-061-1/+1
| |
* | fix NPE in /messages by checking if all events were filtered out (#4330)Matthew Hodgson2019-01-021-10/+11
|/
* Refactor state group lookup to reduce DB hits (#4011)Erik Johnston2018-10-251-8/+7
| | | | | | | | Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.
* don't filter membership events based on history visibility (#3874)Matthew Hodgson2018-09-141-8/+1
| | | | | | | | don't filter membership events based on history visibility as we will already have filtered the messages in the timeline, and state events are always visible. and because @erikjohnston said so.
* lazyload aware /messages (#3589)Matthew Hodgson2018-08-161-1/+34
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
|
* Move PaginationHandler to its own fileErik Johnston2018-07-201-0/+265