summary refs log tree commit diff
path: root/changelog.d (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-07-07Bg update to populate new `events` table columnsRichard van der Hoff3-0/+147
These columns were added back in Synapse 1.52, and have been populated for new events since then. It's now (beyond) time to back-populate them for existing events.
2022-07-07Add --build-only option to complement.sh to prevent actually running ↵reivilibre2-3/+19
Complement. (#13158)
2022-07-07Remove obsolete RoomEventsStoreTestCase (#13200)Petr Vaněk2-69/+1
All tests are prefixed with `STALE_` and therefore they are silently skipped. They were moved to `STALE_` in version `v0.5.0` in commit 2fcce3b3c508 - `Remove stale tests`. Tests from `RoomEventsStoreTestCase` class are not used for last 8 years, I believe the best would be to remove them entirely. Signed-off-by: Petr Vaněk <arkamar@atlas.cz>
2022-07-07Faster room joins: fix race in recalculation of current room state (#13151)Sean Quah8-55/+214
Bounce recalculation of current state to the correct event persister and move recalculation of current state into the event persistence queue, to avoid concurrent updates to a room's current state. Also give recalculation of a room's current state a real stream ordering. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-07Use a single query in `ProfileHandler.get_profile` (#13209)Nick Mills-Barrett2-12/+8
2022-07-07Bump lxml from 4.8.0 to 4.9.1 (#13207)dependabot[bot]2-62/+72
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Robertson <davidr@element.io>
2022-07-07Check that `auto_vacuum` is disabled when porting a SQLite database to ↵reivilibre3-0/+43
Postgres, as `VACUUM`s must not be performed between runs of the script. (#13195)
2022-07-07Make `_get_state_map_for_room` not break when room state events don't ↵David Teller2-8/+2
contain an event id. (#13174) Method `_get_state_map_for_room` seems to break in presence of some ill-formed events in the database. Reimplementing this method to use `get_current_state`, which is more robust to such events.
2022-07-06Add information on how the Synapse team does reviews. (#13132)Patrick Cloke4-1/+47
2022-07-06Fix bug where we failed to delete old push actions (#13194)Erik Johnston2-2/+5
This happened if we encountered a stream ordering in `event_push_actions` that had more rows than the batch size of the delete, as If we don't delete any rows in an iteration then the next time round we get the exact same stream ordering and get stuck.
2022-07-05Handle race between persisting an event and un-partial stating a room (#13100)Sean Quah10-74/+234
Whenever we want to persist an event, we first compute an event context, which includes the state at the event and a flag indicating whether the state is partial. After a lot of processing, we finally try to store the event in the database, which can fail for partial state events when the containing room has been un-partial stated in the meantime. We detect the race as a foreign key constraint failure in the data store layer and turn it into a special `PartialStateConflictError` exception, which makes its way up to the method in which we computed the event context. To make things difficult, the exception needs to cross a replication request: `/fed_send_events` for events coming over federation and `/send_event` for events from clients. We transport the `PartialStateConflictError` as a `409 Conflict` over replication and turn `409`s back into `PartialStateConflictError`s on the worker making the request. All client events go through `EventCreationHandler.handle_new_client_event`, which is called in *a lot* of places. Instead of trying to update all the code which creates client events, we turn the `PartialStateConflictError` into a `429 Too Many Requests` in `EventCreationHandler.handle_new_client_event` and hope that clients take it as a hint to retry their request. On the federation event side, there are 7 places which compute event contexts. 4 of them use outlier event contexts: `FederationEventHandler._auth_and_persist_outliers_inner`, `FederationHandler.do_knock`, `FederationHandler.on_invite_request` and `FederationHandler.do_remotely_reject_invite`. These events won't have the partial state flag, so we do not need to do anything for then. The remaining 3 paths which create events are `FederationEventHandler.process_remote_join`, `FederationEventHandler.on_send_membership_event` and `FederationEventHandler._process_received_pdu`. We can't experience the race in `process_remote_join`, unless we're handling an additional join into a partial state room, which currently blocks, so we make no attempt to handle it correctly. `on_send_membership_event` is only called by `FederationServer._on_send_membership_event`, so we catch the `PartialStateConflictError` there and retry just once. `_process_received_pdu` is called by `on_receive_pdu` for incoming events and `_process_pulled_event` for backfill. The latter should never try to persist partial state events, so we ignore it. We catch the `PartialStateConflictError` in `on_receive_pdu` and retry just once. Refering to the graph of code paths in https://github.com/matrix-org/synapse/issues/12988#issuecomment-1156857648 may make the above make more sense. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-05Type `tests.utils` (#13028)David Robertson5-46/+102
* Cast to postgres types when handling postgres db * Remove unused method * Easy annotations * Annotate create_room * Use `ParamSpec` to annotate looping_call * Annotate `default_config` * Track `now` as a float `time_ms` returns an int like the proper Synapse `Clock` * Introduce a `Timer` dataclass * Introduce a Looper type * Suppress checking of a mock * tests.utils is typed * Changelog * Whoops, import ParamSpec from typing_extensions * ditch the psycopg2 casts
2022-07-05Factor out some common Complement CI setup commands to a script. (#13157)reivilibre4-47/+42
2022-07-05Use upserts for updating `event_push_summary` (#13153)Erik Johnston2-40/+8
2022-07-05Fix application service not being able to join remote federated room without ↵Eric Eastwood2-9/+24
a profile set (#13131) Fix https://github.com/matrix-org/synapse/issues/4778 Complement tests: https://github.com/matrix-org/complement/pull/399
2022-07-05Mention the spamchecker plugins v1.62.0David Robertson1-0/+2
2022-07-051.62.0David Robertson3-1/+13
2022-07-05Add the ability to set the log level using the `SYNAPSE_TEST_LOG_LEVEL` ↵reivilibre6-6/+44
environment when using `complement.sh`. (#13152)
2022-07-05Add missing links to config options (#13166)Dirk Klimpel2-3/+4
2022-07-04annotate tests.server.FakeChannel (#13136)David Robertson7-26/+36
2022-07-04Revert "Up the dependency on canonicaljson to ^1.5.0"Brendan Abolivier1-3/+1
This reverts commit dcc4e0621cc101271efc573600bd7591a12cea7c.
2022-07-04Up the dependency on canonicaljson to ^1.5.0Brendan Abolivier1-1/+3
2022-07-04Update changelog for v1.62.0rc2 v1.62.0rc3Andrew Morgan1-2/+2
2022-07-041.62.0rc3Andrew Morgan5-3/+17
2022-07-04Fix stuck notification counts on small servers (#13168)Erik Johnston3-7/+13
2022-07-04Remove tests/utils.py from mypy's exclude list (#13159)Andrew Morgan4-4/+3
2022-07-04[Complement] Allow device_name lookup over federation (#13167)Till2-0/+3
2022-07-01matrix-synapse-ldap3: 0.2.0 -> 0.2.1 (#13156)David Robertson2-4/+5
2022-07-01Enable Complement testing in the 'Twisted Trunk' CI runs. (#13079)reivilibre4-2/+82
2022-07-01complement.sh: Permit skipping docker build (#13143)Richard van der Hoff2-18/+55
Add a `-f` argument to `complement.sh` to skip the docker build
2022-07-011.62.0rc2 v1.62.0rc2Andrew Morgan5-3/+17
2022-07-01Extra validation for rest/client/account_data (#13148)David Robertson2-2/+18
* Extra validation for rest/client/account_data This is a fairly simple endpoint and we did pretty well here. * Changelog
2022-07-01`_process_received_pdu`: Improve exception handling (#13145)Richard van der Hoff2-7/+7
`_check_event_auth` is expected to raise `AuthError`s, so no need to log it again.
2022-07-01Skip waiting for full state for incoming events (#13144)Richard van der Hoff3-4/+13
When we receive an event over federation during a faster join, there is no need to wait for full state, since we have a whole reconciliation process designed to take the partial state into account.
2022-06-30Add documentation for phone home stats (#13086)Andrew Morgan3-0/+83
2022-06-30Allow dependency errors to pass through (#13113)Jacek Kuśnierz8-58/+16
Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-06-30Don't process /send requests for users who have hit their ratelimit (#13134)Shay2-0/+4
2022-06-30Cleanup references to sample config in the docs and redirect users to ↵Shay12-89/+73
configuration manual (#13077)
2022-06-30Add a link to the configuration manual from the homeserver sample config ↵Andrew Morgan2-0/+4
documentation page (#13139)
2022-06-30Fix unread counts on large servers (#13140)Erik Johnston3-29/+32
2022-06-30Add index to help delete old push actions (#13141)Erik Johnston4-4/+32
2022-06-30More type hints for `synapse.logging` (#13103)Patrick Cloke5-46/+56
Completes type hints for synapse.logging.scopecontextmanager and (partially) for synapse.logging.opentracing.
2022-06-30Improve startup times in Complement test runs against workers, particularly ↵reivilibre9-51/+243
in CPU-constrained environments. (#13127) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30Actually typecheck `tests.test_server` (#13135)David Robertson2-1/+1
2022-06-30Rate limiting invites per issuer (#13125)David Teller3-2/+24
Co-authored-by: reivilibre <oliverw@matrix.org>
2022-06-30Don't actually one-line the SQL statements we send to the DB (#13129)Brendan Abolivier2-3/+5
2022-06-29Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)Šimon Brandner11-13/+345
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-06-29Fix documentation header for `allow_public_rooms_over_federation` (#13116)Moritz Stückler2-1/+2
Signed-off-by: Moritz Stückler <moritz.stueckler@gmail.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-06-29Improve performance of getting unread counts in rooms (#13119)Erik Johnston6-4/+40
2022-06-29Document the `--report-stats` argument (#13029)jejo862-1/+4
Signed-off-by: jejo86 <28619134+jejo86@users.noreply.github.com>
2022-06-28fix linting error from the 1.61.1 main -> develop mergeAndrew Morgan1-1/+2
2022-06-281.62.0rc1Andrew Morgan80-80/+102
2022-06-28Linkify GHSA commit v1.61.1Andrew Morgan1-1/+1
2022-06-281.61.1Andrew Morgan3-1/+28
2022-06-28Merge pull request from GHSA-22p3-qrh9-cx32reivilibre2-24/+56
* Make _iterate_over_text easier to read by using simple data structures * Prefer a set of tags to ignore In my tests, it's 4x faster to check for containment in a set of this size * Add a stack size limit to _iterate_over_text * Continue accepting the case where there is no body element * Use an early return instead for None Co-authored-by: Richard van der Hoff <richard@matrix.org>
2022-06-28Fix serialization errors when rotating notifications (#13118)Erik Johnston5-83/+202
2022-06-28Extra type annotations in `test_server` (#13124)David Robertson3-37/+48
2022-06-28Remove unspecced DELETE endpoint that modifies room visibility (#13123)santhoshivan232-11/+1
2022-06-27Update MSC3786 implementation: Check the `state_key` (#12939)Šimon Brandner2-1/+8
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-06-27Add Cross-Origin-Resource-Policy header to thumbnail and download media ↵Robert Long5-2/+44
endpoints (#12944)
2022-06-27Refactor the Dockerfile-workers configuration script to use Jinja2 templates ↵reivilibre4-38/+43
in Synapse workers' Supervisord blocks. (#13054) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-27Remove docs for Delete Group Admin API (#13112)Aaron Raimist3-15/+1
This API no longer exists. Signed-off-by: Aaron Raimist <aaron@raim.ist>
2022-06-22validate room alias before interacting with the room directory (#13106)santhoshivan233-0/+20
2022-06-17Use caret (semver bounds) for matrix.org packages (#13082)David Robertson4-6/+9
2022-06-17Update opentracing docs to reference the configuration manual rather than ↵Shay2-2/+4
the configuation file. (#13076)
2022-06-17Fix inconsistencies in event validation (#13088)Richard van der Hoff5-7/+118
2022-06-17Fix inconsistencies in event validation for `m.room.create` events (#13087)Richard van der Hoff3-25/+88
* Extend the auth rule checks for `m.room.create` events ... and move them up to the top of the function. Since the no auth_events are allowed for m.room.create events, we may as well get the m.room.create event checks out of the way first. * Add a test for create events with prev_events
2022-06-17Add type hints to event push actions tests. (#13099)Patrick Cloke3-12/+19
2022-06-17Fix type error that made its way onto develop (#13098)reivilibre2-2/+3
* Fix type error introduced accidentally by #13045 * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-06-17Update info on downstream debs (#13095)Richard van der Hoff2-9/+9
2022-06-17Simplify the alias deletion logic as an application service. (#13093)Quentin Gliech3-22/+48
2022-06-17Rotate notifications more frequently (#13096)Erik Johnston2-1/+2
2022-06-17Use new `device_list_changes_in_room` table when getting device list changes ↵Erik Johnston4-31/+117
(#13045)
2022-06-17Allow MSC3030 'timestamp_to_event' calls from anyone on world-readable ↵Quentin Gliech2-1/+4
rooms. (#13062) Signed-off-by: Quentin Gliech <quenting@element.io>
2022-06-17Fix logging context misuse when we fail to persist a federation event (#13089)Sean Quah2-4/+3
When we fail to persist a federation event, we kick off a task to remove its push actions in the background, using the current logging context. Since we don't `await` that task, we may finish our logging context before the task finishes. There's no reason to not `await` the task, so let's do that. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-06-16Add desc to `get_earliest_token_for_stats` (#13085)Erik Johnston2-0/+2
2022-06-16Add a Subject Alternative Name to the certificate generated for Complement ↵reivilibre2-3/+20
tests. (#13071)
2022-06-16Improve URL previews for sites with only Twitter card information. (#13056)Patrick Cloke3-17/+137
Pull out `twitter:` meta tags when generating a preview and use it to augment any `og:` meta tags. Prefers Open Graph information over Twitter card information.
2022-06-16Reduce the duplication of code that invokes the rate limiter. (#13070)reivilibre2-27/+4
2022-06-16Complement: use SQLite by default (#13075)Richard van der Hoff3-2/+3
If no database is configured explicitly, use sqlite. This means that you don't have to pass any variables into the image.
2022-06-16Add custom well-known (#13035)Jacek Kuśnierz5-1/+68
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-06-16Add instructions for running Complement with `gotestfmt`-formatted output ↵reivilibre2-0/+15
locally. (#13073)
2022-06-15Move some event auth checks out to a different method (#13065)Richard van der Hoff7-98/+219
* Add auth events to events used in tests * Move some event auth checks out to a different method Some of the event auth checks apply to an event's auth_events, rather than the state at the event - which means they can play no part in state resolution. Move them out to a separate method. * Rename check_auth_rules_for_event Now it only checks the state-dependent auth rules, it needs a better name.
2022-06-15Add headers to individual options in config documentation to allow for ↵Shay2-163/+164
linking. (#13055)
2022-06-15Add more tests for room upgrades (#13074)Sean Quah2-5/+79
Signed-off-by: Sean Quah <seanq@element.io>
2022-06-15Change default `sync_response_cache_duration` (#13042)Erik Johnston5-4/+11
2022-06-15Replace pyjwt with authlib in `org.matrix.login.jwt` (#13011)Hannes Lerchl8-57/+100
2022-06-15Sort failing jobs in Complement CI to the top of the logs to make them ↵reivilibre3-0/+104
easier to read. (#13057)
2022-06-15Add a CI job to check that schema deltas are in the correct folder. (#13063)Erik Johnston5-6/+126
2022-06-15Type annotations in `synapse.databases.main.devices` (#13025)David Robertson5-21/+36
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-06-15Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston14-152/+323
Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
2022-06-15Rename complement-developonly (#13046)Erik Johnston2-0/+3
2022-06-15Use updated `update_user_directory_from_worker` options in Complement tests. ↵reivilibre2-3/+4
(#13069)
2022-06-15Collapse Docker build commands in Complement CI runs to make the logs easier ↵reivilibre6-7/+23
to read. (#13058)
2022-06-15Track a histogram of state res durations (#13036)David Robertson2-0/+13
2022-06-15Fix a long-standing bug which meant that rate limiting was not restrictive ↵reivilibre3-12/+45
enough in some cases. (#13018)
2022-06-15Don't use keyword arguments when initialising modules (#13060)Brendan Abolivier2-1/+2
2022-06-15Clean up schema for `event_edges` (#12893)Richard van der Hoff7-11/+216
* Remove redundant references to `event_edges.room_id` We don't need to care about the room_id here, because we are already checking the event id. * Clean up the event_edges table We make a number of changes to `event_edges`: * We give the `room_id` and `is_state` columns defaults (null and false respectively) so that we can stop populating them. * We drop any rows that have `is_state` set true - they should no longer exist. * We drop any rows that do not exist in `events` - these should not exist either. * We drop the old unique constraint on all the colums, which wasn't much use. * We create a new unique index on `(event_id, prev_event_id)`. * We add a foreign key constraint to `events`. These happen rather differently depending on whether we are on Postgres or SQLite. For SQLite, we just rebuild the whole table, copying only the rows we want to keep. For Postgres, we try to do things in the background as much as possible. * Stop populating `event_edges.room_id` and `is_state` We can just rely on the defaults.
2022-06-15Fix typechecks against twisted trunk (#13061)David Robertson3-4/+6
2022-06-14Fix incorrect link in changelog.Patrick Cloke1-1/+1
2022-06-14Refactor entry points so that they all have a `main` function. (#13052)reivilibre11-10/+61
2022-06-14Fix `destination_is` errors seen in sentry. (#13041)David Robertson7-8/+59
* Rename test_fedclient to match its source file * Require at least one destination to be truthy * Explicitly validate user ID in profile endpoint GETs Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-06-14Up complement time outs (#13048)Erik Johnston2-3/+4
2022-06-14Fix version number in spam checker callbacks doc (#13047)Brendan Abolivier2-6/+7
2022-06-14Remove code generating comments in configuration file (#12941)Shay41-5711/+66
2022-06-14Rename delta to apply in the proper schema version. (#13050)Patrick Cloke2-0/+1
2022-06-14Move the "email unsubscribe" resource, refactor the macaroon generator & ↵Quentin Gliech16-441/+619
simplify the access token verification logic. (#12986) This simplifies the access token verification logic by removing the `rights` parameter which was only ever used for the unsubscribe link in email notifications. The latter has been moved under the `/_synapse` namespace, since it is not a standard API. This also makes the email verification link more secure, by embedding the app_id and pushkey in the macaroon and verifying it. This prevents the user from tampering the query parameters of that unsubscribe link. Macaroon generation is refactored: - Centralised all macaroon generation and verification logic to the `MacaroonGenerator` - Moved to `synapse.utils` - Changed the constructor to require only a `Clock`, hostname, and a secret key (instead of a full `Homeserver`). - Added tests for all methods.
2022-06-14Fix Complement runs always being Postgres (#13034)reivilibre2-1/+2
* Fix Complement runs always being Postgres * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-06-14Fixup upgrades v1.61.0Erik Johnston1-3/+3
2022-06-14Update changelogErik Johnston1-3/+6
2022-06-141.61.0Erik Johnston4-2/+16
2022-06-14Mention removed community/group worker endpoints in upgrade.md (#13023)Sami Olmari2-0/+19
2022-06-14Mention removed community/group worker endpoints in upgrade.md (#13023)Sami Olmari2-0/+17
2022-06-14Decouple `synapse.api.auth_blocking.AuthBlocking` from ↵Quentin Gliech14-50/+63
`synapse.api.auth.Auth`. (#13021)
2022-06-13Uniformize spam-checker API, part 4: port other spam-checker callbacks to ↵David Teller12-182/+604
return `Union[Allow, Codes]`. (#12857) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-06-13Replace noop background updates with DELETE. (#