summary refs log tree commit diff
path: root/synapse (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Support rendering previews with data: URLs in them (#11767)Patrick Cloke2022-01-242-64/+191
| | | | | Images which are data URLs will no longer break URL previews and will properly be "downloaded" and thumbnailed.
* Merge tag 'v1.51.0rc2' into developAndrew Morgan2022-01-242-2/+5
|\ | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc2 (2022-01-24) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
| * 1.51.0rc2 v1.51.0rc2Andrew Morgan2022-01-241-1/+1
| |
| * Fix logic for dropping old events in fed queue (#11806)Andrew Morgan2022-01-241-1/+4
| | | | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
* | Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-242-2/+74
| | | | | | | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Drop support for and remove references to EOL Python 3.6 (#11683)Shay2022-01-214-14/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
* | Merge tag 'v1.51.0rc1' into developOlivier Wilkinson (reivilibre)2022-01-211-1/+1
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc1 (2022-01-21) ============================== Features -------- - Add `track_puppeted_user_ips` config flag to record client IP addresses against puppeted users, and include the puppeted users in monthly active user counts. ([\#11561](https://github.com/matrix-org/synapse/issues/11561), [\#11749](https://github.com/matrix-org/synapse/issues/11749), [\#11757](https://github.com/matrix-org/synapse/issues/11757)) - Include whether the requesting user has participated in a thread when generating a summary for [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11577](https://github.com/matrix-org/synapse/issues/11577)) - Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room. ([\#11672](https://github.com/matrix-org/synapse/issues/11672)) - Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. ([\#11675](https://github.com/matrix-org/synapse/issues/11675), [\#11770](https://github.com/matrix-org/synapse/issues/11770)) Bugfixes -------- - Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events received over federation. ([\#11530](https://github.com/matrix-org/synapse/issues/11530)) - Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices. ([\#11587](https://github.com/matrix-org/synapse/issues/11587)) - Fix an error that occurs whilst trying to get the federation status of a destination server that was working normally. This admin API was newly introduced in Synapse v1.49.0. ([\#11593](https://github.com/matrix-org/synapse/issues/11593)) - Fix bundled aggregations not being included in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675). ([\#11612](https://github.com/matrix-org/synapse/issues/11612), [\#11659](https://github.com/matrix-org/synapse/issues/11659), [\#11791](https://github.com/matrix-org/synapse/issues/11791)) - Fix the `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0. ([\#11667](https://github.com/matrix-org/synapse/issues/11667)) - Fix preview of some GIF URLs (like tenor.com). Contributed by Philippe Daouadi. ([\#11669](https://github.com/matrix-org/synapse/issues/11669)) - Fix a bug where only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0. ([\#11695](https://github.com/matrix-org/synapse/issues/11695)) - Fix a bug introduced in Synapse v1.18.0 where password reset and address validation emails would not be sent if their subject was configured to use the 'app' template variable. Contributed by @br4nnigan. ([\#11710](https://github.com/matrix-org/synapse/issues/11710), [\#11745](https://github.com/matrix-org/synapse/issues/11745)) - Make the 'List Rooms' Admin API sort stable. Contributed by Daniël Sonck. ([\#11737](https://github.com/matrix-org/synapse/issues/11737)) - Fix a long-standing bug where space hierarchy over federation would only work correctly some of the time. ([\#11775](https://github.com/matrix-org/synapse/issues/11775)) - Fix a bug introduced in Synapse v1.46.0 that prevented `on_logged_out` module callbacks from being correctly awaited by Synapse. ([\#11786](https://github.com/matrix-org/synapse/issues/11786)) Improved Documentation ---------------------- - Warn against using a Let's Encrypt certificate for TLS/DTLS TURN server client connections, and suggest using ZeroSSL certificate instead. This works around client-side connectivity errors caused by WebRTC libraries that reject Let's Encrypt certificates. Contibuted by @AndrewFerr. ([\#11686](https://github.com/matrix-org/synapse/issues/11686)) - Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide. ([\#11715](https://github.com/matrix-org/synapse/issues/11715)) - Document that the minimum supported PostgreSQL version is now 10. ([\#11725](https://github.com/matrix-org/synapse/issues/11725)) - Fix typo in demo docs: differnt. ([\#11735](https://github.com/matrix-org/synapse/issues/11735)) - Update room spec URL in config files. ([\#11739](https://github.com/matrix-org/synapse/issues/11739)) - Mention `python3-venv` and `libpq-dev` dependencies in the contribution guide. ([\#11740](https://github.com/matrix-org/synapse/issues/11740)) - Update documentation for configuring login with Facebook. ([\#11755](https://github.com/matrix-org/synapse/issues/11755)) - Update installation instructions to note that Python 3.6 is no longer supported. ([\#11781](https://github.com/matrix-org/synapse/issues/11781)) Deprecations and Removals ------------------------- - Remove the unstable `/send_relation` endpoint. ([\#11682](https://github.com/matrix-org/synapse/issues/11682)) - Remove `python_twisted_reactor_pending_calls` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724)) - Remove the `password_hash` field from the response dictionaries of the [Users Admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html). ([\#11576](https://github.com/matrix-org/synapse/issues/11576)) - Deprecate support for `webclient` listeners and non-HTTP(S) `web_client_location` configuration. ([\#11774](https://github.com/matrix-org/synapse/issues/11774), [\#11783](https://github.com/matrix-org/synapse/issues/11783)) Internal Changes ---------------- - Run `pyupgrade --py37-plus --keep-percent-format` on Synapse. ([\#11685](https://github.com/matrix-org/synapse/issues/11685)) - Use buildkit's cache feature to speed up docker builds. ([\#11691](https://github.com/matrix-org/synapse/issues/11691)) - Use `auto_attribs` and native type hints for attrs classes. ([\#11692](https://github.com/matrix-org/synapse/issues/11692), [\#11768](https://github.com/matrix-org/synapse/issues/11768)) - Remove debug logging for #4422, which has been closed since Synapse 0.99. ([\#11693](https://github.com/matrix-org/synapse/issues/11693)) - Remove fallback code for Python 2. ([\#11699](https://github.com/matrix-org/synapse/issues/11699)) - Add a test for [an edge case](https://github.com/matrix-org/synapse/pull/11532#discussion_r769104461) in the `/sync` logic. ([\#11701](https://github.com/matrix-org/synapse/issues/11701)) - Add the option to write SQLite test dbs to disk when running tests. ([\#11702](https://github.com/matrix-org/synapse/issues/11702)) - Improve Complement test output for Gitub Actions. ([\#11707](https://github.com/matrix-org/synapse/issues/11707)) - Fix docstring on `add_account_data_for_user`. ([\#11716](https://github.com/matrix-org/synapse/issues/11716)) - Complement environment variable name change and update `.gitignore`. ([\#11718](https://github.com/matrix-org/synapse/issues/11718)) - Simplify calculation of Prometheus metrics for garbage collection. ([\#11723](https://github.com/matrix-org/synapse/issues/11723)) - Improve accuracy of `python_twisted_reactor_tick_time` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724), [\#11771](https://github.com/matrix-org/synapse/issues/11771)) - Minor efficiency improvements when inserting many values into the database. ([\#11742](https://github.com/matrix-org/synapse/issues/11742)) - Invite PR authors to give themselves credit in the changelog. ([\#11744](https://github.com/matrix-org/synapse/issues/11744)) - Add optional debugging to investigate [issue 8631](https://github.com/matrix-org/synapse/issues/8631). ([\#11760](https://github.com/matrix-org/synapse/issues/11760)) - Remove `log_function` utility function and its uses. ([\#11761](https://github.com/matrix-org/synapse/issues/11761)) - Add a unit test that checks both `client` and `webclient` resources will function when simultaneously enabled. ([\#11765](https://github.com/matrix-org/synapse/issues/11765)) - Allow overriding complement commit using `COMPLEMENT_REF`. ([\#11766](https://github.com/matrix-org/synapse/issues/11766)) - Add some comments and type annotations for `_update_outliers_txn`. ([\#11776](https://github.com/matrix-org/synapse/issues/11776))
| * 1.51.0rc1Olivier Wilkinson (reivilibre)2022-01-211-1/+1
| |
| * Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke2022-01-213-18/+10
| |
* | Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-213-4/+37
| | | | | | | | ... and start populating them for new events
* | Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke2022-01-213-18/+10
| |
* | Drop unused table `public_room_list_stream`. (#11795)Richard van der Hoff2022-01-213-2/+21
| | | | | | This is a follow-up to #10565.
* | Stop reading from `event_reference_hashes` (#11794)Richard van der Hoff2022-01-214-33/+30
| | | | | | | | Preparation for dropping this table altogether. Part of #6574.
* | Add `FrozenEvent.get_state_key` and use it in a couple of places (#11793)Richard van der Hoff2022-01-212-5/+10
| | | | | | | | This is more efficient, since we only have to look up `state_key` in the event dict once, rather than three (!) times.
* | Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre2022-01-214-7/+7
| | | | | | | | whose key is prefixed with the user ID (#11788)
* | Make `get_account_data_for_room_and_type` a tree cache (#11789)reivilibre2022-01-211-1/+1
|/
* Correctly await on_logged_out callbacks (#11786)Brendan Abolivier2022-01-201-1/+1
|
* Fix redirecting to the webclient for non-HTTP(S) web_client_location. (#11783)Patrick Cloke2022-01-201-1/+8
| | | | | To not change the behaviour during the deprecation period. Follow-up to #11774.
* Partially revert #11675; prevent attempting to create pushers on workers ↵Andrew Morgan2022-01-201-7/+5
| | | | (#11770)
* Add deprecation warnings for `webclient` listener and non-HTTP(S) ↵Patrick Cloke2022-01-202-19/+27
| | | | | | | `web_client_location`. (#11774) This changes the behaviour of the root endpoint to redirect directly to the configuration of `web_client_location` if it is given an HTTP(S) URL.
* Debug for device lists updates (#11760)David Robertson2022-01-203-0/+45
| | | | | | | | | | | | | | | | | | Debug for #8631. I'm having a hard time tracking down what's going wrong in that issue. In the reported example, I could see server A sending federation traffic to server B and all was well. Yet B reports out-of-sync device updates from A. I couldn't see what was _in_ the events being sent from A to B. So I have added some crude logging to track - when we have updates to send to a remote HS - the edus we actually accumulate to send - when a federation transaction includes a device list update edu - when such an EDU is received This is a bit of a sledgehammer.
* Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah2022-01-202-10/+11
| | | | `FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
* Comments and typing for `_update_outliers_txn` (#11776)Richard van der Hoff2022-01-191-12/+23
| | | | A couple of surprises for me here, so thought I'd document them
* Fix mypy for platforms without epoll support. (#11771)Patrick Cloke2022-01-191-2/+2
|
* Add missing `auto_attribs=True` to the `_WrappedRustReporter` class (#11768)Andrew Morgan2022-01-191-1/+1
|
* Fix preview of imgur and Tenor URLs. (#11669)Philippe Daouadi2022-01-182-13/+32
| | | | | | By scraping Open Graph information from the HTML even when an autodiscovery endpoint is found. The results are then combined to capture as much information as possible from the page.
* Merge branch 'master' into developAndrew Morgan2022-01-182-3/+8
|\
| * 1.50.1Andrew Morgan2022-01-181-1/+1
| |
| * Partially revert #11602 to prevent webclient overriding client resource (#11764)Andrew Morgan2022-01-181-2/+7
| |
* | Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke2022-01-187-18/+81
| | | | | | | | | | | | Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
* | Remove `log_function` and its uses (#11761)Richard van der Hoff2022-01-1812-155/+0
| | | | | | | | | | | | | | I've never found this terribly useful. I think it was added in the early days of Synapse, without much thought as to what would actually be useful to log, and has just been cargo-culted ever since. Rather, it tends to clutter up debug logs with useless information.
* | Merge branch 'master' into developAndrew Morgan2022-01-181-1/+1
|\|
| * 1.50.0Andrew Morgan2022-01-181-1/+1
| |
* | Add a flag to the `synapse_review_recent_signups` script to ignore and ↵lukasdenk2022-01-172-8/+22
| | | | | | | | filter appservice users. (#11675)
* | Fix up docs for `track_puppeted_user_ips` (again) (#11757)Richard van der Hoff2022-01-171-4/+12
| | | | | | Fixes #11741
* | Update documentation for configuring facebook login (#11755)Richard van der Hoff2022-01-171-3/+6
| | | | | | ... and a minor thinko fix in the sample config.
* | Improve `reactor_tick_time` metric (#11724)Richard van der Hoff2022-01-172-93/+84
| | | | | | | | | | | | | | | | | | | | | | The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only* measures the time taken to execute timed calls and callbacks from threads. That neglects everything that happens off the back of I/O, which is obviously quite a lot for us. To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is the only place it should wait for something to happen - the rest of the loop *should* be quick. I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and it's a nuisance to populate.
* | Make pagination of rooms in admin api stable (#11737)Daniel Sonck2022-01-171-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Always add state.room_id after the configurable ORDER BY. Otherwise, for any sort, certain pages can contain results from other pages. (Especially when sorting by creator, since there may be many rooms by the same creator) * Document different order direction of numerical fields "joined_members", "joined_local_members", "version" and "state_events" are ordered in descending direction by default (dir=f). Added a note in tests to explain the differences in ordering. Signed-off-by: Daniël Sonck <daniel@sonck.nl>
* | Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-142-18/+86
|\|
| * 1.50.0rc2Olivier Wilkinson (reivilibre)2022-01-141-1/+1
| |
| * Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could ↵reivilibre2022-01-131-16/+78
| | | | | | | | | | fail because too many EDUs were produced for device updates. (#11730) Co-authored-by: David Robertson <davidr@element.io>
| * Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not ↵reivilibre2022-01-121-1/+7
| | | | | | | | | | be sent to remote homeservers if there were too many to send at once. (#11729) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Remove the 'password_hash' from the Users Admin API endpoint response ↵Andrew Morgan2022-01-142-22/+47
| | | | | | | | dictionary (#11576)
* | Fix sample_config.yaml in regards track_puppeted_user_ips (#11749)Jason Robinson2022-01-141-5/+6
| | | | | | | | | | | | | | * Fix sample_config.yaml in regards track_puppeted_user_ips Closes #11741 Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-1316-277/+242
| | | | | | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* | Fix missing app variable in mail subject (#11745)David Robertson2022-01-131-2/+2
| | | | | | | | | | documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500 Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>
* | update room spec url in config files (#11739)qwertyforce2022-01-131-1/+1
| | | | | | | | | | | | | | * change spec url in config files * Create 11739.txt * .txt -> .doc
* | Include bundled aggregations in the sync response cache. (#11659)Patrick Cloke2022-01-132-14/+13
| |
* | Simplify GC prometheus metrics (#11723)Richard van der Hoff2022-01-133-160/+208
| | | | | | | | | | Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection. Part 1 of a quest to simplify the reactor monkey-patching.
* | Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-1339-307/+299
| |
* | Allow tracking puppeted users for MAU (#11561)Jason Robinson2022-01-122-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently when puppeting another user, the user doing the puppeting is tracked for client IPs and MAU (if configured). When tracking MAU is important, it becomes necessary to be possible to also track the client IPs and MAU of puppeted users. As an example a client that manages user creation and creation of tokens via the Synapse admin API, passing those tokens for the client to use. This PR adds optional configuration to enable tracking of puppeted users into monthly active users. The default behaviour stays the same. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Fix docstring on `add_account_data_for_user`. (#11716)reivilibre2022-01-102-2/+2
| |
* | Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke2022-01-071-8/+22
| | | | | | | | | | | | | | By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
* | Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-073-1/+11
|\|
| * Work around Mjolnir compatibility issue by adding an import for ↵reivilibre2022-01-061-0/+7
| | | | | | | | `glob_to_regex` in `synapse.util`, where it moved from. (#11696)
| * Include `io.element.thread` capability for MSC3440. (#11690)Patrick Cloke2022-01-051-0/+3
| |
| * 1.50.0rc1Olivier Wilkinson (reivilibre)2022-01-051-1/+1
| |
* | Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke2022-01-0715-155/+247
| | | | | | | | | | | | | | | | This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
* | Remove the /send_relation endpoint. (#11682)Patrick Cloke2022-01-061-120/+5
| | | | | | | | This was removed from MSC2674 before that was approved and is not used by any known clients.
* | Remove a Python 2-ism and improve type hints. (#11699)Patrick Cloke2022-01-061-12/+7
| | | | | | | | On Python 2, indexing a byte-string gives back a byte-string, while on Python 3 it gives back the ASCII equivalent as an int.
* | Strip unauthorized fields from `unsigned` object in events received over ↵Shay2022-01-061-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | federation (#11530) * add some tests to verify we are stripping unauthorized fields out of unsigned * add function to strip unauthorized fields from the unsigned object of event * newsfragment * update newsfragment number * add check to on_send_membership_event * refactor tests * fix lint error * slightly refactor tests and add some comments * slight refactor * refactor tests * fix import error * slight refactor * remove unsigned filtration code from synapse/handlers/federation_event.py * lint * move unsigned filtering code to event base * refactor tests * update newsfragment * requested changes * remove unused retun values
* | Make room creations denied by `user_may_create_room` cause an `M_FORBIDDEN` ↵lukasdenk2022-01-061-2/+6
| | | | | | | | | | error to be returned, not `M_UNKNOWN` (#11672) Co-authored-by: reivilibre <olivier@librepush.net>
* | Remove debug logging for #4422 (#11693)David Robertson2022-01-061-37/+1
| | | | | | | | as per https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
* | Fix space hierarchy endpoint to match MSC2946 (#11667)Travis Ralston2022-01-051-2/+4
| | | | | | | | | | | | | | Fixes minor discrepancies between the /hierarchy endpoint described in MSC2946 and the implementation. Note that the changes impact the stable and unstable /hierarchy and unstable /spaces endpoints for both client and federation APIs.
* | Fix get federation status of destination if no error occured (#11593)Dirk Klimpel2022-01-052-7/+30
| |
* | Run `pyupgrade --py37-plus --keep-percent-format` on Synapse (#11685)Shay2022-01-052-2/+1
| | | | | | | | | | | | | | | | | | * newsfragment * fix newsfragment number * update changelog * remove extra space
* | Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-052-3/+15
|/ | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Refactor the way we set `outlier` (#11634)Richard van der Hoff2022-01-053-44/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
* Fix link from generated configuration file to documentation (#11678)Philipp Matthias Schäfer2022-01-051-1/+1
| | | | Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: reivilibre <oliverw@matrix.org>
* Fix SimpleHttpClient not sending Accept header in `get_json` (#11677)Fr3shTea2022-01-051-1/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Add admin API to get users' account data (#11664)Dirk Klimpel2022-01-052-0/+32
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-056-64/+12
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Fix AssertionErrors after purging events (#11642)Richard van der Hoff2022-01-042-3/+26
| | | | | | | | | | | * Fix AssertionErrors after purging events If you purged a bunch of events from your database, and then restarted synapse without receiving more events, then you would get a bunch of AssertionErrors on restart. This fixes the situation by rewinding the stream processors. * `check-newsfragment`: ignore deleted newsfiles
* `FederationClient.backfill`: stop flagging events as outliers (#11632)Richard van der Hoff2022-01-042-2/+4
| | | | | | | | | | | | | | | | | | | | | Events returned by `backfill` should not be flagged as outliers. Fixes: ``` AssertionError: null File "synapse/handlers/federation.py", line 313, in try_backfill dom, room_id, limit=100, extremities=extremities File "synapse/handlers/federation_event.py", line 517, in backfill await self._process_pulled_events(dest, events, backfilled=True) File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events await self._process_pulled_event(origin, ev, backfilled=backfilled) File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event assert not event.internal_metadata.is_outlier() ``` See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992 Fixes #8894.
* Remove redundant `get_current_events_token` (#11643)Richard van der Hoff2022-01-046-36/+19
| | | | | | | | | | | | | | | | | * Push `get_room_{min,max_stream_ordering}` into StreamStore Both implementations of this are identical, so we may as well push it down and get rid of the abstract base class nonsense. * Remove redundant `StreamStore` class This is empty now * Remove redundant `get_current_events_token` This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid of it. * newsfile
* Better error messages from `get_create_event_for_room` (#11638)Richard van der Hoff2022-01-041-1/+5
| | | | "Unknown room" can mean a multitude of things here. To help with debugging, add some more words to the exception text.
* Include the topic event in the prejoin state, per MSC3173. (#11666)Patrick Cloke2022-01-041-0/+2
| | | | Invites and knocks will now include the topic in the stripped state send to clients before joining the room.
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-3020-205/+225
| | | To improve type hints throughout the code.
* Add type hints to `synapse/storage/databases/main/events_bg_updates.py` (#11654)Dirk Klimpel2021-12-301-29/+40
|
* Improve type hints in storage classes. (#11652)Dirk Klimpel2021-12-299-34/+43
| | | By using cast and making ignores more specific.
* Do not attempt to bundled aggregations for /members and /state. (#11623)Patrick Cloke2021-12-291-3/+1
| | | | Both of those APIs return state events, which will not have bundled aggregations added anyway.
* Add type hints to `synapse/storage/databases/main/stats.py` (#11653)Dirk Klimpel2021-12-291-41/+53
|
* Update to the current version of Black and run it on Synapse codebase (#11596)Shay2021-12-232-2/+0
| | | | | | | | | * update black version * run updated version of black on code * newsfragment * enumerate python versions
* Fix mypy error with opentracing.tags. (#11622)Patrick Cloke2021-12-231-0/+1
|
* Merge branch 'master' into developRichard van der Hoff2021-12-212-2/+3
|\
| * 1.49.2Richard van der Hoff2021-12-211-1/+1
| |
| * Pin to frozendict<2.1.2 (#11625)Richard van der Hoff2021-12-211-1/+2
| | | | | | | | ... to work around breakage on buster (https://github.com/Marco-Sulla/python-frozendict/issues/41)
| * 1.49.1Richard van der Hoff2021-12-211-1/+1
| |
* | Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-217-145/+210
| |
* | Various opentracing enhancements (#11619)Richard van der Hoff2021-12-214-21/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Wrap `auth.get_user_by_req` in an opentracing span give `get_user_by_req` its own opentracing span, since it can result in a non-trivial number of sub-spans which it is useful to group together. This requires a bit of reorganisation because it also sets some tags (and may force tracing) on the servlet span. * Emit opentracing span for encoding json responses This can be a significant time sink. * Rename all sync spans with a prefix * Write an opentracing span for encoding sync response * opentracing span to group generate_room_entries * opentracing spans within sync.encode_response * changelog * Use the `trace` decorator instead of context managers
* | Do not bundle aggregations for APIs which shouldn't include them. (#11592)Patrick Cloke2021-12-208-27/+30
| | | | | | | | | | And make bundling aggregations opt-in, instead of opt-out to avoid having APIs to include extraneous data (and being much heavier than necessary).
* | Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-203-37/+114
| | | | | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* | Improve opentracing for incoming HTTP requests (#11618)Richard van der Hoff2021-12-203-73/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove `start_active_span_from_request` Instead, pull out a separate function, `span_context_from_request`, to extract the parent span, which we can then pass into `start_active_span` as normal. This seems to be clearer all round. * Remove redundant tags from `incoming-federation-request` These are all wrapped up inside a parent span generated in AsyncResource, so there's no point duplicating all the tags that are set there. * Leave request spans open until the request completes It may take some time for the response to be encoded into JSON, and that JSON to be streamed back to the client, and really we want that inside the top-level span, so let's hand responsibility for closure to the SynapseRequest. * opentracing logs for HTTP request events * changelog
* | Merge remote-tracking branch 'origin/release-v1.49' into developRichard van der Hoff2021-12-201-1/+9
|\|
| * Disable aggregation bundling on `/sync` responses (#11583)Richard van der Hoff2021-12-201-1/+9
| | | | | | | | | | | | | | | | | | | | | | * Disable aggregation bundling on `/sync` responses A partial revert of #11478. This turns out to have had a significant CPU impact on initial-sync handling. For now, let's disable it, until we find a more efficient way of achieving this. * Fix tests. Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* | Return JSON errors for unknown resources under /matrix/client. (#11602)Patrick Cloke2021-12-202-10/+5
| | | | | | | | Instead of returning 404 errors with HTML bodies when an unknown prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
* | Add opentracing types (#11603)Shay2021-12-202-10/+16
| |
* | Add type hints to `synapse/storage/databases/main/transactions.py` (#11589)Dirk Klimpel2021-12-161-24/+25
| |
* | Add MSC2716 and MSC3030 to `/versions` -> `unstable_features` (#11582)Eric Eastwood2021-12-162-1/+5
| | | | | | | | As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
* | Add type hints to `synapse/storage/databases/main/room.py` (#11575)Sean Quah2021-12-153-76/+104
| |
* | Require Collections as the parameters for simple_* methods. (#11580)Patrick Cloke2021-12-152-40/+11
| | | | | | | | Instead of Iterable since the generators are not allowed due to the potential for their re-use.
* | Convert EventStreamResult to attrs. (#11574)Patrick Cloke2021-12-152-9/+23
| |
* | Add experimental support for MSC3202: allowing application services to ↵reivilibre2021-12-153-14/+97
| | | | | | | | masquerade as specific devices. (#11538)
* | Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-143-77/+181
| |
* | Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-149-83/+206
| |
* | Add missing type hints to `synapse.appservice` (#11360)Patrick Cloke2021-12-144-85/+141
| |
* | Merge branch 'master' into developOlivier Wilkinson (reivilibre)2021-12-141-1/+1
|\|
| * 1.49.0Olivier Wilkinson (reivilibre)2021-12-141-1/+1
| |
* | Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-147-20/+18
| | | | | | | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* | Add missing type hints to synapse.http. (#11571)Patrick Cloke2021-12-146-51/+72
| |
* | checks for generators in database functions (#11564)Richard van der Hoff2021-12-132-7/+48
| | | | | | | | | | A couple of safety-checks to hopefully stop people doing what I just did, and create a storage function which only works the first time it is called (and not when it is re-run due to a database concurrency error or similar).
* | Move HTML parsing to a separate file for URL previews. (#11566)Patrick Cloke2021-12-133-378/+407
| | | | | | | | | | | | | | * Splits the logic for parsing HTML from the resource handling code. * Fix a circular import in the oEmbed code (which uses the HTML parsing code). * Renames some of the HTML parsing methods to: * Make it clear which methods are "internal" to the module. * Clarify what the methods do.
* | Type hint the constructors of the data store classes (#11555)Sean Quah2021-12-1334-87/+350
| |
* | Add type hints to `synapse/storage/databases/main/end_to_end_keys.py` (#11551)Sean Quah2021-12-132-68/+146
| |
* | Add type hints to `synapse/storage/databases/main/account_data.py` (#11546)Sean Quah2021-12-132-32/+83
| |
* | Make `get_device` return None if the device doesn't exist rather than ↵reivilibre2021-12-135-13/+19
| | | | | | | | | | raising an exception. (#11565) Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-102-7/+20
| | | | | | | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* | Adjust _get_rooms_changed comments (#11550)David Robertson2021-12-101-22/+29
| | | | | | C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
* | skip some dict munging in event persistence (#11560)Richard van der Hoff2021-12-102-60/+113
| | | | | | | | | | | | Create a new dict helper method `simple_insert_many_values_txn`, which takes raw row values, rather than {key=>value} dicts. This saves us a bunch of dict munging, and makes it easier to use generators rather than creating intermediate lists and dicts.
* | Stop populating `state_events.prev_state` (#11558)Richard van der Hoff2021-12-102-5/+4
| | | | | | this field is never read, so we may as well stop populating it.
* | Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke2021-12-103-3/+8
| | | | | | | | This should fix pushers with an email in non-canonical form is used as the pushkey.
* | Do not allow cross-room relations, per MSC2674. (#11516)Patrick Cloke2021-12-094-17/+45
| |
* | Additional type hints for the config module, part 2. (#11480)Patrick Cloke2021-12-094-19/+27
| |
* | Use HTTPStatus constants in place of literals in `synapse.http` (#11543)Dirk Klimpel2021-12-093-19/+46
| |
* | Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-092-1/+6
| |
* | Add missing `errcode` to `parse_string` and `parse_boolean` (#11542)Dirk Klimpel2021-12-091-2/+2
| |
* | Allow guests to send state events (#11378)Robert Long2021-12-091-1/+1
| |
* | Add a constant for receipt types (m.read). (#11531)Patrick Cloke2021-12-088-45/+86
| | | | | | And expand some type hints in the receipts storage module.
* | Clean up `synapse.rest.admin` (#11535)Dirk Klimpel2021-12-0813-164/+94
| |
* | Improve log messages for stream ids (#11536)Richard van der Hoff2021-12-082-4/+6
| | | | | | | | Somehow I'd managed to get my database in a pickle with stream ids. These changes were useful to debug.
* | Send and handle cross-signing messages using the stable prefix. (#10520)Hubert Chathi2021-12-082-3/+9
|/
* 1.49.0rc1Sean Quah2021-12-071-1/+1
|
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-076-12/+64
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Refactors in `_generate_sync_entry_for_rooms` (#11515)David Robertson2021-12-071-45/+77
| | | | | | | | * Move sync_token up to the top * Pull out _get_ignored_users * Try to signpost the body of `_generate_sync_entry_for_rooms` * Pull out _calculate_user_changes Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correctly register shutdown handler for presence workers (#11518)David Robertson2021-12-071-1/+1
| | | Fixes #11517
* Fix `ModuleApi.looping_background_call` for non-async functions (#11524)Sean Quah2021-12-071-3/+2
| | | | | | After #10847, `looping_background_call` would print an error in the logs every time a non-async function was called. Since the error would be caught and ignored immediately, there were no other side effects.
* Fix 'delete room' admin api to work on incomplete rooms (#11523)Richard van der Hoff2021-12-073-20/+7
| | | | | If, for some reason, we don't have the create event, we should still be able to purge a room.
* Correctly ignore invites from ignored users (#11511)David Robertson2021-12-071-5/+6
|
* Stabilise support for MSC2918 refresh tokens as they have now been merged ↵reivilibre2021-12-063-29/+61
| | | | into the Matrix specification. (#11435)
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-0610-32/+181
| | | As a step towards allowing back-channel logout for OIDC.
* Add admin API to get some information about federation status (#11407)Dirk Klimpel2021-12-063-0/+211
|
* Include bundled aggregations in /sync and related fixes (#11478)Patrick Cloke2021-12-068-62/+72
| | | | | | | | Due to updates to MSC2675 this includes a few fixes: * Include bundled aggregations for /sync. * Do not include bundled aggregations for /initialSync and /events. * Do not bundle aggregations for state events. * Clarifies comments and variable names.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-066-64/+12
|
* Support configuring the lifetime of non-refreshable access tokens separately ↵reivilibre2021-12-032-3/+66
| | | | to refreshable access tokens. (#11445)
* Disambiguate queries on `state_key` (#11497)Richard van der Hoff2021-12-026-16/+20
| | | | | We're going to add a `state_key` column to the `events` table, so we need to add some disambiguation to queries which use it.
* Comments on the /sync tentacles (#11494)David Robertson2021-12-022-43/+128
| | | | | This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add most of the missing type hints to `synapse.federation`. (#11483)Patrick Cloke2021-12-028-49/+77
| | | This skips a few methods which are difficult to type.
* Fix media repository failing when media store path contains symlinks (#11446)Sean Quah2021-12-021-44/+71
|
* Add type annotations to `tests.storage.test_appservice`. (#11488)reivilibre2021-12-022-4/+5
|
* Add MSC3030 experimental client and federation API endpoints to get the ↵Eric Eastwood2021-12-0212-31/+673
| | | | | | | | | | | | | | | | | | | | | | | | | closest event to a given timestamp (#9445) MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030 Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about. ``` GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction> { "event_id": ... "origin_server_ts": ... } ``` Federation API endpoint: ``` GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction> { "event_id": ... "origin_server_ts": ... } ``` Co-authored-by: Erik Johnston <erik@matrix.org>
* Clean-up get_version_string (#11468)Patrick Cloke2021-12-011-57/+25
|
* Additional type hints for config module. (#11465)Patrick Cloke2021-12-0114-99/+128
| | | | This adds some misc. type hints to helper methods used in the `synapse.config` module.
* Register the login redirect endpoint for v3. (#11451)Patrick Cloke2021-12-012-3/+3
| | | As specified for Matrix v1.1.
* Don't start Synapse master process if `worker_app` is set (#11416)Shay2021-11-301-0/+7
| | | | | | | | | | | | | | | | | | | | | * Add check to catch syanpse master process starting when workers are configured * add test to verify that starting master process with worker config raises error * newsfragment * specify config.worker.worker_app in check * update test * report specific config option that triggered the error Co-authored-by: reivilibre <oliverw@matrix.org> * clarify error message Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* Expose worker & homeserver as entrypoints in `setup.py` (#11449)Maximilian Bosch2021-11-301-1/+5
| | | Co-authored-by: reivilibre <oliverw@matrix.org>
* Bundle relations of relations into the `/relations` result. (#11284)Patrick Cloke2021-11-302-6/+11
| | | | | Per updates to MSC2675 which now states that bundled aggregations should be included from the `/relations` endpoint.
* Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)Sean Quah2021-11-301-1/+4
| | | | | | | | | | | When all entries in an `LruCache` have a size of 0 according to the provided `size_callback`, and `drop_from_cache` is called on a cache node, the node would be unlinked from the LRU linked list but remain in the cache dictionary. An assertion would be later be tripped due to the inconsistency. Avoid unintentionally calling `__len__` and use a strict `is None` check instead when unwrapping the weak reference.
* Eliminate a few `Any`s in `LruCache` type hints (#11453)Sean Quah2021-11-303-19/+31
|
* Merge branch 'master' into developBrendan Abolivier2021-11-301-1/+1
|\
| * 1.48.0Brendan Abolivier2021-11-301-1/+1
| |
| * 1.48.0rc1Brendan Abolivier2021-11-251-1/+1
| |
* | Improved push typing (#11409)Marcus2021-11-305-36/+204
| | | | | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Convert status codes to `HTTPStatus` in `synapse.rest.admin` (#11452)Dirk Klimpel2021-11-2911-171/+275
| |
* | Refactor `backfilled` into specific behavior function arguments ↵Eric Eastwood2021-11-292-21/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (`_persist_events_and_state_updates`) (#11417) Part of https://github.com/matrix-org/synapse/issues/11300 Call stack: - `_persist_events_and_state_updates` (added `use_negative_stream_ordering`) - `_persist_events_txn` - `_update_room_depths_txn` (added `update_room_forward_stream_ordering`) - `_update_metadata_tables_txn` - `_store_room_members_txn` (added `inhibit_local_membership_updates`) Using keyword-only arguments (`*`) to reduce the mistakes from `backfilled` being left as a positional argument somewhere and being interpreted wrong by our new arguments.
* | Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-296-18/+64
| | | | | | | | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* | Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-292-24/+222
| | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Return the stable `event` field from `/send_join` per MSC3083. (#11413)Patrick Cloke2021-11-292-2/+16
| | | | | | | | | | This does not remove the unstable field and still parses both. Handling of the unstable field will need to be removed in the future.
* | Add type annotations to some of the configuration surrounding refresh ↵reivilibre2021-11-292-2/+10
| | | | | | | | tokens. (#11428)
* | Fix perspectives requests for multiple keys for the same server (#11440)Richard van der Hoff2021-11-291-11/+19
| | | | | | | | | | If we tried to request multiple keys for the same server, we would end up dropping some of those requests.
* | Fix using MSC2716 batch sending with event persistence workers (#11220)Tulir Asokan2021-11-292-0/+7
| | | | | | | | Signed-off-by: Tulir Asokan <tulir@beeper.com>
* | disallow-untyped-defs for the module_api (#11029)David Robertson2021-11-291-30/+55
| |
* | Bump ijson dependency to 3.1 (#11438) (#11439)Daniel Molkentin2021-11-261-1/+1
| | | | | | | | | | | | | | | | | | | | Since e81fa9264873369653171157514ff68226491fff, Synapse depends on the use_float flag which has been introduced in ijson 3.1 and is not available in 3.0. This is known to cause runtime errors with send_join. Signed-off-by: Daniel Molkentin <danimo@infra.run> Co-authored-by: Daniel Molkentin <danimo@infra.run>
* | Update MSC2918 refresh token support to confirm with the latest revision: ↵reivilibre2021-11-262-8/+13
| | | | | | | | accept the `refresh_tokens` parameter in the request body rather than in the URL parameters. (#11430)
* | Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-2610-169/+246
| | | | | | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* | Support expiry of refresh tokens and expiry of the overall session when ↵reivilibre2021-11-266-53/+213
| | | | | | | | refresh tokens are in use. (#11425)
* | Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-42/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* | Annotate string constants in `synapse.api.constants` with `Final` (#11356)Sean Quah2021-11-251-98/+100
|/ | | | | | | | | | This change makes mypy complain if the constants are ever reassigned, and, more usefully, makes mypy type them as `Literal`s instead of `str`s, allowing code of the following form to pass mypy: ```py def do_something(membership: Literal["join", "leave"], ...): ... do_something(Membership.JOIN, ...) ```
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-252-146/+81
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-241-1/+1
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Rename unstable `access_token_lifetime` configuration option to ↵reivilibre2021-11-234-15/+34
| | | | `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. (#11388)
* Add missing type hints to config base classes (#11377)Patrick Cloke2021-11-239-101/+165
|
* Remove code invalidated by deprecated config flag ↵Shay2021-11-231-32/+3
| | | | | | | | | | | | | | | | | | | | | 'trust_identity_servers_for_password_resets' (#11395) * remove background update code related to deprecated config flag * changelog entry * update changelog * Delete 11394.removal Duplicate, wrong number * add no-op background update and change newfragment so it will be consolidated with associated work * remove unused code * Remove code associated with deprecated flag from legacy docker dynamic config file Co-authored-by: reivilibre <oliverw@matrix.org>
* Merge branch 'master' into developSean Quah2021-11-234-51/+231
|\
| * 1.47.1Sean Quah2021-11-191-1/+1
| |
| * Prevent the media store from writing outside of the configured directorySean Quah2021-11-193-50/+230
| | | | | | | | | | Also tighten validation of server names by forbidding invalid characters in IPv6 addresses and empty domain labels.
* | Refactor the code to inject bundled relations during serialization. (#11408)Patrick Cloke2021-11-237-73/+91
| |
* | Add config for customizing the claim used for JWT logins. (#11361)Kostas2021-11-222-1/+11
| | | | | | | | | | Allows specifying a different claim (from the default "sub") to use when calculating the localpart of the Matrix ID used during the JWT login.
* | Store arbitrary relations from events. (#11391)Patrick Cloke2021-11-223-44/+75
| | | | | | | | | | Instead of only known relation types. This also reworks the background update for thread relations to crawl events and search for any relation type, not just threaded relations.
* | Add an admin API to run background jobs. (#11352)Dirk Klimpel2021-11-194-29/+102
| | | | | | | | | | | | Instead of having admins poke into the database directly. Can currently run jobs to populate stats and to populate the user directory.
* | Fix checking whether a room can be published on creation. (#11392)Patrick Cloke2021-11-192-23/+32
| | | | | | | | | | | | | | | | If `room_list_publication_rules` was configured with a rule with a non-wildcard alias and a room was created with an alias then an internal server error would have been thrown. This fixes the error and properly applies the publication rules during room creation.
* | Remove dead code from acme support. (#11393)Patrick Cloke2021-11-191-50/+0
| |
* | Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-11/+40
| |
* | Fix verification of objects signed with old local keys (#11379)Richard van der Hoff2021-11-191-28/+41
| | | | | | | | | | | | | | Fixes a bug introduced in #11129: objects signed by the local server, but with keys other than the current one, could not be successfully verified. We need to check the key id in the signature, and track down the right key.
* | Prevent historical state from being pushed to an application service via ↵Eric Eastwood2021-11-183-2/+38
| | | | | | | | | | | | | | | | | | `/transactions` (MSC2716) (#11265) Mark historical state from the MSC2716 `/batch_send` endpoint as `historical` which makes it `backfilled` and have a negative `stream_ordering` so it doesn't get queried by `/transactions`. Fix https://github.com/matrix-org/synapse/issues/11241 Complement tests: https://github.com/matrix-org/complement/pull/221
* | Remove legacy code related to deprecated ↵Shay2021-11-183-47/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `trust_identity_server_for_password_resets` config flag (#11333) * remove code legacy code related to deprecated config flag "trust_identity_server_for_password_resets" from synapse/config/emailconfig.py * remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/config/registration.py * remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/handlers/identity.py * add tests to ensure config error is thrown and synapse refuses to start when depreciated config flag is found * add changelog * slightly change behavior to only check for deprecated flag if set to 'true' * Update changelog.d/11333.misc Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* | Add dedicated admin API for blocking a room (#11324)Dirk Klimpel2021-11-183-0/+97
| |
* | Add/Unerase annotations to Module API (#11341)reivilibre2021-11-181-23/+134
| |
* | Rename `get_refresh_token_for_user_id` to `create_refresh_token_for_user_id` ↵reivilibre2021-11-182-3/+3
| | | | | | | | (#11370)
* | Use auto_attribs for RefreshTokenLookupResult (#11386)reivilibre2021-11-181-7/+7
| |
* | Do not allow MSC3440 threads to fork threads (#11161)Patrick Cloke2021-11-182-8/+113
| | | | | | | | | | | | | | | | | | | | | | Adds validation to the Client-Server API to ensure that the potential thread head does not relate to another event already. This results in not allowing a thread to "fork" into other threads. If the target event is unknown for some reason (maybe it isn't visible to your homeserver), but is the target of other events it is assumed that the thread can be created from it. Otherwise, it is rejected as an unknown event.
* | Docs: Quote wildcard `federation_certificate_verification_whitelist` (#11381)Nicolai Søborg2021-11-181-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Otherwise I get this beautiful stacktrace: ``` python3 -m synapse.app.homeserver --config-path /etc/matrix/homeserver.yaml Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/root/synapse/synapse/app/homeserver.py", line 455, in <module> main() File "/root/synapse/synapse/app/homeserver.py", line 445, in main hs = setup(sys.argv[1:]) File "/root/synapse/synapse/app/homeserver.py", line 345, in setup config = HomeServerConfig.load_or_generate_config( File "/root/synapse/synapse/config/_base.py", line 671, in load_or_generate_config config_dict = read_config_files(config_files) File "/root/synapse/synapse/config/_base.py", line 717, in read_config_files yaml_config = yaml.safe_load(file_stream) File "/root/synapse/env/lib/python3.8/site-packages/yaml/__init__.py", line 125, in safe_load return load(stream, SafeLoader) File "/root/synapse/env/lib/python3.8/site-packages/yaml/__init__.py", line 81, in load return loader.get_single_data() File "/root/synapse/env/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node document = self.compose_document() File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document node = self.compose_node(None, None) File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 82, in compose_node node = self.compose_sequence_node(anchor) File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 110, in compose_sequence_node while not self.check_event(SequenceEndEvent): File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 379, in parse_block_sequence_first_entry return self.parse_block_sequence_entry() File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 384, in parse_block_sequence_entry if not self.check_token(BlockEntryToken, BlockEndToken): File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 116, in check_token self.fetch_more_tokens() File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 227, in fetch_more_tokens return self.fetch_alias() File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 610, in fetch_alias self.tokens.append(self.scan_anchor(AliasToken)) File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 922, in scan_anchor raise ScannerError("while scanning an %s" % name, start_mark, yaml.scanner.ScannerError: while scanning an alias in "/etc/matrix/homeserver.yaml", line 614, column 5 expected alphabetic or numeric character, but found '.' in "/etc/matrix/homeserver.yaml", line 614, column 6 ``` Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
* | Add type annotations to `synapse.metrics` (#10847)Sean Quah2021-11-1710-85/+169
| |
* | Add support for `/_matrix/media/v3` APIs (#11371)Aaron R2021-11-173-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | * Add support for `/_matrix/media/v3` APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Update `workers.md` to use v3 client and media APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Merge branch 'master' into developDavid Robertson2021-11-171-1/+1
|\|
| * 1.47.0 v1.47.0David Robertson2021-11-171-1/+1
| |
* | Rename `get_access_token_for_user_id` method to ↵reivilibre2021-11-173-4/+4
| | | | | | | | `create_access_token_for_user_id` (#11369)
* | Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-163-20/+36
|\| | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * 1.47.0rc3Andrew Morgan2021-11-161-1/+1
| |
| * Rename `remove_deleted_devices_from_device_inbox` to ensure it is always run ↵Andrew Morgan2021-11-161-1/+13
| | | | | | | | | | (#11353) Co-authored-by: reivilibre <oliverw@matrix.org>
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-18/+22
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-151-1/+1
| |
* | Avoid sharing room hierarchy responses between users (#11355)Sean Quah2021-11-161-2/+9
| | | | | | | | | | Different users may be allowed to see different rooms within a space, so sharing responses between users is inadvisable.
* | Add support for `/_matrix/client/v3` APIs (#11318)Aaron R2021-11-163-3/+4
| | | | | | | | | | This is one of the changes required to support Matrix 1.1 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Add most missing type hints to synapse.util (#11328)Patrick Cloke2021-11-168-81/+157
| |
* | Properly register all callback hooks for legacy password authentication ↵reivilibre2021-11-161-13/+13
| | | | | | | | | | providers (#11340) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Add ability to un-shadow-ban via the admin API. (#11347)Patrick Cloke2021-11-162-3/+23
| |
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-4/+8
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Get directory db file to pass mypy (#11339)David Robertson2021-11-152-6/+7
| |
* | Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-151-2/+7
| | | | | | | | | | | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* | Remove unused tables `room_stats_historical` and `user_stats_historical` ↵Shay2021-11-122-1/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#11280) * remove unused tables room_stats_historical and user_stats_historical * update changelog number * Bump schema compat version comment * make linter happy * Update comment to give more info Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* | Annotations for state_deltas.py (#11316)David Robertson2021-11-121-3/+13
| | | | | | | | | | I was sad that I couldn't do better for `_curr_state_delta_stream_cache`. At least it's explicitly called out in a comment with #TODO.
* | Test room alias deletion (#11327)David Robertson2021-11-122-3/+10
| | | | | | | | | | | | | | | | | | | | * Prefer `HTTPStatus` over plain `int` This is an Opinion that no-one has seemed to object to yet. * `--disallow-untyped-defs` for `tests.rest.client.test_directory` * Improve synapse's annotations for deleting aliases * Test case for deleting a room alias * Changelog
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Add type hints to media repository storage module (#11311)Patrick Cloke2021-11-122-61/+88
| |
* | Attempt to annotate events_forward_extremities (#11314)David Robertson2021-11-122-7/+16
| | | | | | | | | | | | * Make DataStore inherit from EventForwardExtremitiesStore before CacheInvalidationWorkerStore the former implicitly inherits from the latter, so they should be ordered like this when used.
* | Misc typing fixes for `tests`, part 1 of N (#11323)David Robertson2021-11-122-2/+5
| | | | | | | | | | | | | | | | | | | | * Annotate HomeserverTestCase.servlets * Correct annotation of federation_auth_origin * Use AnyStr custom_headers instead of a Union This allows (str, str) and (bytes, bytes). This disallows (str, bytes) and (bytes, str) * DomainSpecificString.SIGIL is a ClassVar
* | Add more type hints to synapse.util. (#11321)Patrick Cloke2021-11-122-15/+23
| |
* | Convert delete room admin API to async endpoint (#11223)Dirk Klimpel2021-11-124-10/+432
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Annotations for user_erasure_store (#11313)David Robertson2021-11-111-4/+5
| | | | | | | | I'm not sure why this was excluded---it seemed to be passing for me. But it's easy enough to fixup.
* | Get db signatures file to pass mypy (#11312)David Robertson2021-11-112-9/+7
| |
* | Correct type hint for room_batch.py (#11310)David Robertson2021-11-111-10/+8
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-111-1/+1
| |
* | Add type hints to some storage classes (#11307)Patrick Cloke2021-11-117-47/+115
| |
* | Fix error in thumbnail generation (#11288)Neeeflix2021-11-101-4/+4
| | | | | | Signed-off-by: Jonas Zeunert <jonas@zeunert.org>
* | Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-1015-174/+219
| |
* | Add type hints to synapse._scripts (#11297)Patrick Cloke2021-11-102-18/+28
| |
* | Remove redundant parameters on `_check_event_auth` (#11292)Richard van der Hoff2021-11-101-10/+0
| | | | | | as of #11012, these parameters are unused.
* | Merge tag 'v1.47.0rc2' into developOlivier Wilkinson (reivilibre)2021-11-101-1/+1
|\| | | | | | | | | | | | | | | Synapse 1.47.0rc2 (2021-11-10) ============================== This fixes an issue with publishing the Debian packages for 1.47.0rc1. It is otherwise identical to 1.47.0rc1.
| * 1.47.0rc2Olivier Wilkinson (reivilibre)2021-11-101-1/+1
| |
| * Update __init__.py v1.47.0rc1Olivier Wilkinson (reivilibre)2021-11-091-1/+1
| |
* | Split out federated PDU retrieval into a non-cached version (#11242)Eric Eastwood2021-11-091-22/+58
| | | | | | | | Context: https://github.com/matrix-org/synapse/pull/11114/files#r741643968
* | Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-094-15/+71
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Allow admins to proactively block rooms (#11228)David Robertson2021-11-093-16/+63
| | | | | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-0910-75/+241
| | | | | | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* | Fix typo in `RelationAggregationPaginationServlet` error response (#11278)Andrew Morgan2021-11-091-1/+3
|/
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-091-86/+125
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Require body for read receipts with user-agent exceptions (#11157)rogersheu2021-11-091-1/+11
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Rename to more clear `get_insertion_event_id_by_batch_id` (MSC2716) (#11244)Eric Eastwood2021-11-083-3/+3
| | | | | `get_insertion_event_by_batch_id` -> `get_insertion_event_id_by_batch_id` Split out from https://github.com/matrix-org/synapse/pull/11114
* Add some background update admin APIs (#11263)Erik Johnston2021-11-084-18/+164
| | | Fixes #11259
* Fix typo in comment from #11255. (#11276)Patrick Cloke2021-11-081-1/+1
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-15 02:09:09 +0000