summary refs log tree commit diff
path: root/synapse/util/distributor.py (unfollow)
Commit message (Collapse)AuthorFilesLines
2020-11-13Fix changelogErik Johnston1-2/+2
2020-11-13 1.23.0rc1Erik Johnston71-70/+87
2020-11-13Migrate documentation `docs/admin_api/event_reports` to markdown (#8742)Dirk Klimpel3-165/+173
Related to #8714. `event_reports.rst` was introduced in Synapse 1.21.0.
2020-11-13SAML: add <mdui:UIInfo> element examples (#8718)Marcus Schopen3-0/+45
add some mdui:UIInfo element examples for saml2_config in homeserver.yaml
2020-11-13Updating README.rst (#8746)chagai952-6/+11
Minor corrections and advice... Should help beginners.
2020-11-13Add metrics for tracking 3PID /requestToken requests. (#8712)Erik Johnston4-0/+33
The main use case is to see how many requests are being made, and how many are second/third/etc attempts. If there are large number of retries then that likely indicates a delivery problem.
2020-11-13Fix port script so that it can be run again after failure. (#8755)Erik Johnston2-30/+55
If the script fails (or is CTRL-C'ed) between porting some of the events table and copying of the sequences then the port script will immediately die if run again due to the postgres DB having inconsistencies between sequences and tables. The fix is to move the porting of sequences to before porting the tables, so that there is never a period where the Postgres DB is inconsistent. To do that we need to change how we port the sequences so that it calculates the values from the SQLite DB rather than the Postgres DB. Fixes #8619
2020-11-13Cache event ID to auth event IDs lookups (#8752)Erik Johnston2-12/+71
This should hopefully speed up `get_auth_chain_difference` a bit in the case of repeated state res on the same rooms. `get_auth_chain_difference` does a breadth first walk of the auth graphs by repeatedly looking up events' auth events. Different state resolutions on the same room will end up doing a lot of the same event to auth events lookups, so by caching them we should speed things up in cases of repeated state resolutions on the same room.
2020-11-12Enable reconnection in DB pool (#8726)Erik Johnston2-1/+7
`adbapi.ConnectionPool` let's you turn on auto reconnect of DB connections. This is off by default. As far as I can tell if its not enabled dead connections never get removed from the pool. Maybe helps #8574
2020-11-11Fix port script fails when DB has no backfilled events. (#8729)Erik Johnston2-7/+6
Fixes #8618
2020-11-11Fix port script to handle foreign key constraints (#8730)Erik Johnston3-6/+63
2020-11-11Check support room has only two users before sending a notice (#8728)Will Hunt2-1/+2
* Check support room has only two users * Create 8728.bugfix * Update synapse/server_notices/server_notices_manager.py Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
2020-11-11Better error message when a remote resource uses invalid Content-Type (#8719)Andrew Morgan2-2/+9
2020-11-11Catch exceptions in password_providers (#8636)Nicolai Søborg2-4/+10
Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
2020-11-06Notes on SSO logins and media_repository worker (#8701)Marcus Schopen2-0/+6
If SSO login is used (e.g. SAML) in a multi worker setup, it should be mentioned that currently all SAML logins must run on the same worker, see https://github.com/matrix-org/synapse/issues/7530 Also, if you are using different ports (for example 443 and 8448) in a reverse proxy for client and federation, the path `/_matrix/media` on the client and federation port must point to the listener of the `media_repository` worker, otherwise you'll get a 404 on the federation port for the path `/_matrix/media`, if a remote server is trying to get the media object on federation port, see https://github.com/matrix-org/synapse/issues/8695
2020-11-06Add documentation about documentation to CONTRIBUTING.md (#8714)Andrew Morgan2-0/+19
This PR adds some documentation that: * Describes who the audience for the `docs/`, `docs/dev/` and `docs/admin/` directories are, as well as Synapse's wiki page. * Stresses that we'd like all documentation to be down in markdown.
2020-11-06Fix SIGHUP handler (#8697)Richard van der Hoff2-3/+3
Fixes: ``` builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given ```
2020-11-05Add an admin API for users' media statistics (#8700)Dirk Klimpel6-0/+820
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users. Related to #6094 It is the first API for statistics. Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats) Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-11-05Add `displayname` to Shared-Secret Registration for admins (#8722)Dirk Klimpel5-9/+138
Add `displayname` to Shared-Secret Registration for admins to `POST /_synapse/admin/v1/register`
2020-11-04Consolidate purge table lists to prevent desyncronisation (#8713)Andrew Morgan2-67/+39
I idly noticed that these lists were out of sync with each other, causing us to miss a table in a test case (`local_invites`). Let's consolidate this list instead to prevent this from happening in the future.
2020-11-03Remove the "draft" status of the Room Details Admin API (#8702)Dirk Klimpel2-3/+2
Fixes #8550
2020-11-03Block clients from sending server ACLs that lock the local server out. (#8708)Erik Johnston5-9/+80
Fixes #4042
2020-11-03Document how to set up multiple event persisters (#8706)Erik Johnston2-0/+13
2020-11-02grammarMatthew Hodgson1-1/+1
2020-11-02Improve error messages of non-str displayname/avatar_url (#8705)Andrew Morgan2-2/+7
This PR fixes two things: * Corrects the copy/paste error of telling the client their displayname is wrong when they are submitting an `avatar_url`. * Returns a `M_INVALID_PARAM` instead of `M_UNKNOWN` for non-str type parameters. Reported by @t3chguy.
2020-11-02Add base pushrule to notify for jitsi conferences (#8286)David Baker2-0/+25
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
2020-11-02Use Python 3.8 in Docker images by default (#8698)Dan Callahan2-1/+2
This bumps us closer to current Python without going all the way to 3.9. Fixes #8674 Signed-off-by: Dan Callahan <danc@element.io>
2020-11-02Improve start time by adding index to e2e_cross_signing_keys (#8694)Erik Johnston4-2/+36
We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
2020-11-02typoMatthew Hodgson1-1/+1
2020-11-02Fix typos in systemd-with-workers docAndrew Morgan1-2/+2
2020-11-02Fix typo in workers docAndrew Morgan1-1/+1
2020-10-30Fix error handling around when completing an AS transaction (#8693)Brendan Abolivier2-1/+2
2020-10-30Fix changelog v1.22.1 github/release-v1.22.1 release-v1.22.1Erik Johnston1-2/+2
2020-10-30use correct versionErik Johnston2-2/+2
2020-10-301.22.1Erik Johnston4-2/+16
2020-10-30Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff7-40/+32
Fix serialisation errors when using third-party event rules.
2020-10-30Tie together matches_user_in_member_list and get_users_in_room caches (#8676)Will Hunt2-4/+7
* Tie together matches_user_in_member_list and get_users_in_room * changelog * Remove type to fix mypy * Add `on_invalidate` to the function signature in the hopes that may make things work well * Remove **kwargs * Update 8676.bugfix
2020-10-30Improve the sample config for SSO (OIDC, SAML, and CAS). (#8635)Patrick Cloke5-104/+157
2020-10-30Implement and use an @lru_cache decorator (#8595)Richard van der Hoff4-61/+272
We don't always need the full power of a DeferredCache.
2020-10-30Fail test cases if they fail to await all awaitables (#8690)Patrick Cloke3-2/+39
2020-10-30Fix race for concurrent downloads of remote media. (#8682)Erik Johnston6-71/+431
Fixes #6755
2020-10-30Fix optional parameter in stripped state storage method (#8688)Andrew Morgan2-1/+2
Missed in #8671.
2020-10-29Fix unit tests (#8689)Erik Johnston2-1/+2
* Fix unit tests * Newsfile
2020-10-29Tie together matches_user_in_member_list and get_users_in_room caches (#8676)Will Hunt2-4/+7
* Tie together matches_user_in_member_list and get_users_in_room * changelog * Remove type to fix mypy * Add `on_invalidate` to the function signature in the hopes that may make things work well * Remove **kwargs * Update 8676.bugfix
2020-10-29Gracefully handle a pending logging connection during shutdown. (#8685)Patrick Cloke5-10/+36
2020-10-29Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston22-138/+197
another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29Fix cache call signature to accept `on_invalidate`. (#8684)Erik Johnston3-16/+35
Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.
2020-10-29Use `%r` rather than `%s` for stringifying events (#8679)Richard van der Hoff2-2/+3
otherwise non-state events get written as `<FrozenEvent ... state_key='None'>` which is indistinguishable from state events with the actual state_key `None`.
2020-10-29Optimise createRoom with multiple invites (#8559)Richard van der Hoff3-13/+25
By not dropping the membership lock between invites, we can stop joins from grabbing the lock when we're half-done and slowing the whole thing down.
2020-10-29Support generating structured logs in addition to standard logs. (#8607)Patrick Cloke19-1011/+706
This modifies the configuration of structured logging to be usable from the standard Python logging configuration. This also separates the formatting of logs from the transport allowing JSON logs to files or standard logs to sockets.
2020-10-29Don't require hiredis to run unit tests (#8680)Erik Johnston2-1/+11
2020-10-28remove unused importsRichard van der Hoff1-2/+0
2020-10-28Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff7-38/+32
Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
2020-10-28Add type hints to application services. (#8655)Patrick Cloke5-79/+122
2020-10-28Add an admin APIs to allow server admins to list users' pushers (#8610)Dirk Klimpel5-1/+259
Add an admin API `GET /_synapse/admin/v1/users/<user_id>/pushers` like https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers
2020-10-28Run mypy as part of the lint.sh script. (#8633)kleph3-2/+4
2020-10-28Don't pull event from DB when handling replication traffic. (#8669)Erik Johnston5-31/+87
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-27Tell Black to format code for Python 3.5 (#8664)Dan Callahan10-11/+12
This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27Minor updates to docs on how to run tests (#8666)Dan Callahan3-10/+6
The test runner isn't present in the `[all]` set of extras, so the previous instructions did not work without also installing `[test]`. Note that this does not include the `[lint]` extras, since those do not install on all supported Python versions (specifically, isort 5.x requires Python 3.6, while we still support 3.5). Instructions for that are included in our pull request template, so we should be fine there. I've also dropped the `--no-use-pep517` arg to `pip install` since it seems to have been added to address a temporary regression in pip 19.1 which was fixed in pip 19.1.1 the following month. Lastly, updated the example output of the test suite to set more realistic expectations around run time. Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27Note support for Python 3.9 (#8665)Dan Callahan4-2/+4
As expected, all tests pass locally without modification. Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27Cross-link documentation to the prometheus recording rules. (#8667)Michael Kaye3-1/+4
2020-10-27Abstract code for stripping room state into a separate method (#8671)Andrew Morgan3-29/+61
This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change. For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included. The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event. This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem. In essence this is just moving some inline code to a reusable store method.
2020-10-27Don't unnecessarily start bg process in replication sending loop. (#8670)Erik Johnston2-0/+11
2020-10-27Don't unnecessarily start bg process while handling typing. (#8668)Erik Johnston2-8/+14
There's no point starting a background process when all its going to do is bail if federation isn't enabled.
2020-10-27Add admin API to list users' local media (#8647)Dirk Klimpel8-1/+494
Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
2020-10-271.22.0 v1.22.0 github/release-v1.22.0 release-v1.22.0Erik Johnston3-1/+13
2020-10-26e2e: ensure we have both master and self-signing key (#8455)Jonas Jelten2-5/+23
it seems to be possible that only one of them ends up to be cached. when this was the case, the missing one was not fetched via federation, and clients then failed to validate cross-signed devices. Signed-off-by: Jonas Jelten <jj@sft.lol>
2020-10-26Add type hints for account validity handler (#8620)Patrick Cloke6-12/+31
This also fixes a bug by fixing handling of an account which doesn't expire.
2020-10-26Split admin API for reported events into a detail and a list view (#8539)Dirk Klimpel6-86/+411
Split admin API for reported events in detail und list view. API was introduced with #8217 in synapse v.1.21.0. It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview. The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`. It is similar to room and users API. It is a kind of regression in `GET /_synapse/admin/v1/event_reports`. `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec). Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-10-26Added basic instructions for Azure AD to OpenId documentation (#8582)Peter Krantz2-0/+27
Signed-off-by: Peter Krantz peter.krantz@gmail.com
2020-10-26Add an admin api to delete local media. (#8519)Dirk Klimpel7-3/+868
Related to: #6459, #3479 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server.
2020-10-26Fix filepath of Dex example config (#8657)Andrew Morgan2-3/+3
2020-10-26Fix a bug in the joined_rooms admin API (#8643)Dirk Klimpel3-4/+20
If the user was not in any rooms then the API returned the same error as if the user did not exist.
2020-10-26Fixup changelog v1.22.0rc2Erik Johnston1-3/+1
2020-10-26Expand changelog entryErik Johnston1-1/+1
2020-10-261.22.0rc2Erik Johnston5-4/+12
2020-10-26Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt4-13/+85
2020-10-26Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt4-13/+85
2020-10-26Add field `total` to device list in admin API (#8644)Dirk Klimpel4-2/+23
2020-10-26Check status codes that profile handler returns (#8580)LEdoian2-0/+8
Fixes #8520 Signed-off-by: Pavel Turinsky <pavel.turinsky@matfyz.cz> Co-authored-by: Erik Johnston <erikj@jki.re>
2020-10-26Properly handle presence events for application services. (#8656)Patrick Cloke2-4/+6
2020-10-26Fix user_daily_visits to not have duplicate rows for UA. (#8654)Erik Johnston2-4/+6
* Fix user_daily_visits to not have duplicate rows for UA. Fixes #8641. * Newsfile * Fix typo. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-26