summary refs log tree commit diff
path: root/synapse/notifier.py (unfollow)
Commit message (Collapse)AuthorFilesLines
2020-11-13Add knock membership events to stats generationAndrew Morgan3-0/+23
2020-11-13Implement locally rescinding a federated knockAndrew Morgan2-28/+100
As mentioned in the MSC, a user can rescind (take back) a knock while it is pending by sending a leave event to the room. This will set their membership to leave instead of knock. Now, this functionality already worked before this commit for rooms that the homeserver was already in. What didn't work was: * Rescinding a knock over federation to a room with active homeservers * Rescinding a knock over federation to a room with inactive homeservers This commit addresses the second bullet point, and leaves the first bullet point as a TODO (as it is an edge case an not immediately obvious how it would be done). What this commit does is crib off the same functionality as locally rejecting an invite. That occurs when we are unable to contact the homeserver that originally sent us an invite. Instead an out-of-band leave membership event will be generated and sent to clients locally. The same is happening here. You can mostly ignore the new generate_local_out_of_band_membership methods, those are just some structural bits to allow us to call that method from RoomMemberHandler. The real meat of this commit is moving about and adding some logic in `update_membership_locked`, specifically for when we're updating a user's membership to "leave". There was already some code in there to check whether the room to send the leave to was a room the homeserver is not currently a part of. In that case, we'd remote reject the knock. This commit just extends that to also rescind knocks if the user's membership in the room is currently "knock". We skip the remote attempt for now and go straight to generating a local leave membership event.
2020-11-13Generalise _locally_reject_inviteAndrew Morgan1-18/+20
We'll be using this to rescind knocks locally too, so generalise the method.
2020-11-13Add some handy db methods for knockingAndrew Morgan1-0/+33
This just adds some database methods that we'll need to use when implementing rescinding of knocks of clients. They are equivalent to their invite-related counterparts.
2020-11-13Auto-add displaynames to knock events if they're missingAndrew Morgan1-6/+4
Tiny commit to just bring knocking up to feature parity.
2020-11-13Extend sync to inform clients about the progress of their knocksAndrew Morgan3-23/+127
So we've got federation so that homeservers can communicate knocking information between them - but how does that information actually get down to the client? The client knows that it knocked successfully from a 200 in its original request, but what else does it need? This commit adds a new "knock" section to /sync (in addition to "invite", "join", and "leave") all help give the client the information it needs. The new "knock" section is used for sending down the stripped state events we collected earlier. The client will use these to display the room and its metadata in a little "pending knocks" section or similar. This is all this commit adds. If the user's knock has been accepted or rejected, they will receive that information in the "join" or "leave" sections of /sync. Most of this code is just cribbing off the invite and join sync code yet again, with some minor differences. For instance, we don't need to exclude knock events from sync if the sender is in your ignore list, as you are the only ones that can send knocks for yourself. The structure of the "knock" dict in sync is modeled after "invite", as clients also receive stripped state in that. The structure can be viewed in the linked MSC.
2020-11-13Send stripped state events back to the knocking homeserverAndrew Morgan4-9/+46
Here we finally send the stripped state events back to the knocking homeserver, which then ingests and stores those events. A future commit will actually start sending those events down /sync to the relevant user.
2020-11-13Federation: make_knock and send_knock implementationsAndrew Morgan7-4/+359
Most of this is explained in the linked MSC (and don't miss the sequence diagram in the MSC comments), but roughly knocking takes inspiration from room joins and room invites. This commit is the room join stuff. First the knocking homeserver POSTs to the make_knock endpoint on another homeserver. The other homeserver will send back a knock event that is valid for the knocking user and the room that they are knocking on. The knocking homeserver will sign the event and send it back, before the other homeserver takes that event and then sends it into the room on the knocking homeserver's behalf. It's worth noting that the accepting/rejecting knocks all happen over existing room invite/leave flows. A homeserver rescinding its knock as well is also just sending a leave. Once the event has been inserted into the room, the homeserver that's in the room will send back a 200 and an empty JSON dict to confirm everything went well to the knocker. In a future commit, this dict will instead be filled with some stripped state events from the room which the knocking homeserver will pass back to the knocking user. And yes, the logging statements in this commit are intentional. They're consistent with the rest of the file :)
2020-11-11Rename maybe_store_room_on_{invite,outlier_membership}Andrew Morgan3-13/+18
There's a handy function called maybe_store_room_on_invite which allows us to create an entry in the rooms table for a room and its version for which we aren't joined to yet, but we can reference when ingesting events about. This is currently used for invites where we receive some stripped state about the room and pass it down via /sync to the client, without us being in the room yet. There is a similar requirement for knocking, where we will eventually do the same thing, and need an entry in the rooms table as well. Thus, reusing this function works, however its name needs to be generalised a bit. So that is what this commit does.
2020-11-11Add room_knock_state_types config optionAndrew Morgan2-8/+34
This option serves the same purpose as the existing room_invite_state_types option, which defines what state events are sent over to a user that is invited to a room. This information is necessary for the user - who isn't in the room yet - to get some metadata about the room in order to display it in a pretty fashion in the user's pending-invites list. It includes information such as the room's name, avatar, topic, canonical alias, room encryption state etc. as well as the invite membership event which the invited user's homeserver can reference. This new option is the exact same, but is sent by a homeserver in the room to the knocker during the knock process. This option will actually be utilised in a later commit.
2020-11-11Add CS /_matrix/client/r0/knock/{roomIdOrAlias} endpointAndrew Morgan3-1/+106
We're ditching the usual idea of having two endpoints for each membership-related endpoint as per the MSC. Thus knocking only gets the more powerful variant (the one that supports room aliases as well as IDs. The reason is also optional. The other small change is just to ensure displaynames get added to the content of this particular membership event.
2020-11-11Update the event auth rules for knockingAndrew Morgan1-4/+18
Hopefully most of these changes are explained through the added comments and error messages. The changes are also described conceptually in the MSC: https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#join-rules
2020-11-10Add `xyz.amorgan.knock` /versions stringAndrew Morgan1-0/+2
2020-11-10Add new experimental room version for knockingAndrew Morgan1-2/+23
2020-11-06Notes on SSO logins and media_repository worker (#8701)Marcus Schopen2-0/+6
If SSO login is used (e.g. SAML) in a multi worker setup, it should be mentioned that currently all SAML logins must run on the same worker, see https://github.com/matrix-org/synapse/issues/7530 Also, if you are using different ports (for example 443 and 8448) in a reverse proxy for client and federation, the path `/_matrix/media` on the client and federation port must point to the listener of the `media_repository` worker, otherwise you'll get a 404 on the federation port for the path `/_matrix/media`, if a remote server is trying to get the media object on federation port, see https://github.com/matrix-org/synapse/issues/8695
2020-11-06Add documentation about documentation to CONTRIBUTING.md (#8714)Andrew Morgan2-0/+19
This PR adds some documentation that: * Describes who the audience for the `docs/`, `docs/dev/` and `docs/admin/` directories are, as well as Synapse's wiki page. * Stresses that we'd like all documentation to be down in markdown.
2020-11-06Fix SIGHUP handler (#8697)Richard van der Hoff2-3/+3
Fixes: ``` builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given ```
2020-11-05Add an admin API for users' media statistics (#8700)Dirk Klimpel6-0/+820
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users. Related to #6094 It is the first API for statistics. Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats) Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-11-05Add `displayname` to Shared-Secret Registration for admins (#8722)Dirk Klimpel5-9/+138
Add `displayname` to Shared-Secret Registration for admins to `POST /_synapse/admin/v1/register`
2020-11-04Consolidate purge table lists to prevent desyncronisation (#8713)Andrew Morgan2-67/+39
I idly noticed that these lists were out of sync with each other, causing us to miss a table in a test case (`local_invites`). Let's consolidate this list instead to prevent this from happening in the future.
2020-11-03Remove the "draft" status of the Room Details Admin API (#8702)Dirk Klimpel2-3/+2
Fixes #8550
2020-11-03Block clients from sending server ACLs that lock the local server out. (#8708)Erik Johnston5-9/+80
Fixes #4042
2020-11-03Document how to set up multiple event persisters (#8706)Erik Johnston2-0/+13
2020-11-02grammarMatthew Hodgson1-1/+1
2020-11-02Improve error messages of non-str displayname/avatar_url (#8705)Andrew Morgan2-2/+7
This PR fixes two things: * Corrects the copy/paste error of telling the client their displayname is wrong when they are submitting an `avatar_url`. * Returns a `M_INVALID_PARAM` instead of `M_UNKNOWN` for non-str type parameters. Reported by @t3chguy.
2020-11-02Add base pushrule to notify for jitsi conferences (#8286)David Baker2-0/+25
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
2020-11-02Use Python 3.8 in Docker images by default (#8698)Dan Callahan2-1/+2
This bumps us closer to current Python without going all the way to 3.9. Fixes #8674 Signed-off-by: Dan Callahan <danc@element.io>
2020-11-02Improve start time by adding index to e2e_cross_signing_keys (#8694)Erik Johnston4-2/+36
We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
2020-11-02typoMatthew Hodgson1-1/+1
2020-11-02Fix typos in systemd-with-workers docAndrew Morgan1-2/+2
2020-11-02Fix typo in workers docAndrew Morgan1-1/+1
2020-10-30Fix error handling around when completing an AS transaction (#8693)Brendan Abolivier2-1/+2
2020-10-30Fix changelog v1.22.1 github/release-v1.22.1 release-v1.22.1Erik Johnston1-2/+2
2020-10-30use correct versionErik Johnston2-2/+2
2020-10-301.22.1Erik Johnston4-2/+16
2020-10-30Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff7-40/+32
Fix serialisation errors when using third-party event rules.
2020-10-30Tie together matches_user_in_member_list and get_users_in_room caches (#8676)Will Hunt2-4/+7
* Tie together matches_user_in_member_list and get_users_in_room * changelog * Remove type to fix mypy * Add `on_invalidate` to the function signature in the hopes that may make things work well * Remove **kwargs * Update 8676.bugfix
2020-10-30Improve the sample config for SSO (OIDC, SAML, and CAS). (#8635)Patrick Cloke5-104/+157
2020-10-30Implement and use an @lru_cache decorator (#8595)Richard van der Hoff4-61/+272
We don't always need the full power of a DeferredCache.
2020-10-30Fail test cases if they fail to await all awaitables (#8690)Patrick Cloke3-2/+39
2020-10-30Fix race for concurrent downloads of remote media. (#8682)Erik Johnston6-71/+431
Fixes #6755
2020-10-30Fix optional parameter in stripped state storage method (#8688)Andrew Morgan2-1/+2
Missed in #8671.
2020-10-29Fix unit tests (#8689)Erik Johnston2-1/+2
* Fix unit tests * Newsfile
2020-10-29Tie together matches_user_in_member_list and get_users_in_room caches (#8676)Will Hunt2-4/+7
* Tie together matches_user_in_member_list and get_users_in_room * changelog * Remove type to fix mypy * Add `on_invalidate` to the function signature in the hopes that may make things work well * Remove **kwargs * Update 8676.bugfix
2020-10-29Gracefully handle a pending logging connection during shutdown. (#8685)Patrick Cloke5-10/+36
2020-10-29Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston22-138/+197
another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29Fix cache call signature to accept `on_invalidate`. (#8684)Erik Johnston3-16/+35
Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.
2020-10-29Use `%r` rather than `%s` for stringifying events (#8679)Richard van der Hoff2-2/+3
otherwise non-state events get written as `<FrozenEvent ... state_key='None'>` which is indistinguishable from state events with the actual state_key `None`.
2020-10-29Optimise createRoom with multiple invites (#8559)Richard van der Hoff3-13/+25
By not dropping the membership lock between invites, we can stop joins from grabbing the lock when we're half-done and slowing the whole thing down.
2020-10-29Support generating structured logs in addition to standard logs. (#8607)Patrick Cloke19-1011/+706
This modifies the configuration of structured logging to be usable from the standard Python logging configuration. This also separates the formatting of logs from the transport allowing JSON logs to files or standard logs to sockets.
2020-10-29Don't require hiredis to run unit tests (#8680)Erik Johnston2-1/+11
2020-10-28remove unused importsRichard van der Hoff1-2/+0
2020-10-28Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff7-38/+32
Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
2020-10-28Add type hints to application services. (#8655)Patrick Cloke5-79/+122
2020-10-28Add an admin APIs to allow server admins to list users' pushers (#8610)Dirk Klimpel5-1/+259
Add an admin API `GET /_synapse/admin/v1/users/<user_id>/pushers` like https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers
2020-10-28Run mypy as part of the lint.sh script. (#8633)kleph3-2/+4
2020-10-28Don't pull event from DB when handling replication traffic. (#8669)Erik Johnston5-31/+87
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-27Tell Black to format code for Python 3.5 (#8664)Dan Callahan10-11/+12
This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27Minor updates to docs on how to run tests (#8666)Dan Callahan3-10/+6
The test runner isn't present in the `[all]` set of extras, so the previous instructions did not work without also installing `[test]`. Note that this does not include the `[lint]` extras, since those do not install on all supported Python versions (specifically, isort 5.x requires Python 3.6, while we still support 3.5). Instructions for that are included in our pull request template, so we should be fine there. I've also dropped the `--no-use-pep517` arg to `pip install` since it seems to have been added to address a temporary regression in pip 19.1 which was fixed in pip 19.1.1 the following month. Lastly, updated the example output of the test suite to set more realistic expectations around run time. Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27Note support for Python 3.9 (#8665)Dan Callahan4-2/+4
As expected, all tests pass locally without modification. Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27Cross-link documentation to the prometheus recording rules. (#8667)Michael Kaye3-1/+4
2020-10-27Abstract code for stripping room state into a separate method (#8671)Andrew Morgan3-29/+61
This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change. For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included. The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event. This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem. In essence this is just moving some inline code to a reusable store method.
2020-10-27Don't unnecessarily start bg process in replication sending loop. (#8670)Erik Johnston2-0/+11
2020-10-27Don't unnecessarily start bg process while handling typing. (#8668)Erik Johnston2-8/+14
There's no point starting a background process when all its going to do is bail if federation isn't enabled.
2020-10-27Add admin API to list users' local media (#8647)Dirk Klimpel8-1/+494
Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
2020-10-271.22.0 v1.22.0 github/release-v1.22.0 release-v1.22.0Erik Johnston3-1/+13
2020-10-26e2e: ensure we have both master and self-signing key (#8455)Jonas Jelten2-5/+23
it seems to be possible that only one of them ends up to be cached. when this was the case, the missing one was not fetched via federation, and clients then failed to validate cross-signed devices. Signed-off-by: Jonas Jelten <jj@sft.lol>
2020-10-26Add type hints for account validity handler (#8620)Patrick Cloke6-12/+31
This also fixes a bug by fixing handling of an account which doesn't expire.
2020-10-26Split admin API for reported events into a detail and a list view (#8539)Dirk Klimpel6-86/+411
Split admin API for reported events in detail und list view. API was introduced with #8217 in synapse v.1.21.0. It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview. The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`. It is similar to room and users API. It is a kind of regression in `GET /_synapse/admin/v1/event_reports`. `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec). Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-10-26Added basic instructions for Azure AD to OpenId documentation (#8582)Peter Krantz2-0/+27
Signed-off-by: Peter Krantz peter.krantz@gmail.com
2020-10-26Add an admin api to delete local media. (#8519)Dirk Klimpel7-3/+868
Related to: #6459, #3479 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server.
2020-10-26Fix filepath of Dex example config (#8657)Andrew Morgan2-3/+3
2020-10-26Fix a bug in the joined_rooms admin API (#8643)Dirk Klimpel3-4/+20
If the user was not in any rooms then the API returned the same error as if the user did not exist.
2020-10-26Fixup changelog v1.22.0rc2Erik Johnston1-3/+1
2020-10-26Expand changelog entryErik Johnston1-1/+1
2020-10-261.22.0rc2Erik Johnston5-4/+12
2020-10-26Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt4-13/+85
2020-10-26Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt4-13/+85
2020-10-26Add field `total` to device list in admin API (#8644)Dirk Klimpel4-2/+23
2020-10-26Check status codes that profile handler returns (#8580)LEdoian2-0/+8
Fixes #8520 Signed-off-by: Pavel Turinsky <pavel.turinsky@matfyz.cz> Co-authored-by: Erik Johnston <erikj@jki.re>
2020-10-26Properly handle presence events for application services. (#8656)Patrick Cloke2-4/+6
2020-10-26Fix user_daily_visits to not have duplicate rows for UA. (#8654)Erik Johnston2-4/+6