summary refs log tree commit diff
path: root/synapse/handlers/federation.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
...
| * Add `include_event_in_state` to _get_state_for_room (#6521)Richard van der Hoff2019-12-161-18/+21
| | | | | | | | | | | | Make it return the state *after* the requested event, rather than the one before it. This is a bit easier and requires fewer calls to get_events_from_store_or_dest.
| * Move get_state methods into FederationHandler (#6503)Richard van der Hoff2019-12-161-6/+95
| | | | | | | | | | This is a non-functional refactor as a precursor to some other work.
* | Exclude rejected state events when calculating state at backwards extrems ↵Richard van der Hoff2019-12-161-1/+1
| | | | | | | | | | (#6527) This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
* | Persist auth/state events at backwards extremities when we fetch them (#6526)Richard van der Hoff2019-12-161-167/+80
| | | | | | The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
* | sanity-checking for events used in state res (#6531)Richard van der Hoff2019-12-131-0/+1
| | | | | | | | | | | | | | When we perform state resolution, check that all of the events involved are in the right room.
* | Check the room_id of events when fetching room state/auth (#6524)Richard van der Hoff2019-12-121-23/+51
| | | | | | | | | | | | | | | | | | | | When we request the state/auth_events to populate a backwards extremity (on backfill or in the case of missing events in a transaction push), we should check that the returned events are in the right room rather than blindly using them in the room state or auth chain. Given that _get_events_from_store_or_dest takes a room_id, it seems clear that it should be sanity-checking the room_id of the requested events, so let's do it there.
* | Add `include_event_in_state` to _get_state_for_room (#6521)Richard van der Hoff2019-12-111-22/+28
| | | | | | | | | | Make it return the state *after* the requested event, rather than the one before it. This is a bit easier and requires fewer calls to get_events_from_store_or_dest.
* | convert to async: FederationHandler._process_received_pduRichard van der Hoff2019-12-111-11/+10
| | | | | | | | also fix user_joined_room to consistently return deferreds
* | convert to async: FederationHandler._get_state_for_roomRichard van der Hoff2019-12-111-21/+21
| | | | | | | | ... and _get_events_from_store_or_dest
* | convert to async: FederationHandler.on_receive_pduRichard van der Hoff2019-12-111-27/+22
| | | | | | | | | | | | | | and associated functions: * on_receive_pdu * handle_queued_pdus * get_missing_events_for_pdu
* | Convert federation backfill to asyncRichard van der Hoff2019-12-111-25/+22
| | | | | | | | | | | | | | | | | | PaginationHandler.get_messages is only called by RoomMessageListRestServlet, which is async. Chase the code path down from there: - FederationHandler.maybe_backfill (and nested try_backfill) - FederationHandler.backfill
* | Clean up some logging (#6515)Richard van der Hoff2019-12-111-18/+19
| | | | | | | | This just makes some of the logging easier to follow when things start going wrong.
* | Prevent redacted events from appearing in message search (#6377)Andrew Morgan2019-12-111-2/+5
| |
* | Prevent message search in upgraded rooms we're not in (#6385)Andrew Morgan2019-12-111-2/+2
| |
* | Move get_state methods into FederationHandler (#6503)Richard van der Hoff2019-12-101-6/+95
|/ | | | | This is a non-functional refactor as a precursor to some other work.
* Stronger typing in the federation handler (#6480)Richard van der Hoff2019-12-051-24/+57
| | | | | replace the event_info dict with an attrs thing
* Sanity-check the rooms of auth events before pulling them in. (#6472)Richard van der Hoff2019-12-051-9/+25
|
* get rid of (most of) have_events from ↵Richard van der Hoff2019-12-041-38/+24
| | | | | | | | | | | | | _update_auth_events_and_context_for_auth (#6468) have_events was a map from event_id to rejection reason (or None) for events which are in our local database. It was used as filter on the list of event_ids being passed into get_events_as_list. However, since get_events_as_list will ignore any event_ids that are unknown or rejected, we can equivalently just leave it to get_events_as_list to do the filtering. That means that we don't have to keep `have_events` up-to-date, and can use `have_seen_events` instead of `get_seen_events_with_rejection` in the one place we do need it.
* Add ephemeral messages support (MSC2228) (#6409)Brendan Abolivier2019-12-031-0/+8
| | | | | | | | Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are: * the feature is hidden behind a configuration flag (`enable_ephemeral_messages`) * self-destruction doesn't happen for state events * only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one) * doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
* Propagate reason in remotely rejected invitesErik Johnston2019-11-281-2/+2
|
* Merge pull request #6358 from matrix-org/babolivier/message_retentionBrendan Abolivier2019-11-271-2/+2
|\ | | | | Implement message retention policies (MSC1763)
| * Merge branch 'develop' into babolivier/message_retentionBrendan Abolivier2019-11-261-3/+8
| |\
| * | Fix 3PID invite exchangeBrendan Abolivier2019-11-191-1/+1
| | |
| * | Implement per-room message retention policiesBrendan Abolivier2019-11-041-1/+1
| | |
* | | remove confusing fixmeRichard van der Hoff2019-11-261-6/+0
| | |
* | | Merge remote-tracking branch 'origin/develop' into rav/event_auth/4Richard van der Hoff2019-11-181-2/+2
|\ \ \ | | |/ | |/|
| * | Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-2/+2
| | |
* | | Use get_events_as_list rather than lots of calls to get_eventRichard van der Hoff2019-11-081-16/+8
| | | | | | | | | | | | It's more efficient and clearer.
* | | Update some docstrings and commentsRichard van der Hoff2019-11-081-8/+31
| | |
* | | Simplify _update_auth_events_and_context_for_authRichard van der Hoff2019-11-081-11/+9
|/ / | | | | | | | | move event_key calculation into _update_context_for_auth_events, since it's only used there.
* | Fix bug which caused rejected events to be stored with the wrong room state ↵Richard van der Hoff2019-11-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | (#6320) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.
* | Add some checks that we aren't using state from rejected events (#6330)Richard van der Hoff2019-11-051-1/+5
|/ | | | | | | | | | | | | * Raise an exception if accessing state for rejected events Add some sanity checks on accessing state_group etc for rejected events. * Skip calculating push actions for rejected events It didn't actually cause any bugs, because rejected events get filtered out at various later points, but there's not point in trying to calculate the push actions for a rejected event.
* Factor out an _AsyncEventContextImpl (#6298)Richard van der Hoff2019-11-011-19/+19
| | | | | | The intention here is to make it clearer which fields we can expect to be populated when: notably, that the _event_type etc aren't used for the synchronous impl of EventContext.
* Depublish a room from the public rooms list when it is upgraded (#6232)Andrew Morgan2019-11-011-1/+29
|
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-4/+5
| | | * update version of black and also fix the mypy config being overridden
* Merge pull request #6294 from matrix-org/erikj/add_state_storageErik Johnston2019-10-311-9/+10
|\ | | | | Add StateGroupStorage interface
| * Port to use state storageErik Johnston2019-10-301-9/+10
| |
* | Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-16/+20
|/ | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-301-3/+19
|\ | | | | | | erikj/split_out_persistence_store
| * Fix log line that was printing undefined value (#6278)Andrew Morgan2019-10-301-1/+1
| |
| * Improve signature checking on some federation APIs (#6262)Richard van der Hoff2019-10-281-2/+18
| | | | | | | | | | Make sure that we check that events sent over /send_join, /send_leave, and /invite, are correctly signed and come from the expected servers.
* | Use new EventPersistenceStoreErik Johnston2019-10-231-1/+2
|/
* Remove Auth.check method (#6217)Richard van der Hoff2019-10-181-3/+4
| | | This method was somewhat redundant, and confusing.
* rip out some unreachable codeRichard van der Hoff2019-10-171-102/+0
| | | | The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
* TypoBrendan Abolivier2019-10-041-1/+1
|
* LintBrendan Abolivier2019-10-041-3/+1
|
* Incorporate reviewBrendan Abolivier2019-10-041-14/+8
|
* Add test caseBrendan Abolivier2019-10-031-1/+1
|
* LintBrendan Abolivier2019-10-021-1/+3
|
* Don't 500 code when trying to exchange a revoked 3PID inviteBrendan Abolivier2019-10-021-2/+13
| | | | | | | | While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state. When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events. When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it). This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
* Remove origin parameter from add_display_name_to_third_party_invite and add ↵Andrew Morgan2019-09-111-1/+6
| | | | | | | params to docstring (#6010) Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument. Since it's not used anywhere in the handler function though, we should remove it from the function arguments.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Implement a structured logging output system. (#5680)Amber Brown2019-08-281-2/+3
|
* Merge pull request #5790 from matrix-org/erikj/groups_request_errorsErik Johnston2019-08-011-0/+3
|\ | | | | Handle RequestSendFailed exception correctly in more places.
| * Don't log as exception when failing durig backfillErik Johnston2019-07-301-0/+3
| |
* | Room Complexity Client Implementation (#5783)Amber Brown2019-07-301-0/+25
|/
* Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatchRichard van der Hoff2019-07-261-2/+35
|\ | | | | Log when we receive a /make_* request from a different origin
| * Log when we receive a /make_* request from a different originRichard van der Hoff2019-07-261-2/+35
| |
* | Replace returnValue with return (#5736)Amber Brown2019-07-231-42/+40
|/
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-21/+22
|
* Run Black. (#5482)Amber Brown2019-06-201-521/+363
|
* Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_errorErik Johnston2019-06-191-9/+19
|\ | | | | Handle the case of `get_missing_events` failing
| * Fix loglineErik Johnston2019-06-181-1/+1
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-06-181-132/+289
| |\ | | | | | | | | | erikj/fix_get_missing_events_error
| * | Handle the case of `get_missing_events` failingErik Johnston2019-04-101-9/+20
| | | | | | | | | | | | | | | | | | | | | Currently if a call to `/get_missing_events` fails we log an exception and stop processing the top level event we received over federation. Instead let's try and handle it sensibly given it is a somewhat expected failure mode.
* | | Add some logging to 3pid invite sig verification (#5015)David Baker2019-06-181-8/+38
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | I had to add quite a lot of logging to diagnose a problem with 3pid invites - we only logged the one failure which isn't all that informative. NB. I'm not convinced the logic of this loop is right: I think it should just accept a single valid signature from a trusted source rather than fail if *any* signature is invalid. Also it should probably not skip the rest of middle loop if a check fails? However, I'm deliberately not changing the logic here.
* | Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_stateErik Johnston2019-06-171-6/+0
|\ \ | | | | | | Fix 3PID invite room state over federation.
| * | Fix 3PID invite room state over federation.Erik Johnston2019-06-141-6/+0
| | | | | | | | | | | | | | | | | | | | | Fixes that when a user exchanges a 3PID invite for a proper invite over federation it does not include the `invite_room_state` key. This was due to synapse incorrectly sending out two invite requests.
* | | Add plugin APIs for implementations of custom event rules.Brendan Abolivier2019-06-141-2/+66
|/ /
* | Add logging when request fails and clarify we ignore errors.Erik Johnston2019-06-051-4/+12
| |
* | Fix handling of failures when calling /event_auth.Erik Johnston2019-06-031-12/+38
| | | | | | | | | | | | | | | | | | | | When processing an incoming event over federation, we may try and resolve any unexpected differences in auth events. This is a non-essential process and so should not stop the processing of the event if it fails (e.g. due to the remote disappearing or not implementing the necessary endpoints). Fixes #3330
* | Simplifications and comments in do_auth (#5227)Richard van der Hoff2019-05-231-120/+181
| | | | | | | | | | I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.
* | Exclude soft-failed events from fwd-extremity candidates. (#5146)Richard van der Hoff2019-05-211-1/+6
|/ | | | | | | | When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.
* Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-011-8/+5
| | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
* Factor out soft fail checksErik Johnston2019-03-081-5/+22
|
* Implement soft failErik Johnston2019-03-061-1/+76
|
* clarify commentsErik Johnston2019-03-051-8/+11
|
* Only check history visibility when filteringErik Johnston2019-03-041-1/+3
| | | | | | When filtering events to send to server we check more than just history visibility. However when deciding whether to backfill or not we only care about the history visibility.
* s/get_forward_events/get_successor_events/Erik Johnston2019-03-041-1/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-03-041-3/+31
|\ | | | | | | erikj/stop_fed_not_in_room
| * Fix typoErik Johnston2019-02-251-1/+1
| |
| * More commentsErik Johnston2019-02-251-0/+4
| |
| * Assert rather than clobber the valuesErik Johnston2019-02-251-2/+2
| |
| * Add comments and paranoiaErik Johnston2019-02-251-2/+23
| |
| * Fix backfill storing incorrect state for eventsErik Johnston2019-02-221-1/+4
| |
* | Update commentsErik Johnston2019-02-271-3/+13
| |
* | Stop backpaginating when events not visibleErik Johnston2019-02-201-0/+31
|/
* Fix flake8 (#4519)Amber Brown2019-01-301-1/+1
|
* Use snder and not event ID domain to check if oursErik Johnston2019-01-291-0/+8
| | | | | | The transaction queue only sends out events that we generate. This was done by checking domain of event ID, but that can no longer be used. Instead, we may as well use the sender field.
* Fixup calls to `comput_event_signature`Erik Johnston2019-01-291-1/+1
| | | | | | | | We currently pass FrozenEvent instead of `dict` to `compute_event_signature`, which works by accident due to `dict(event)` producing the correct result. This fixes PR #4493 commit 855a151
* Merge pull request #4494 from matrix-org/erikj/fixup_event_validatorErik Johnston2019-01-291-2/+5
|\ | | | | Split up event validation between event and builder
| * Split up event validation between event and builderErik Johnston2019-01-281-2/+5
| | | | | | | | | | | | | | | | | | The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.
* | Pass through room version to event authErik Johnston2019-01-251-8/+12
|/
* Require event format version to parse or create eventsErik Johnston2019-01-251-28/+44
|
* Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813Erik Johnston2019-01-251-31/+18
|\
| * Merge pull request #4448 from matrix-org/erikj/get_pdu_versionsErik Johnston2019-01-241-2/+15
| |\ | | | | | | Add room_version param to get_pdu
| | * Review commentsErik Johnston2019-01-241-0/+5
| | |
| | * Add room_version param to get_pduErik Johnston2019-01-231-2/+10
| | | | | | | | | | | | | | | When we add new event format we'll need to know the event format or room version when parsing events.
| * | Remove unecessary setting of outlier bitErik Johnston2019-01-241-2/+0
| | |
| * | Use term 'out of band membership' insteadErik Johnston2019-01-241-2/+2
| | |
| * | Clarify the invite flowsErik Johnston2019-01-231-1/+11
| | |
| * | Remove unnecessary '_sign_event'Erik Johnston2019-01-231-14/+0
| | |
| * | Store rejected remote invite events as outliersErik Johnston2019-01-231-32/+12
| |/ | | | | | | | | | | | | | | | | | | | | Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.
* / Implement MSC 1813 - Add room version to make APIsErik Johnston2019-01-231-1/+1
|/ | | | | We also implement `make_membership_event` converting the returned room version to an event format version.
* Remove hack to support rejoining roomsErik Johnston2018-11-091-74/+42
|
* Simplify to always drop events if server isn't in the roomErik Johnston2018-11-091-14/+9
|
* Add helpers for getting prev and auth events (#4139)Erik Johnston2018-11-061-24/+24
| | | | | | | * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.
* Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebaseErik Johnston2018-10-241-15/+15
|\ | | | | Add v2 state resolution algorithm
| * CommentErik Johnston2018-10-241-2/+5
| |
| * Rename resolve_events_with_factoryErik Johnston2018-10-241-2/+2
| |
| * Fix up use of resolve_events_with_factoryErik Johnston2018-10-161-13/+10
| |
* | Remove redundant run_as_background_process() from pusherpoolRichard van der Hoff2018-10-221-2/+2
| | | | | | | | | | | | | | `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.
* | Fix incorrect truncation in get_missing_eventsRichard van der Hoff2018-10-161-7/+5
|/ | | | | | | | | It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.
* Avoid reraise, to improve stacktracesRichard van der Hoff2018-10-011-10/+10
|
* Include event when resolving state for missing prevsRichard van der Hoff2018-09-271-2/+16
| | | | | | | | | | | If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.
* Include state from remote servers in pdu handlingRichard van der Hoff2018-09-271-4/+19
| | | | | | | | If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.
* Fix "unhashable type: 'list'" exception in federation handlingRichard van der Hoff2018-09-271-7/+12
| | | | | | | | | | | | | get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.
* more commentsRichard van der Hoff2018-09-271-4/+3
|
* Clarifications in FederationHandlerRichard van der Hoff2018-09-271-2/+11
| | | | | | | * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)
* Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2Richard van der Hoff2018-09-271-0/+4
|\ | | | | Logging improvements
| * Logging improvementsRichard van der Hoff2018-09-261-0/+4
| | | | | | | | Some logging tweaks to help with debugging incoming federation transactions
* | Include eventid in log lines when processing incoming federation ↵Richard van der Hoff2018-09-271-26/+39
|/ | | | | | | | | | | transactions (#3959) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.
* Comments and interface cleanup for on_receive_pduRichard van der Hoff2018-09-201-23/+46
| | | | | | | | Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.
* Improve the logging when handling a federation transaction (#3904)Richard van der Hoff2018-09-191-55/+109
| | | | | | | | | | Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.
* Bump timeout on get_missing_events requestRichard van der Hoff2018-09-181-1/+30
|
* Port handlers/ to Python 3 (#3803)Amber Brown2018-09-071-3/+3
|
* Remove unnecessary resolve_events_with_state_mapErik Johnston2018-08-221-1/+1
| | | | | We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-201-20/+59
|\ | | | | | | erikj/refactor_state_handler
| * Fix logcontexts for running pushersRichard van der Hoff2018-08-171-2/+1
| | | | | | | | | | | | | | First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-151-1/+1
| |\ | | | | | | | | | erikj/split_federation
| | * Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
| | |
| * | Use federation handler function rather than duplicateErik Johnston2018-08-151-7/+7
| | | | | | | | | | | | This involves renaming _persist_events to be a public function.
| * | Move clean_room_for_join to masterErik Johnston2018-08-091-2/+14
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-091-2/+11
| |\| | | | | | | | | | erikj/split_federation
| * | Add EDU/query handling over replicationErik Johnston2018-08-061-11/+13
| | |
| * | Add replication APIs for persisting federation eventsErik Johnston2018-08-061-9/+35
| | |
* | | Remove redundant room_version checksErik Johnston2018-08-201-1/+1
| | |
* | | Choose state algorithm based on room versionErik Johnston2018-08-091-2/+6
| |/ |/|
* | include known room versions in outgoing make_joinsRichard van der Hoff2018-08-061-2/+11
|/
* Merge branch 'master' into developRichard van der Hoff2018-08-021-6/+23
|\
| * Avoid extra db lookupsRichard van der Hoff2018-08-021-27/+11
| | | | | | | | | | Since we're about to look up the events themselves anyway, we can skip the extra db queries here.
| * Validation for events/rooms in fed requestsRichard van der Hoff2018-08-021-1/+34
| | | | | | | | | | | | | | When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)
* | Merge pull request #3621 from matrix-org/erikj/split_fed_storeErik Johnston2018-08-021-87/+88
|\ \ | | | | | | Split out DB writes in federation handler
| * | update docsErik Johnston2018-08-011-3/+5
| | |
| * | _persist_auth_tree no longer returns anythingErik Johnston2018-08-011-1/+1
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-301-1/+1
| |\ \ | | | | | | | | | | | | erikj/split_fed_store
| * | | Split out DB writes in federation handlerErik Johnston2018-07-251-83/+82
| | | | | | | | | | | | | | | | | | | | | | | | This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.
* | | | rename replication_layer to federation_clientRichard van der Hoff2018-07-311-14/+14
| |/ / |/| | | | | | | | | | | I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.
* | | Fix a minor documentation typo in on_make_leaveTravis Ralston2018-07-251-1/+1
|/ /
* | Handle delta_ids being None in _update_context_for_auth_eventsRichard van der Hoff2018-07-231-7/+6
| | | | | | | | | | | | | | it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with https://github.com/matrix-org/synapse/issues/3364)
* | Add support for updating stateErik Johnston2018-07-231-9/+23
| |
* | Use new gettersErik Johnston2018-07-231-7/+16
| |
* | Make the rest of the .iterwhatever go away (#3562)Amber Brown2018-07-211-9/+9
| |
* | Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacksRichard van der Hoff2018-07-181-43/+0
|\ \ | |/ |/| Remove event re-signing hacks
| * Merge branch 'develop' into rav/drop_re_signing_hacksRichard van der Hoff2018-07-041-51/+153
| |\
| * | Remove event re-signing hacksRichard van der Hoff2018-06-071-43/+0
| | | | | | | | | | | | | | | | | | These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.
* | | Move filter_events_for_server out of FederationHandlerRichard van der Hoff2018-07-161-138/+6
| | | | | | | | | | | | for easier unit testing.
* | | run isortAmber Brown2018-07-091-18/+22
| |/ |/|
* | try and clean upAmber Brown2018-06-271-3/+6
| |
* | pep8Amber Brown2018-06-271-6/+4
| |
* | cleanupsAmber Brown2018-06-271-13/+15
| |
* | handle federation not telling us about prev_eventsAmber Brown2018-06-271-25/+62
| |
* | Actually fix itErik Johnston2018-06-251-3/+3
| |
* | CommentErik Johnston2018-06-251-0/+8
| |
* | Fix bug with assuming wrong typeErik Johnston2018-06-251-2/+2
| |
* | Add fast path to _filter_events_for_serverErik Johnston2018-06-251-1/+34
| | | | | | | | | | | | Most rooms have a trivial history visibility like "shared" or "world_readable", especially large rooms, so lets not bother getting the full membership of those rooms in that case.
* | Revert "Revert "Merge pull request #3431 from ↵Erik Johnston2018-06-251-0/+13
| | | | | | | | | | | | matrix-org/rav/erasure_visibility"" This reverts commit 1d009013b3c3e814177afc59f066e02a202b21cd.
* | Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"Richard van der Hoff2018-06-221-13/+0
| | | | | | | | | | This reverts commit ce0d911156b355c5bf452120bfb08653dad96497, reversing changes made to b4a5d767a94f1680d07edfd583aae54ce422573e.
* | Merge pull request #3431 from matrix-org/rav/erasure_visibilityErik Johnston2018-06-221-0/+13
|\ \ | | | | | | Support hiding events from deleted users
| * | Filter out erased messagesRichard van der Hoff2018-06-121-0/+13
| | | | | | | | | | | | Redact any messges sent by erased users.
* | | Merge pull request #3428 from matrix-org/erikj/persisted_pduErik Johnston2018-06-221-18/+26
|\| | | | | | | | Simplify get_persisted_pdu
| * | simplify get_persisted_pduRichard van der Hoff2018-06-121-18/+26
| | | | | | | | | | | | | | | it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.
* | | Remove run_on_reactor (#3395)Amber Brown2018-06-141-5/+1
|/ /
* / Fix event filtering in get_missing_events handlerRichard van der Hoff2018-06-081-0/+4
|/
* Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-311-9/+8
| | | | they're not meant to be lazy (#3307)
* Misc. py3 fixesAdrian Tschira2018-05-241-4/+5
| | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* Merge pull request #3261 from matrix-org/erikj/pagination_fixesErik Johnston2018-05-241-9/+22
|\ | | | | Fix federation backfill bugs
| * Use iter* methodsErik Johnston2018-05-221-7/+7
| |
| * Fix that states is a dict of dictsErik Johnston2018-05-221-1/+4
| |
| * get_domains_from_state returns list of tuplesErik Johnston2018-05-221-2/+12
| |
* | s/values/itervalues/Erik Johnston2018-05-231-1/+1
| |
* | Use iter* methods for _filter_events_for_serverErik Johnston2018-05-231-4/+4
|/
* Make sure we reject attempts to invite the notices userRichard van der Hoff2018-05-181-0/+8
|
* Merge pull request #3153 from NotAFile/py3-httplibRichard van der Hoff2018-04-301-3/+3
|\ | | | | move httplib import to six
| * move httplib import to sixAdrian Tschira2018-04-281-3/+3
| |
* | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-6/+10
|\|
| * reraise exceptions more carefullyRichard van der Hoff2018-04-271-6/+10
| | | | | | | | | | | | | | We need to be careful (under python 2, at least) that when we reraise an exception after doing some error handling, we actually reraise the original exception rather than anything that might have been raised (and handled) during the error handling.
* | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-6/+10
|/ | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* Merge pull request #3118 from matrix-org/rav/reject_prev_eventsRichard van der Hoff2018-04-231-7/+71
|\ | | | | Reject events which have lots of prev_events
| * Remove spurious paramRichard van der Hoff2018-04-231-1/+0
| |
| * accept stupid events over backfillRichard van der Hoff2018-04-201-4/+9
| |
| * Add some commentsRichard van der Hoff2018-04-181-3/+12
| |
| * Check events on backfill tooRichard van der Hoff2018-04-181-15/+42
| |
| * Reject events which have too many auth_events or prev_eventsRichard van der Hoff2018-04-181-4/+28
| | | | | | | | | | ... this should protect us from being dossed by people making silly events (deliberately or otherwise)
* | Refactor store.have_eventsRichard van der Hoff2018-04-201-19/+12
|/ | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.
* s/replication_client/federation_client/Erik Johnston2018-03-131-1/+1
|
* Split replication layer into twoErik Johnston2018-03-131-3/+1
|
* Move RoomMemberHandler out of HandlersErik Johnston2018-03-011-2/+2
|
* Delete from push_actions_staging in federation tooErik Johnston2018-02-201-9/+17
|
* Merge pull request #2847 from matrix-org/erikj/separate_event_creationErik Johnston2018-02-061-10/+9
|\ | | | | Split event creation into a separate handler
| * Update copyrightErik Johnston2018-02-061-0/+1
| |
| * s/_create_new_client_event/create_new_client_event/Erik Johnston2018-02-061-5/+5
| |
| * Update places where we create eventsErik Johnston2018-02-051-10/+8
| |
* | Store state groups separately from events (#2784)Erik Johnston2018-02-061-7/+17
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Rename resolve_state_groups -> resolve_state_groups_for_eventsRichard van der Hoff2018-02-011-6/+5
| | | | (to make way for a method that actually just does the state group resolution)
* Add federation_domain_whitelist option (#2820)Matthew Hodgson2018-01-221-0/+4
| | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
* s/items/iteritems/Richard van der Hoff2017-11-081-2/+2
|
* Update deltas when doing auth resolutionRichard van der Hoff2017-11-071-3/+7
| | | | | Fixes a bug where the persisted state groups were different to those actually being used after auth resolution.
* factor out _update_context_for_auth_eventsRichard van der Hoff2017-11-071-20/+42
| | | | This is duplicated, so let's factor it out before fixing it
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-8/+8
| | | | what could possibly go wrong
* Merge pull request #2490 from matrix-org/erikj/drop_left_room_eventsErik Johnston2017-10-101-0/+22
|\ | | | | Ignore incoming events for rooms that we have left
| * Update commentsErik Johnston2017-10-031-1/+1
| |
| * Remove spurious line feedErik Johnston2017-10-031-1/+0
| |
| * Ignore incoming events for rooms that we have leftErik Johnston2017-10-031-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When synapse receives an event for a room its not in over federation, it double checks with the remote server to see if it is in fact in the room. This is done so that if the server has forgotten about the room (usually as a result of the database being dropped) it can recover from it. However, in the presence of state resets in large rooms, this can cause a lot of work for servers that have legitimately left. As a hacky solution that supports both cases we drop incoming events for rooms that we have explicitly left. This means that we no longer support the case of servers having forgotten that they've rejoined a room, but that is sufficiently rare that we're not going to support it for now.
* | Fix up deferred handling in federation.pyRichard van der Hoff2017-10-061-24/+21
| | | | | | | | | | | | | | * Avoid preserve_context_over_deferred, which is broken * set consumeErrors=True on defer.gatherResults, to avoid spurious "unhandled failure" erros
* | Spam checking: add the invitee to user_may_inviteDavid Baker2017-10-051-5/+7
| |
* | pass room id tooDavid Baker2017-10-031-1/+1
| |
* | Fix param name & lintDavid Baker2017-10-031-2/+4
| |
* | Allow spam checker to reject invites tooDavid Baker2017-10-031-0/+4
|/
* Add a config option to block all room invites (#2457)Richard van der Hoff2017-09-191-0/+3
| | | | | - allows sysadmins the ability to lock down their servers so that people can't send their users room invites.
* add some comments to on_exchange_third_party_invite_requestRichard van der Hoff2017-09-191-0/+11
|
* Fix overzealous kicking of guest users (#2453)Richard van der Hoff2017-09-191-1/+1
| | | We should only kick guest users if the guest access event is authorised.
* Don't compute push actions for backfilled eventsErik Johnston2017-07-111-1/+1
|
* Fix bug where pusherpool didn't start and broke some roomsErik Johnston2017-07-061-1/+2
| | | | | | | | Since we didn't instansiate the PusherPool at start time it could fail at run time, which it did for some users. This may or may not fix things for those users, but it should happen at start time and stop the server from starting.
* Improve the error handling for bad invites received over federationMark Haines2017-06-301-2/+17
|
* Add shutdown room APIErik Johnston2017-06-191-0/+4
|
* CommentErik Johnston2017-06-091-0/+2
|
* Mark remote invite rejections as outliersErik Johnston2017-06-091-0/+1
|
* Only store event_auth for state eventsErik Johnston2017-05-241-6/+14
|
* Store ActionGenerator in HomeServerErik Johnston2017-05-191-3/+2
|
* Remove redundant reset of destination timersErik Johnston2017-05-051-7/+0
|
* Merge together redundant calculations/loggingErik Johnston2017-05-031-9/+2
|
* FixupErik Johnston2017-04-281-10/+4
|
* Remove unncessary call in _get_missing_events_for_pduErik Johnston2017-04-281-13/+11
|
* Don't fetch state for missing events that we fetchedErik Johnston2017-04-281-0/+6
|
* TypoErik Johnston2017-04-271-1/+1
|
* Add some extra logging for edge cases of federationErik Johnston2017-04-271-2/+22
|
* Remove redundant try/except clausesRichard van der Hoff2017-04-211-23/+11
| | | | | | | | | | The `except SynapseError` clauses were pointless because the wrapped functions would never throw a `SynapseError` (they either throw a `CodeMessageException` or a `RuntimeError`). The `except CodeMessageException` is now also pointless because the caller treats all exceptions equally, so we may as well just throw the `CodeMessageException`.
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2017-04-111-4/+4
|\
| * When we do an invite rejection, save the signed leave event to the dbRichard van der Hoff2017-04-071-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | During a rejection of an invite received over federation, we ask a remote server to make us a `leave` event, then sign it, then send that with `send_leave`. We were saving the *unsigned* version of the event (which has a different event id to the signed version) to our db (and sending it to the clients), whereas other servers in the room will have seen the *signed* version. We're not aware of any actual problems that caused, except that it makes the database confusing to look at and generally leaves the room in a weird state.
* | Merge pull request #2095 from matrix-org/rav/cull_log_preservesRichard van der Hoff2017-04-031-24/+19
|\ \ | |/ |/| Cull spurious PreserveLoggingContexts
| * Remove more spurious `PreserveLoggingContext`sRichard van der Hoff2017-04-031-24/+19
| | | | | | | | | | Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`; there is no problem if the logcontext is set when calling it.
* | Accept join events from all serversRichard van der Hoff2017-04-031-3/+13
|/ | | | | | | Make sure that we accept join events from any server, rather than just the origin server, to make the federation join dance work correctly. (Fixes #1893).
* Merge pull request #2027 from matrix-org/rav/logcontext_leaksRichard van der Hoff2017-03-201-2/+3
|\ | | | | A few fixes to logcontext things
| * Stop preserve_fn leaking context into the reactorRichard van der Hoff2017-03-181-2/+3
| | | | | | | | | | | | | | | | Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the reactor, and add a test for it. Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do the same thing but had its own, different, set of bugs.
* | Refactoring and cleanupsRichard van der Hoff2017-03-171-0/+10
|/ | | | | | | | | | | | A few non-functional changes: * A bunch of docstrings to document types * Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more readable. * Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid mind-bending conditional. * Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid mind-bending conditional.
* Queue up federation PDUs while a room join is in progressRichard van der Hoff2017-03-151-24/+44
| | | | | | | This just takes the existing `room_queues` logic and moves it out to `on_receive_pdu` instead of `_process_received_pdu`, which ensures that we don't start trying to fetch prev_events and whathaveyou until the join has completed.
* Move FederationServer._handle_new_pdu to FederationHandlerRichard van der Hoff2017-03-091-5/+197
| | | | | | | | | | | Unfortunately this significantly increases the size of the already-rather-big FederationHandler, but the code fits more naturally here, and it paves the way for the tighter integration that I need between handling incoming PDUs and doing the join dance. Other than renaming the existing `FederationHandler.on_receive_pdu` to `_process_received_pdu` to make way for it, this just consists of the move, and replacing `self.handler` with `self` and `self` with `self.replication_layer`.
* Pop with default value to stop throwingErik Johnston2017-02-281-1/+1
|
* Pop rather than del from dictErik Johnston2017-02-271-1/+1
|