summary refs log tree commit diff
path: root/synapse/federation (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #5984 from ↵Erik Johnston2019-09-052-3/+13
|\ | | | | | | | | matrix-org/joriks/opentracing_link_send_to_edu_contexts Link the send loop with the edus contexts
| * use access methods (duh..)Jorik Schellekens2019-09-051-1/+3
| | | | | | Co-Authored-By: Erik Johnston <erik@matrix.org>
| * Link the send loop with the edus contextsJorik Schellekens2019-09-052-3/+11
| | | | | | | | | | | | The contexts were being filtered too early so the send loop wasn't being linked to them unless the destination was whitelisted.
* | Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-051-1/+5
|/
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-302-15/+15
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Merge pull request #5859 from matrix-org/rei/msc2197reivilibre2019-08-282-15/+67
|\ | | | | MSC2197 Search Filters over Federation
| * Use MSC2197 on stable prefix as it has almost finished FCPOlivier Wilkinson (reivilibre)2019-08-202-26/+4
| | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
| * Add support for inbound MSC2197 requests on unstable Federation APIOlivier Wilkinson (reivilibre)2019-08-151-1/+59
| | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
| * Support MSC2197 outbound with unstable prefixOlivier Wilkinson (reivilibre)2019-08-151-15/+31
| | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | Propagate opentracing contexts through EDUs (#5852)Jorik Schellekens2019-08-223-80/+108
| | | | | | | | | | Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Opentracing across workers (#5771)Jorik Schellekens2019-08-221-15/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* | Opentrace e2e keys (#5855)Jorik Schellekens2019-08-221-0/+3
| | | | | | Add opentracing tags and logs for e2e keys
* | Opentracing misc (#5856)Jorik Schellekens2019-08-161-6/+7
| | | | | | | | | | | | | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* | Wrap `get_local_public_room_list` call in `maybeDeferred` because itOlivier Wilkinson (reivilibre)2019-08-141-2/+8
|/ | | | | | | is cached and so does not always return a `Deferred`. `await` does not silently pass-through non-Deferreds like `yield` used to. Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Return 502 not 500 when failing to reach any remote server.Erik Johnston2019-08-011-4/+3
|
* Room Complexity Client Implementation (#5783)Amber Brown2019-07-302-7/+60
|
* Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatchRichard van der Hoff2019-07-261-2/+2
|\ | | | | Log when we receive a /make_* request from a different origin
| * Log when we receive a /make_* request from a different originRichard van der Hoff2019-07-261-2/+2
| |
* | Merge tag 'v1.2.0rc2' into developAndrew Morgan2019-07-241-1/+3
|\| | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
| * Fix servlet metric names (#5734)Jorik Schellekens2019-07-241-1/+3
| | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths
* | Replace returnValue with return (#5736)Amber Brown2019-07-236-90/+75
|/
* Convert synapse.federation.transport.server to async (#5689)Richard van der Hoff2019-07-181-242/+188
| | | | | | | | | | | | | | | | * Convert BaseFederationServlet._wrap to async Empirically, this fixes some lost stacktraces. It should be safe because the wrapped function is called from JsonResource._async_render, which is already async. * Convert the rest of synapse.federation.transport.server to async We may as well do the whole file while we're here. * changelog * flake8
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-111-5/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* remove dead transaction persist code (#5622)Richard van der Hoff2019-07-052-41/+0
| | | | this hasn't done anything for years
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-047-22/+32
|
* Split public rooms directory auth config in twoBrendan Abolivier2019-06-241-4/+4
|
* Fix logging error when a tampered event is detected. (#5500)Richard van der Hoff2019-06-211-1/+0
|
* Run Black. (#5482)Amber Brown2019-06-2011-752/+613
|
* Add experimental option to reduce extremities.Erik Johnston2019-06-181-0/+3
| | | | | | | Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.
* Clean up code for sending federation EDUs. (#5381)Richard van der Hoff2019-06-131-14/+26
| | | | This code confused the hell out of me today. Split _get_new_device_messages into its two (unrelated) parts.
* Prevent multiple device list updates from breaking a batch send (#5156)Andrew Morgan2019-06-061-2/+3
| | | fixes #5153
* Associate a request_name with each verify request, for loggingRichard van der Hoff2019-06-052-1/+5
| | | | | | | Also: * rename VerifyKeyRequest->VerifyJsonRequest * calculate key_ids on VerifyJsonRequest construction * refactor things to pass around VerifyJsonRequests instead of 4-tuples
* Add a test room version where we enforce key validity (#5348)Richard van der Hoff2019-06-051-15/+24
|
* Clean up debug logging (#5347)Richard van der Hoff2019-06-051-3/+13
| | | Remove some spurious stuff, clarify some other stuff
* Fix failure to fetch batches of PDUs (#5342)Richard van der Hoff2019-06-041-0/+1
| | | | | FederationClient.get_pdu is called in a loop to fetch a batch of PDUs. A failure to fetch one should not result in a failure of the whole batch. Add the missing `continue`.
* Rename get_events->get_events_from_store_or_dest (#5344)Richard van der Hoff2019-06-041-20/+13
| | | | | We have too many things called get_event, and it's hard to figure out what we mean. Also remove some unused params from the signature, and add some logging.
* Enforce validity period on server_keys for fed requests. (#5321)Richard van der Hoff2019-06-032-3/+5
| | | | | | | | When handling incoming federation requests, make sure that we have an up-to-date copy of the signing key. We do not yet enforce the validity period for event signatures.
* Implement the SHHS complexity API (#5216)Amber Brown2019-05-301-1/+30
|
* Fix error handling for rooms whose versions are unknown. (#5219)Richard van der Hoff2019-05-211-1/+13
| | | | | | | If we remove support for a particular room version, we should behave more gracefully. This should make client requests fail with a 400 rather than a 500, and will ignore individiual PDUs in a federation transaction, rather than the whole transaction.
* Make all the rate limiting options more consistent (#5181)Amber Brown2019-05-151-5/+1
|
* Run `black` on per_destination_queueRichard van der Hoff2019-05-091-35/+39
| | | | ... mostly to fix pep8 fails
* Limit the number of EDUs in transactions to 100 as expected by receiver (#5138)Quentin Dufour2019-05-091-26/+30
| | | Fixes #3951.
* add options to require an access_token to GET /profile and /publicRooms on ↵Matthew Hodgson2019-05-081-0/+10
| | | | | | | | | | | | | | | | | CS API (#5083) This commit adds two config options: * `restrict_public_rooms_to_local_users` Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API. * `require_auth_for_profile_requests` When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301. MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though. Groups have been intentionally omitted from this commit.
* more logging improvementsRichard van der Hoff2019-04-251-8/+11
|
* remove extraneous exception loggingRichard van der Hoff2019-04-251-2/+2
|
* Clarify logging when PDU signature checking failsRichard van der Hoff2019-04-251-0/+19
|
* Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-014-25/+27
| | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
* Use event streams to calculate presenceErik Johnston2019-03-272-3/+89
| | | | | | | | | | | | | | | | | Primarily this fixes a bug in the handling of remote users joining a room where the server sent out the presence for all local users in the room to all servers in the room. We also change to using the state delta stream, rather than the distributor, as it will make it easier to split processing out of the master process (as well as being more flexible). Finally, when sending presence states to newly joined servers we filter out old presence states to reduce the number sent. Initially we filter out states that are offline and have a last active more than a week ago, though this can be changed down the line. Fixes #3962
* Make federation endpoints more tolerant of trailing slashes v2 (#4935)Andrew Morgan2019-03-262-8/+8
| | | Redo of https://github.com/matrix-org/synapse/pull/4840
* Remove trailing slashes from outbound federation requests and retry on 400 ↵Andrew Morgan2019-03-211-7/+14
|\ | | | | | | | | (#4840) As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.
| * Switch to wrapper function around _send_requestAndrew Morgan2019-03-131-5/+5
| |
| * Cleaner way of implementing trailing slashesAndrew Morgan2019-03-121-5/+10
| |
| * Merge branch 'develop' into anoa/trailing_slashes_clientAndrew Morgan2019-03-111-7/+7
| |\
| * | lintAndrew Morgan2019-03-081-1/+3
| | |
| * | Retry certain federation requests on 404Andrew Morgan2019-03-081-5/+5
| | |
| * | Remove trailing slashes from outbound federation requestsAndrew Morgan2019-03-081-7/+7
| | |
* | | Batch up outgoing read-receipts to reduce federation traffic. (#4890)Richard van der Hoff2019-03-202-21/+158
| | | | | | | | | | | | Rate-limit outgoing read-receipts as per #4730.
* | | Merge pull request #4855 from matrix-org/rav/refactor_transaction_queueRichard van der Hoff2019-03-155-758/+860
|\ \ \ | | | | | | | | Split TransactionQueue up
| * | | Rename and move the classesRichard van der Hoff2019-03-135-808/+860
| | | |
| * | | Factor per-destination stuff out of TransactionQueueRichard van der Hoff2019-03-131-132/+182
| | | | | | | | | | | | | | | | This is easier than having to have a million fields keyed on destination.
* | | | Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_workerRichard van der Hoff2019-03-152-0/+44
|\| | | | | | | | | | | Move client receipt processing to federation sender worker.
| * | | Move client receipt processing to federation sender worker.Richard van der Hoff2019-03-132-0/+44
| | |/ | |/| | | | | | | | | | This is mostly a prerequisite for #4730, but also fits with the general theme of "move everything off the master that we possibly can".
* / | Revert "Make federation endpoints more tolerant of trailing slashes for some ↵Erik Johnston2019-03-142-8/+8
|/ / | | | | | | | | | | endpoints (#4793)" This reverts commit 290552fd836f4ae2dc1d893a7f72f7fff85365d3.
* / Make federation endpoints more tolerant of trailing slashes for some ↵Andrew Morgan2019-03-112-8/+8
|/ | | | | endpoints (#4793) Server side of a solution towards #3622.
* Remove unnecessary dollar signsAndrew Morgan2019-03-041-22/+22
| | | | | A dollar sign is already appended to the end of each PATH, so there's no need to add one in the PATH declaration as well.
* Avoid rebuilding Edu objects in worker mode (#4770)Richard van der Hoff2019-03-042-14/+31
| | | | | In worker mode, on the federation sender, when we receive an edu for sending over the replication socket, it is parsed into an Edu object. There is no point extracting the contents of it so that we can then immediately build another Edu.
* Make 'event_id' a required parameter in federated state requests (#4741)Amber Brown2019-02-271-2/+2
| | | | | | | | | | | | | | * make 'event_id' a required parameter in federated state requests As per the spec: https://matrix.org/docs/spec/server_server/r0.1.1.html#id40 Signed-off-by: Joseph Weston <joseph@weston.cloud> * add changelog entry for bugfix Signed-off-by: Joseph Weston <joseph@weston.cloud> * Update server.py
* When presence is enabled don't send over replicationErik Johnston2019-02-271-0/+3
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Andrew Morgan2019-02-267-243/+588
|\ | | | | | | anoa/public_rooms_federate_develop
| * Revert "Prevent showing non-fed rooms in fed /publicRooms"Andrew Morgan2019-02-261-2/+1
| |
| * Merge pull request #4736 from matrix-org/anoa/public_rooms_federateAndrew Morgan2019-02-261-1/+2
| |\ | | | | | | Config option to prevent showing non-fed rooms in fed /publicRooms
| | * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-02-257-241/+561
| | |\ | | | | | | | | | | | | anoa/public_rooms_federate
| | * | Don't restrict non-fed rooms over client APIsAndrew Morgan2019-02-251-1/+2
| | | |
| * | | Merge pull request #4737 from matrix-org/erikj/failure_log_tbErik Johnston2019-02-251-2/+3
| |\ \ \ | | | | | | | | | | Log tracebacks correctly
| | * | | Log tracebacks correctlyErik Johnston2019-02-251-2/+3
| | | |/ | | |/|
| * / | MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite APIErik Johnston2019-02-232-2/+26
| |/ /
| * | Add metrics for number of outgoing EDUs, by type (#4695)Richard van der Hoff2019-02-201-4/+18
| | |
| * | Merge pull request #4420 from matrix-org/jaywink/openid-listenerErik Johnston2019-02-111-42/+95
| |\ \ | | | | | | | | New listener resource for the federation API "openid/userinfo" endpoint
| | * | Document `servlet_groups` parametersJason Robinson2019-01-231-0/+23
| | | | | | | | | | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
| | * | Split federation OpenID userinfo endpoint out of the federation resourceJason Robinson2019-01-231-42/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the OpenID userinfo endpoint to be active even if the federation resource is not active. The OpenID userinfo endpoint is called by integration managers to verify user actions using the client API OpenID access token. Without this verification, the integration manager cannot know that the access token is valid. The OpenID userinfo endpoint will be loaded in the case that either "federation" or "openid" resource is defined. The new "openid" resource is defaulted to active in default configuration. Signed-off-by: Jason Robinson <jasonr@matrix.org>
| * | | Reject large transactions on federation (#4513)Andrew Morgan2019-01-311-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Reject large transactions on federation * Add changelog * lint * Simplify large transaction handling
| * | | No vdh tests!Erik Johnston2019-01-291-2/+1
| | | |
| * | | Enable room version v3Erik Johnston2019-01-291-2/+9
| | | |
| * | | Update commentErik Johnston2019-01-291-3/+3
| | | |
| * | | Only check event ID domain for signatures for V1 eventsErik Johnston2019-01-293-31/+44
| | | | | | | | | | | | | | | | | | | | In future version events won't have an event ID, so we won't be able to do this check.
| * | | Use event origin for filtering incoming eventsErik Johnston2019-01-291-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | We only process events sent to us from a server if the event ID matches the server, to help guard against federation storms. We replace this with a check against the event origin.
| * | | Use snder and not event ID domain to check if oursErik Johnston2019-01-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | The transaction queue only sends out events that we generate. This was done by checking domain of event ID, but that can no longer be used. Instead, we may as well use the sender field.
| * | | Merge pull request #4481 from matrix-org/erikj/event_builderErik Johnston2019-01-291-14/+6
| |\ \ \ | | | | | | | | | | Refactor event building into EventBuilder
| | * | | Refactor event building into EventBuilderErik Johnston2019-01-291-14/+6
| | | | | | | | | | | | | | | | | | | | | | | | | This is so that everything is done in one place, making it easier to change the event format based on room version
| * | | | Merge pull request #4510 from matrix-org/erikj/fixup_compute_event_signatureErik Johnston2019-01-291-1/+1
| |\ \ \ \ | | | | | | | | | | | | Fixup calls to `comput_event_signature`
| | * | | | Fixup calls to `comput_event_signature`Erik Johnston2019-01-291-1/+1
| | |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We currently pass FrozenEvent instead of `dict` to `compute_event_signature`, which works by accident due to `dict(event)` producing the correct result. This fixes PR #4493 commit 855a151
| * / / / Implement fallback for V2 invite APIErik Johnston2019-01-282-14/+89
| |/ / / | | | | | | | | | | | | | | | | If the room version is either 1 or 2 then a server should retry failed `/v2/invite` requests with the v1 API
| * | | Require event format version to parse or create eventsErik Johnston2019-01-254-41/+65
| | | |
| * | | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813Erik Johnston2019-01-253-13/+64
| |\ \ \
| | * \ \ Merge pull request #4448 from matrix-org/erikj/get_pdu_versionsErik Johnston2019-01-243-11/+51
| | |\ \ \ | | | | | | | | | | | | Add room_version param to get_pdu
| | | * | | Review commentsErik Johnston2019-01-241-1/+2
| | | | | |
| | | * | | Add room_version param to get_pduErik Johnston2019-01-233-11/+50
| | | |/ / | | | | | | | | | | | | | | | | | | | | When we add new event format we'll need to know the event format or room version when parsing events.
| | * | | Update make_membership_event docsErik Johnston2019-01-241-2/+5
| | | | |
| | * | | Remove unnecessary '_sign_event'Erik Johnston2019-01-231-1/+8
| | | | |
| | * | | Store rejected remote invite events as outliersErik Johnston2019-01-231-2/+8
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.
| * | | Review commentsErik Johnston2019-01-241-1/+2
| | | |
| * | | Implement MSC 1813 - Add room version to make APIsErik Johnston2019-01-232-6/+23
| |/ / | | | | | | | | | | | | We also implement `make_membership_event` converting the returned room version to an event format version.
| * | Add /v2/invite federation APIErik Johnston2019-01-152-6/+40
| | |
| * | Add groundwork for new versions of federation APIsErik Johnston2019-01-152-67/+71
| | |
| * | Don't log stack traces for HTTP error responsesErik Johnston2019-01-081-1/+6
| | |
| * | Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-5/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
| * | Update synapse/federation/federation_server.pyRichard van der Hoff2018-11-091-1/+1
| | | | | | | | | Co-Authored-By: erikjohnston <erikj@jki.re>
| * | Update synapse/federation/federation_server.pyRichard van der Hoff2018-11-091-1/+1
| | | | | | | | | Co-Authored-By: erikjohnston <erikj@jki.re>
| * | Drop PDUs of unknown roomsErik Johnston2018-11-081-1/+23
| | | | | | | | | | | | | | | | | | When we receive events over federation we will need to know the room version to be able to correctly handle them, e.g. once we start changing event formats. Currently, we attempt to handle events in unknown rooms.
| * | Add helpers for getting prev and auth events (#4139)Erik Johnston2018-11-061-3/+1
| | | | | | | | | | | | | | | | | | | | | * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.
| * | Clean up event accesses and testsErik Johnston2018-11-021-3/+0
| | | | | | | | | | | | | | | This is in preparation to refactor FrozenEvent to support different event formats for different room versions
| * | Remove the unused /pull federation API (#4118)Amber Brown2018-10-312-14/+0
| |/
* / Prevent showing non-fed rooms in fed /publicRoomsAndrew Morgan2019-02-261-1/+2
|/
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-10-251-1/+1
|\ | | | | | | erikj/alias_disallow_list
| * Fix a number of flake8 errorsRichard van der Hoff2018-10-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.
* | Add config option to control alias creationErik Johnston2018-10-191-14/+2
|/
* Fix incorrect truncation in get_missing_eventsRichard van der Hoff2018-10-162-6/+4
| | | | | | | | | It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.
* Various cleanups in the federation client code (#4031)Richard van der Hoff2018-10-162-23/+23
| | | | | | | | | | | | | | - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions
* Merge pull request #3964 from matrix-org/rav/remove_localhost_checksRichard van der Hoff2018-09-281-30/+6
|\ | | | | remove spurious federation checks on localhost
| * Fix complete fail to do the right thingRichard van der Hoff2018-09-281-1/+2
| |
| * remove spurious federation checks on localhostRichard van der Hoff2018-09-261-31/+6
| | | | | | | | | | | | | | | | | | | | | | There's really no point in checking for destinations called "localhost" because there is nothing stopping people creating other DNS entries which point to 127.0.0.1. The right fix for this is https://github.com/matrix-org/synapse/issues/3953. Blocking localhost, on the other hand, means that you get a surprise when trying to connect a test server on localhost to an existing server (with a 'normal' server_name).
* | more commentsRichard van der Hoff2018-09-271-2/+0
| |
* | Include eventid in log lines when processing incoming federation ↵Richard van der Hoff2018-09-271-15/+17
| | | | | | | | | | | | | | | | | | | | | | transactions (#3959) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.
* | fix docstring for FederationClient.get_state_for_roomRichard van der Hoff2018-09-261-4/+4
|/ | | | trivial fixes for docstring
* Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pduAmber Brown2018-09-261-1/+1
|\ | | | | Comments and interface cleanup for on_receive_pdu
| * Comments and interface cleanup for on_receive_pduRichard van der Hoff2018-09-201-1/+1
| | | | | | | | | | | | | | | | Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.
* | Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-16/+12
|/ | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* Fix handling of redacted events from federationErik Johnston2018-09-131-1/+1
| | | | | | | | | | If we receive an event that doesn't pass their content hash check (e.g. due to already being redacted) then we hit a bug which causes an exception to be raised, which then promplty stops the event (and request) from being processed. This effects all sorts of federation APIs, including joining rooms with a redacted state event.
* Port federation/ to py3 (#3847)Amber Brown2018-09-123-19/+18
|
* Merge pull request #3826 from matrix-org/rav/logging_for_keyringAmber Brown2018-09-121-7/+27
|\ | | | | add some logging for the keyring queue
| * clearer logging when things fail, tooRichard van der Hoff2018-09-061-7/+27
| |
* | Limit the number of PDUs/EDUs per fedreation transactionErik Johnston2018-09-061-0/+12
|/
* Merge branch 'master' into developRichard van der Hoff2018-09-064-31/+125
|\
| * Fix origin handling for pushed transactionsRichard van der Hoff2018-09-053-15/+15
| | | | | | | | | | Use the actual origin for push transactions, rather than whatever the remote server claimed.
| * Check that signatures on events are validRichard van der Hoff2018-09-051-16/+110
| | | | | | | | | | | | We should check that both the sender's server, and the server which created the event_id (which may be different from whatever the remote server has told us the origin is), have signed the event.
* | Port http/ to Python 3 (#3771)Amber Brown2018-09-061-5/+5
| |
* | fix #3445Krombel2018-08-291-2/+2
| | | | | | | | | | itervalues(d) calls d.itervalues() [PY2] and d.values() [PY3] but SortedDict only implements d.values()
* | Merge pull request #3735 from matrix-org/travis/federation-spellingRichard van der Hoff2018-08-221-1/+1
|\ \ | | | | | | limt -> limit
| * | limt -> limitTravis Ralston2018-07-311-1/+1
| | |
* | | Log more detail when we fail to authenticate requestErik Johnston2018-08-211-3/+3
| | |
* | | Don't log exceptions when failing to fetch server keysErik Johnston2018-08-211-2/+2
| |/ |/| | | | | | | Not being able to resolve or connect to remote servers is an expected error, so we shouldn't log at ERROR with stacktraces.
* | Integrate presence from hotfixes (#3694)Amber Brown2018-08-181-0/+4
| |
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-151-4/+4
|\ \ | | | | | | | | | erikj/split_federation
| * | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-4/+4
| | |
* | | Fixup doc commentsErik Johnston2018-08-091-0/+11
| | |
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-095-13/+132
|\| | | | | | | | | | | erikj/split_federation
| * | Merge pull request #3664 from matrix-org/rav/federation_metricsRichard van der Hoff2018-08-081-1/+9
| |\ \ | | | | | | | | more metrics for the federation and appservice senders
| | * | more metrics for the federation and appservice sendersRichard van der Hoff2018-08-071-1/+9
| | | |
| * | | Check m.room.create for sane room_versionsRichard van der Hoff2018-08-061-3/+23
| | | |
| * | | include known room versions in outgoing make_joinsRichard van der Hoff2018-08-062-4/+9
| | | |
| * | | sanity check response from make_joinRichard van der Hoff2018-08-031-1/+3
| | | |
| * | | Enforce compatibility when processing make_join requestsRichard van der Hoff2018-08-032-4/+41
| | | | | | | | | | | | | | | | | | | | | | | | Reject make_join requests from servers which do not support the room version. Also include the room version in the response.
| * | | Docstrings for BaseFederationServletRichard van der Hoff2018-08-031-0/+47
| |/ / | | | | | | | | | ... to save me reverse-engineering this stuff again.
* / / Add EDU/query handling over replicationErik Johnston2018-08-061-0/+43
|/ /
* | Merge pull request #3639 from matrix-org/rav/refactor_error_handlingRichard van der Hoff2018-08-021-12/+17
|\ \ | | | | | | Clean up handling of errors from outbound requests
| * | Refactor code for turning HttpResponseException into SynapseErrorRichard van der Hoff2018-08-011-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call.
| * | Be more careful which errors we send back over the C-S APIRichard van der Hoff2018-08-011-12/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We really shouldn't be sending all CodeMessageExceptions back over the C-S API; it will include things like 401s which we shouldn't proxy. That means that we need to explicitly turn a few HttpResponseExceptions into SynapseErrors in the federation layer. The effect of the latter is that the matrix errcode will get passed through correctly to calling clients, which might help with some of the random M_UNKNOWN errors when trying to join rooms.
* | | Merge pull request #3638 from ↵Richard van der Hoff2018-08-021-129/+148
|\| | | | | | | | | | | | | | matrix-org/rav/refactor_federation_client_exception_handling Factor out exception handling in federation_client
| * | Factor out exception handling in federation_clientRichard van der Hoff2018-08-011-129/+148
| | | | | | | | | | | | | | | Factor out the error handling from make_membership_event, send_join, and send_leave, so that it can be shared.
* | | Merge branch 'master' into developRichard van der Hoff2018-08-021-0/+1
|\ \ \ | |/ / |/| |
| * | Validation for events/rooms in fed requestsRichard van der Hoff2018-08-021-0/+1
| | | | | | | | | | | | | | | | | | | | | When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)
* | | Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-1/+1
| |/ |/|
* | Remove pdu_failures from transactionsTravis Ralston2018-07-305-96/+7
| | | | | | The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
* | Update the send_leave path to be an event_idTravis Ralston2018-07-261-2/+2
| | | | | | It's still not used, however the parameter is an event ID not a transaction ID.
* | Improve logging for exceptions handling PDUsRichard van der Hoff2018-07-231-3/+8
| | | | | | | | when we get an exception handling a federation PDU, log the whole stacktrace.
* | Run things as background processesRichard van der Hoff2018-07-181-9/+6
| | | | | | | | | | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).
* | Resource tracking for background processesRichard van der Hoff2018-07-181-5/+7
|/ | | | | | | | | | | | | | | | This introduces a mechanism for tracking resource usage by background processes, along with an example of how it will be used. This will help address #3518, but more importantly will give us better insights into things which are happening but not being shown up by the request metrics. We *could* do this with Measure blocks, but: - I think having them pulled out as a completely separate metric class will make it easier to distinguish top-level processes from those which are nested. - I want to be able to report on in-flight background processes, and I don't think we want to do this for *all* Measure blocks.
* rename assert_params_in_request to assert_params_in_dictKrombel2018-07-131-2/+2
| | | | | | the method "assert_params_in_request" does handle dicts and not requests. A request body has to be parsed to json before this method can be used
* run isortAmber Brown2018-07-099-64/+57
|
* Implementation of server_aclsRichard van der Hoff2018-07-042-4/+154
| | | | | ... as described at https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
* More server_name validationRichard van der Hoff2018-07-041-2/+3
| | | | | | | | We need to do a bit more validation when we get a server name, but don't want to be re-doing it all over the shop, so factor out a separate parse_and_validate_server_name, and do the extra validation. Also, use it to verify the server name in the config file.
* Reject invalid server names (#3480)Richard van der Hoff2018-07-031-22/+44
| | | | | Make sure that server_names used in auth headers are sane, and reject them with a sensible error code, before they disappear off into the depths of the system.
* Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checkingErik Johnston2018-06-291-1/+3
|\ | | | | Check the state of prev_events a bit more thoroughly when coming over federation
| * handle federation not telling us about prev_eventsAmber Brown2018-06-271-1/+3
| |
* | Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-1/+1
|/
* Merge pull request #3428 from matrix-org/erikj/persisted_pduErik Johnston2018-06-221-12/+1
|\ | | | | Simplify get_persisted_pdu
| * simplify get_persisted_pduRichard van der Hoff2018-06-121-12/+1
| | | | | | | | | | it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.
* | Populate synapse_federation_client_sent_pdu_destinations:count again (#3386)Amber Brown2018-06-211-3/+7
| |
* | Remove run_on_reactor (#3395)Amber Brown2018-06-141-4/+0
|/
* federation/send_queue.py: fix usage of sortedcontainers.SortedDictIvan Shapovalov2018-06-061-27/+22
|
* Port to sortedcontainers (with tests!) (#3332)Amber Brown2018-06-061-7/+7
|
* federation/send_queue.py: fix usage of LaterGaugeIvan Shapovalov2018-06-031-1/+1
| | | | | Fixes a startup crash due to commit df9f72d9e5fe264b86005208e0f096156eb03e4b "replacing portions".
* Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-313-5/+7
| | | | they're not meant to be lazy (#3307)
* fixesAmber Brown2018-05-231-4/+4
|
* cleanupAmber Brown2018-05-221-3/+2
|
* cleanup pep8 errorsAmber Brown2018-05-222-6/+21
|
* fixesAmber Brown2018-05-221-3/+3
|
* replacing portionsAmber Brown2018-05-214-53/+35
|
* Merge pull request #3209 from damir-manapov/masterRichard van der Hoff2018-05-111-2/+0
|\ | | | | transaction_id, destination defined twice
| * transaction_id, destination twiceDamir Manapov2018-05-101-2/+0
| |
* | Merge branch 'release-v0.28.1' into developRichard van der Hoff2018-05-011-3/+18
|\|
| * Apply some limits to depth to counter abuseRichard van der Hoff2018-05-011-3/+18
| | | | | | | | | | | | | | * When creating a new event, cap its depth to 2^63 - 1 * When receiving events, reject any without a sensible depth As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
* | Merge branch 'develop' into py3-xrange-1Richard van der Hoff2018-04-301-2/+3
|\ \
| * \ Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-272-3/+12
| |\ \
| * | | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-2/+3
| | |/ | |/| | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* | | Move more xrange to sixAdrian Tschira2018-04-281-1/+3
| |/ |/| | | | | | | | | plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Improve exception handling for background processesRichard van der Hoff2018-04-272-3/+12
|/ | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* Merge pull request #3118 from matrix-org/rav/reject_prev_eventsRichard van der Hoff2018-04-231-2/+23
|\ | | | | Reject events which have lots of prev_events
| * Add some commentsRichard van der Hoff2018-04-181-2/+23
| |
* | Merge pull request #3106 from NotAFile/py3-six-itervalues-1Richard van der Hoff2018-04-202-9/+13
|\ \ | | | | | | Use six.itervalues in some places
| * | Use six.itervalues in some placesAdrian Tschira2018-04-152-9/+13
| |/ | | | | | | | | | | There's more where that came from Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Merge pull request #3093 from matrix-org/rav/response_cache_wrapRichard van der Hoff2018-04-201-11/+11
|\ \ | | | | | | Refactor ResponseCache usage
| * | Reinstate linearizer for federation_server.on_context_state_requestRichard van der Hoff2018-04-201-5/+11
| | |
| * | Refactor ResponseCache usageRichard van der Hoff2018-04-121-11/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.
* | | Refactor store.have_eventsRichard van der Hoff2018-04-201-1/+1
| |/ |/| | | | | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.
* | fix federation_domain_whitelistMatthew Hodgson2018-04-131-6/+6
| | | | | | | | we were checking the wrong server_name on inbound requests
* | revert last to PR properlyMatthew Hodgson2018-04-131-6/+6
| |
* | correctly auth inbound federation_domain_whitelist reqsMatthew Hodgson2018-04-131-6/+6
| |
* | Revert "Use sortedcontainers instead of blist"Richard van der Hoff2018-04-131-7/+7
|/ | | | | | | | | | | This reverts commit 9fbe70a7dc3afabfdac176ba1f4be32dd44602aa. It turns out that sortedcontainers.SortedDict is not an exact match for blist.sorteddict; in particular, `popitem()` removes things from the opposite end of the dict. This is trivial to fix, but I want to add some unit tests, and potentially some more thought about it, before we do so.
* Merge pull request #3092 from matrix-org/rav/response_cache_metricsRichard van der Hoff2018-04-121-1/+1
|\ | | | | Add metrics for ResponseCache
| * Add metrics for ResponseCacheRichard van der Hoff2018-04-101-1/+1
| |
* | Set all metrics at the same timeErik Johnston2018-04-121-6/+6
| |
* | Track last processed event received_tsErik Johnston2018-04-111-0/+11
| |
* | Track where event stream processing have gotten up toErik Johnston2018-04-111-0/+4
| |
* | Merge pull request #2760 from Valodim/pypyRichard van der Hoff2018-04-111-7/+7
|\ \ | | | | | | Synapse on PyPy
| * | Use sortedcontainers instead of blistVincent Breitmoser2018-04-101-7/+7
| |/ | | | | | | | | | | | | | | This commit drop-in replaces blist with SortedContainers. They are written in pure python so work with pypy, but perform as good as native implementations, at least in a couple benchmarks: http://www.grantjenks.com/docs/sortedcontainers/performance.html
* | Merge pull request #3078 from matrix-org/erikj/federation_senderErik Johnston2018-04-101-15/+41
|\ \ | | | | | | Send federation events concurrently
| * | Use run_in_background insteadErik Johnston2018-04-101-1/+1
| | |
| * | Preserve log contexts correctlyErik Johnston2018-04-101-1/+4
| | |
| * | Log event ID on exceptionErik Johnston2018-04-101-1/+4
| | |
| * | Handle all events in a room correctlyErik Johnston2018-04-091-1/+2
| | |
| * | Send federation events concurrentlyErik Johnston2018-04-091-4/+18
| | |
| * | Handle exceptions in get_hosts_for_room when sending events over federationErik Johnston2018-04-091-11/+16
| |/
* | Ensure slashes are escapedErik Johnston2018-04-101-1/+1
| |
* | URL quote path segments over federationErik Johnston2018-04-101-48/+80
|/
* pep8Luke Barnard2018-04-061-1/+1
|
* Fix federation client `set_group_joinable` typoLuke Barnard2018-04-061-1/+1
|
* Implement group join APIDavid Baker2018-04-062-0/+31
|
* use PUT instead of POST for federating groups/m.join_policyKrombel2018-04-062-2/+2
|
* Use "/settings/" (plural)Luke Barnard2018-04-052-2/+2
|
* Use join_policy API instead of joinableLuke Barnard2018-04-032-6/+6
| | | | | | | | | | | | | | | | | The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".
* pep8David Baker2018-03-281-2/+1
|
* Add joinability for groupsDavid Baker2018-03-282-0/+37
| | | | | Adds API to set the 'joinable' flag, and corresponding flag in the table.
* Merge pull request #2978 from matrix-org/erikj/refactor_replication_layerErik Johnston2018-03-131-1/+1
|\ | | | | Remove ReplicationLayer and user Client/Server directly
| * s/replication_server/federation_serverErik Johnston2018-03-131-1/+1
| |
* | Merge pull request #2979 from matrix-org/erikj/no_handlersErik Johnston2018-03-134-69/+2
|\| | | | | Don't build handlers on workers unnecessarily
| * Remove unused ReplicationLayerErik Johnston2018-03-132-59/+0
| |
| * Split replication layer into twoErik Johnston2018-03-132-10/+2
| |
* | Merge pull request #2977 from matrix-org/erikj/replication_move_propsErik Johnston2018-03-134-22/+13
|\| | | | | Move property setting from ReplicationLayer to base classes
| * Move property setting from ReplicationLayer to FederationBaseErik Johnston2018-03-134-22/+13
| |
* | Fix docstring typesErik Johnston2018-03-131-2/+2
|/
* Split out edu/query registration to a separate classErik Johnston2018-03-131-48/+69
|
* Add federation_domain_whitelist option (#2820)Matthew Hodgson2018-01-224-3/+18
| | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
* Metrics for events processed in appservice and fed senderRichard van der Hoff2018-01-151-0/+4
| | | | More metrics I wished I'd had
* Check missing fields in event_from_pdu_jsonRichard van der Hoff2017-12-301-0/+7
| | | | Return a 400 rather than a 500 when somebody messes up their send_join
* Factor out `event_from_pdu_json`Richard van der Hoff2017-12-303-37/+44
| | | | | turns out we have two copies of this, and neither needs to be an instance method
* federation_server: clean up importsRichard van der Hoff2017-12-301-14/+11
|
* federation_client: clean up importsRichard van der Hoff2017-12-301-13/+10
|
* Clear logcontext before starting fed txn queue runnerRichard van der Hoff2017-11-281-2/+8
| | | | | These processes take a long time compared to the request, so there is lots of "Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.
* Remove preserve_context_over_{fn, deferred}Richard van der Hoff2017-11-141-2/+2
| | | | | Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.