summary refs log tree commit diff
path: root/tests/replication (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Always notify replication when a stream advances (#14877)Erik Johnston2023-01-201-16/+7
| | | This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
* Wait for streams to catch up when processing HTTP replication. (#14820)Erik Johnston2023-01-181-4/+5
| | | | This should hopefully mitigate a class of races where data gets out of sync due a HTTP replication request racing with the replication streams.
* Fix bug in `wait_for_stream_position` (#14856)Erik Johnston2023-01-171-0/+78
| | | | | We were incorrectly checking if the *local* token had been advanced, rather than the token for the remote instance. In practice, I don't think this has caused any bugs due to where we use `wait_for_stream_position`, as critically we don't use it on instances that also write to the given streams (and so the local token will lag behind all remote tokens).
* Faster remote room joins: unblock tasks waiting for full room state when the ↵reivilibre2022-12-061-0/+65
| | | | un-partial-stating of that room is received over the replication stream. [rei:frrj/streams/unpsr] (#14474)
* Modernize unit tests configuration settings for workers. (#14568)realtyem2022-12-017-49/+55
| | | | Use the newer foo_instances configuration instead of the deprecated flags to enable specific features (e.g. start_pushers).
* Fix an invalid comparison of `UserPresenceState` to `str` (#14393)Andrew Morgan2022-11-161-1/+6
|
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-162-9/+12
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Merge/remove `Slaved*` stores into `WorkerStores` (#14375)Nick Mills-Barrett2022-11-111-3/+3
|
* Add initial power level event to batch of bulk persisted events when ↵Shay2022-10-211-1/+1
| | | | creating a new room. (#14228)
* Track notification counts per thread (implement MSC3773). (#13776)Patrick Cloke2022-10-041-4/+13
| | | | | | | | When retrieving counts of notifications segment the results based on the thread ID, but choose whether to return them as individual threads or as a single summed field by letting the client opt-in via a sync flag. The summarization code is also updated to be per thread, instead of per room.
* Accept & store thread IDs for receipts (implement MSC3771). (#13782)Patrick Cloke2022-09-232-3/+14
| | | | Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.
* Support enabling/disabling pushers (from MSC3881) (#13799)Brendan Abolivier2022-09-211-1/+1
| | | Partial implementation of MSC3881
* Add cache invalidation across workers to module API (#13667)Mathieu Velten2022-09-211-0/+79
| | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Update event push action and receipt tables to support threads. (#13753)Patrick Cloke2022-09-141-0/+1
| | | | | | | | | | | | | | | Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`, and `event_push_summary` tables. This will notifications to be segmented by the thread in a future pull request. The `thread_id` column stores the root event ID or the special value `"main"`. The `thread_id` column for `event_push_actions` and `event_push_summary` is backfilled with `"main"` for all existing rows. New entries into `event_push_actions` and `event_push_actions_staging` will get the proper thread ID. `receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar, except `NULL` is a special value meaning the receipt is "unthreaded". See MSC3771 and MSC3773 for where this data will be useful.
* Generalise the `@cancellable` annotation so it can be used on functions ↵reivilibre2022-08-311-1/+2
| | | | other than just servlet methods. (#13662)
* Reduce the number of tests using TCP replication. (#13543)Patrick Cloke2022-08-193-68/+33
| | | | Uses Redis replication in additional test cases (instead of TCP replication). A small step towards dropping TCP replication.
* Remove old empty/redundant slaved stores. (#13349)Nick Mills-Barrett2022-07-212-307/+0
|
* Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston2022-06-151-6/+17
| | | | | | | | | | | Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
* Clean up the test code for client disconnections (#12929)Sean Quah2022-06-071-6/+4
| | | | | | | * Reword failure message about `await_result=False` * Use `reactor.advance()` instead of `reactor.pump()` * Raise `AssertionError`s ourselves * Un-instance method `_test_disconnect` * Replace `ThreadedMemoryReactorClock` with `MemoryReactorClock`
* Rename storage classes (#12913)Erik Johnston2022-05-313-7/+17
|
* Lay some foundation work to allow workers to only subscribe to some kinds of ↵reivilibre2022-05-192-12/+115
| | | | messages, reducing replication traffic. (#12672)
* Fix room upgrades creating an empty room when auth fails (#12696)Sean Quah2022-05-161-13/+1
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Respect the `@cancellable` flag for `ReplicationEndpoint`s (#12700)Sean Quah2022-05-112-0/+119
| | | | | | | | | While `ReplicationEndpoint`s register themselves via `JsonResource`, they pass a method that calls the handler, instead of the handler itself, to `register_paths`. As a result, `JsonResource` will not correctly pick up the `@cancellable` flag and we have to apply it ourselves. Signed-off-by: Sean Quah <seanq@element.io>
* Use `getClientAddress` instead of `getClientIP`. (#12599)Patrick Cloke2022-05-041-8/+12
| | | | | getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.
* Implement changes to MSC2285 (hidden read receipts) (#12168)Šimon Brandner2022-05-041-9/+229
| | | | | * Changes hidden read receipts to be a separate receipt type (instead of a field on `m.read`). * Updates the `/receipts` endpoint to accept `m.fully_read`.
* Use constants for receipt types in tests. (#12582)Šimon Brandner2022-04-281-3/+8
|
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-4/+4
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Remove redundant `get_success` calls in test code (#12346)Richard van der Hoff2022-04-011-2/+2
| | | There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
* Bump `black` and `click` versions (#12320)David Robertson2022-03-291-1/+1
|
* Rename get_tcp_replication to get_replication_command_handler. (#12192)Patrick Cloke2022-03-104-5/+5
| | | | | | Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-281-1/+1
|
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-238-17/+17
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Add missing type hints to synapse.replication. (#11938)Patrick Cloke2022-02-082-3/+7
|
* Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre2022-01-211-2/+2
| | | | whose key is prefixed with the user ID (#11788)
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-1/+5
| | | To improve type hints throughout the code.
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-211-3/+4
|
* Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-261-1/+5
| | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* Misc typing fixes for `tests`, part 1 of N (#11323)David Robertson2021-11-121-4/+1
| | | | | | | | | | * Annotate HomeserverTestCase.servlets * Correct annotation of federation_auth_origin * Use AnyStr custom_headers instead of a Union This allows (str, str) and (bytes, bytes). This disallows (str, bytes) and (bytes, str) * DomainSpecificString.SIGIL is a ClassVar
* type-hint `HomeserverTestcase.setup_test_homeserver` (#10961)David Robertson2021-10-011-4/+15
| | | | | * type-hint `HomeserverTestcase.setup_test_homeserver` For better IDE completion. A small drive-by.
* Pass str to twisted's IReactorTCP (#10895)David Robertson2021-09-301-2/+2
| | | | | | | This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job. Until that change is in a twisted release, we'll have to ignore the type of the `host` argument. I've raised #10899 to remind us to review the issue in a few months' time.
* Add reactor to `SynapseRequest` and fix up types. (#10868)Erik Johnston2021-09-241-1/+1
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+1
|
* Create a constant for a small png image in tests. (#10834)Patrick Cloke2021-09-161-12/+6
| | | To avoid duplicating it between a few tests.
* Split `FederationHandler` in half (#10692)Richard van der Hoff2021-08-261-1/+1
| | | The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
* Do not include rooms with an unknown room version in a sync response. (#10644)Patrick Cloke2021-08-191-0/+1
| | | | A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.
* Convert room member storage tuples to attrs. (#10629)Patrick Cloke2021-08-181-3/+6
| | | | Instead of using namedtuples. This helps with asserting type hints and code completion.
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-177-8/+7
|
* Use inline type hints in `tests/` (#10350)Jonathan de Jong2021-07-135-18/+18
| | | | | | | | This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
* [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-133-8/+8
|
* Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-241-1/+1
| | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-1/+3
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Use a database table to hold the users that should have full presence sent ↵Andrew Morgan2021-05-181-1/+1
| | | | to them, instead of something in-memory (#9823)
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-0/+1
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* Kill off `_PushHTTPChannel`. (#9878)Richard van der Hoff2021-04-231-115/+19
| | | | | First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation. Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
* pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-231-0/+1
|
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-2/+2
|
* Move some replication processing out of generic_worker (#9796)Erik Johnston2021-04-141-5/+3
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1420-20/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-131-1/+1
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-097-10/+7
|
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-082-8/+12
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Fix CI by ignore type for None module import (#9709)Andrew Morgan2021-03-291-1/+1
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-262-2/+3
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Fix remaining mypy issues due to Twisted upgrade. (#9608)Patrick Cloke2021-03-151-28/+16
|
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-3/+5
|
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-10/+17
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston2021-02-244-4/+4
| | | | | | | | | | | | | | | | | * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1610-89/+83
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-261-16/+25
|
* lintRichard van der Hoff2020-12-152-4/+1
|
* Remove spurious "SynapseRequest" result from `make_request"Richard van der Hoff2020-12-154-24/+24
| | | | This was never used, so let's get rid of it.
* Replace `request.code` with `channel.code`Richard van der Hoff2020-12-152-9/+9
| | | | | | The two are equivalent, but really we want to check the HTTP result that got returned to the channel, not the code that the Request object *intended* to return to the channel.
* Default to blacklisting reserved IP ranges and add a whitelist. (#8870)Patrick Cloke2020-12-091-1/+1
| | | | This defaults `ip_range_blacklist` to reserved IP ranges and also adds an `ip_range_whitelist` setting to override it.
* Add authentication to replication endpoints. (#8853)Patrick Cloke2020-12-042-8/+120
| | | | Authentication is done by checking a shared secret provided in the Synapse configuration file.
* Add additional validation to pusher URLs. (#8865)Patrick Cloke2020-12-041-4/+4
| | | | Pusher URLs now must end in `/_matrix/push/v1/notify` per the specification.
* Merge pull request #8858 from matrix-org/rav/sso_uiaRichard van der Hoff2020-12-021-6/+8
|\ | | | | UIA: offer only available auth flows
| * fix up various test casesRichard van der Hoff2020-12-021-6/+8
| | | | | | | | | | A few test cases were relying on being able to mount non-client servlets on the test resource. it's better to give them their own Resources.
* | Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-023-10/+10
|/ | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Simplify the way the `HomeServer` object caches its internal attributes. ↵Jonathan de Jong2020-11-301-1/+1
| | | | | (#8565) Changes `@cache_in_self` to use underscore-prefixed attributes.
* Remove redundant calls to `render()`Richard van der Hoff2020-11-163-16/+1
|
* Make `make_request` actually render the requestRichard van der Hoff2020-11-161-2/+2
| | | | | | remove the stubbing out of `request.process`, so that `requestReceived` also renders the request via the appropriate resource. Replace render() with a stub for now.
* Merge branch 'develop' into rav/pass_site_to_make_requestRichard van der Hoff2020-11-161-2/+2
|\
| * Rename `create_test_json_resource` to `create_test_resource` (#8759)Richard van der Hoff2020-11-161-2/+2
| | | | | | | | | | The root resource isn't necessarily a JsonResource, so rename this method accordingly, and update a couple of test classes to use the method rather than directly manipulating self.resource.
* | use global make_request() directly where we have a custom ResourceRichard van der Hoff2020-11-153-21/+60
|/ | | | | | Where we want to render a request against a specific Resource, call the global make_request() function rather than the one in HomeserverTestCase, allowing us to pass in an appropriate `Site`.
* Fix race for concurrent downloads of remote media. (#8682)Erik Johnston2020-10-301-0/+277
| | | Fixes #6755
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-1/+1
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Don't require hiredis to run unit tests (#8680)Erik Johnston2020-10-291-1/+10
|
* Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-272-2/+2
| | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Solidify the HomeServer constructor. (#8515)Jonathan de Jong2020-10-152-3/+3
| | | | | | This implements a more standard API for instantiating a homeserver and moves some of the dependency injection into the test suite. More concretely this stops using `setattr` on all `kwargs` passed to `HomeServer`.
* Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_inviteRichard van der Hoff2020-10-151-1/+1
|\ | | | | Simplify `_locally_reject_invite`
| * Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-1/+1
| | | | | | | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
* | Add basic tests for sync/pagination with vector clock tokens. (#8488)Erik Johnston2020-10-141-0/+217
|/ | | | | These are tests for #8439
* Remove the deprecated Handlers object (#8494)Patrick Cloke2020-10-091-1/+1
| | | All handlers now available via get_*_handler() methods on the HomeServer.
* Add unit test for event persister sharding (#8433)Erik Johnston2020-10-022-21/+305
|
* Add EventStreamPosition type (#8388)Erik Johnston2020-09-241-3/+9
| | | | | | | | | | | | | | The idea is to remove some of the places we pass around `int`, where it can represent one of two things: 1. the position of an event in the stream; or 2. a token that partitions the stream, used as part of the stream tokens. The valid operations are then: 1. did a position happen before or after a token; 2. get all events that happened before or after a token; and 3. get all events between two tokens. (Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Allow for make_awaitable's return value to be re-used. (#8261)Patrick Cloke2020-09-081-5/+5
|
* Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-4/+6
|
* Do not propagate typing notifications from shadow-banned users. (#8176)Patrick Cloke2020-08-261-2/+2
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-3/+3
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-7/+6
|
* Convert push to async/await. (#7948)Patrick Cloke2020-07-271-2/+4
|
* Convert the message handler to async/await. (#7884)Patrick Cloke2020-07-221-32/+44
|
* Add ability to run multiple pusher instances (#7855)Erik Johnston2020-07-161-0/+193
| | | This reuses the same scheme as federation sender sharding
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-153-172/+246
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support handling registration requests across multiple client readers. (#7830)Patrick Cloke2020-07-131-0/+133
|
* Add ability to shard the federation sender (#7798)Erik Johnston2020-07-102-0/+287
|
* Back out MSC2625 implementation (#7761)Brendan Abolivier2020-07-011-16/+3
|
* Merge branch 'develop' into babolivier/mark_unreadBrendan Abolivier2020-06-152-21/+141
|\
| * Discard RDATA from already seen positions. (#7648)Patrick Cloke2020-06-152-21/+141
| |
* | LintBrendan Abolivier2020-06-121-3/+1
| |
* | Test that a mark_unread action updates the right counter when using a slave ↵Brendan Abolivier2020-06-121-0/+15
| | | | | | | | store
* | Fix testsBrendan Abolivier2020-06-101-3/+3
|/
* Performance improvements and refactor of Ratelimiter (#7595)Andrew Morgan2020-06-051-7/+2
| | | | | | | | | | While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both: * Rather undocumented, and * causing a *lot* of config checks This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. Best to be reviewed commit-by-commit.
* Fix recording of federation stream token (#7564)Richard van der Hoff2020-05-261-0/+71
| | | | | | | | A couple of changes of significance: * remove the `_last_ack < federation_position` condition, so that updates will still be correctly processed after restart * Correctly wire up send_federation_ack to the right class.
* fix mypy for tests/replication (#7518)Richard van der Hoff2020-05-182-13/+7
|
* Fix limit logic for AccountDataStream (#7384)Richard van der Hoff2020-05-151-0/+117
| | | | | | Make sure that the AccountDataStream presents complete updates, in the right order. This is much the same fix as #7337 and #7358, but applied to a different stream.
* Clean up replication unit tests. (#7490)Erik Johnston2020-05-137-51/+15
|
* Fix catchup-on-reconnect for the Federation Stream (#7374)Richard van der Hoff2020-05-053-12/+94
| | | | looks like we managed to break this during the refactorathon.
* Thread through instance name to replication client. (#7369)Erik Johnston2020-05-013-6/+6
| | | For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
* Use `stream.current_token()` and remove `stream_positions()` (#7172)Erik Johnston2020-05-014-34/+26
| | | | We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
* Add instance name to RDATA/POSITION commands (#7364)Erik Johnston2020-04-292-2/+5
| | | | | This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
* Don't relay REMOTE_SERVER_UP cmds to same conn. (#7352)Erik Johnston2020-04-291-0/+62
| | | | | | | | | | | | | | For direct TCP connections we need the master to relay REMOTE_SERVER_UP commands to the other connections so that all instances get notified about it. The old implementation just relayed to all connections, assuming that sending back to the original sender of the command was safe. This is not true for redis, where commands sent get echoed back to the sender, which was causing master to effectively infinite loop sending and then re-receiving REMOTE_SERVER_UP commands that it sent. The fix is to ensure that we only relay to *other* connections and not to the connection we received the notification from. Fixes #7334.
* Fix limit logic for EventsStream (#7358)Richard van der Hoff2020-04-294-21/+458
| | | | | | | | | | | | | | | | | | | * Factor out functions for injecting events into database I want to add some more flexibility to the tools for injecting events into the database, and I don't want to clutter up HomeserverTestCase with them, so let's factor them out to a new file. * Rework TestReplicationDataHandler This wasn't very easy to work with: the mock wrapping was largely superfluous, and it's useful to be able to inspect the received rows, and clear out the received list. * Fix AssertionErrors being thrown by EventsStream Part of the problem was that there was an off-by-one error in the assertion, but also the limit logic was too simple. Fix it all up and add some tests.
* Add some replication tests (#7278)Erik Johnston2020-04-282-11/+298
| | | | | | | Specifically some tests for the typing stream, which means we test streams that fetch missing updates via HTTP (rather than via the DB). We also shuffle things around a bit so that we create two separate `HomeServer` objects, rather than trying to insert a slaved store into places. Note: `test_typing.py` is heavily inspired by `test_receipts.py`
* Another go at fixing one-word commands (#7326)Richard van der Hoff2020-04-221-0/+42
| | | I messed this up last time I tried (#7239 / e13c6c7).
* Add ability to run replication protocol over redis. (#7040)Erik Johnston2020-04-221-2/+2
| | | This is configured via the `redis` config options.
* Move client command handling out of TCP protocol (#7185)Erik Johnston2020-04-063-29/+25
| | | The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
* Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-252-23/+84
| | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* Convert `*StreamRow` classes to inner classes (#7116)Richard van der Hoff2020-03-231-2/+2
| | | | | This just helps keep the rows closer to their streams, so that it's easier to see what the format of each stream is.
* Read the room version from database when fetching events (#6874)Richard van der Hoff2020-03-041-0/+10
| | | | | This is a precursor to giving EventBase objects the knowledge of which room version they belong to.
* Add a `make_event_from_dict` method (#6858)Richard van der Hoff2020-02-071-4/+8
| | | | | | | ... and use it in places where it's trivial to do so. This will make it easier to pass room versions into the FrozenEvent constructors.
* Port synapse.replication.tcp to async/await (#6666)Erik Johnston2020-01-161-1/+1
| | | | | | | | | | * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing....
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-2/+2
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Add database config class (#6513)Erik Johnston2019-12-181-2/+4
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Fixup testsErik Johnston2019-12-061-1/+4
|
* Implementation of MSC2314 (#6176)Amber Brown2019-11-282-0/+7
|
* Use new EventPersistenceStoreErik Johnston2019-10-232-3/+8
|
* Run Black on the tests again (#5170)Amber Brown2019-05-103-26/+14
|
* Fix sync bug when accepting invites (#4956)Richard van der Hoff2019-04-022-28/+161
| | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.
* Move replication.tcp.streams into a packageRichard van der Hoff2019-03-271-1/+1
|
* Fix bug where read-receipts lost their timestamps (#4927)Richard van der Hoff2019-03-254-0/+148
| | | | | Make sure that they are sent correctly over the replication stream. Fixes: #4898
* Add rate-limiting on registration (#4735)Brendan Abolivier2019-03-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test
* Clean up event accesses and testsErik Johnston2018-11-021-2/+2
| | | | | This is in preparation to refactor FrozenEvent to support different event formats for different room versions
* Disable frozen dicts by default (#3987)Amber Brown2018-10-021-1/+5
|
* Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-32/+3
| | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Fix tests on postgresql (#3740)Amber Brown2018-09-044-124/+116
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-204-67/+64
|\ | | | | | | erikj/refactor_state_handler
| * Run tests under PostgreSQL (#3423)Amber Brown2018-08-131-0/+1
| |
| * Run black.black2018-08-104-67/+63
| |
* | Fix testsErik Johnston2018-08-091-1/+2
|/
* Fix unit testsRichard van der Hoff2018-07-251-7/+30
| | | | | | on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival.
* Fix unit testsErik Johnston2018-07-231-3/+5
|
* run isortAmber Brown2018-07-094-13/+15
|
* remove dead filter_events_for_clientsRichard van der Hoff2018-06-121-8/+0
| | | | | | This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.
* s/replication_client/federation_client/Erik Johnston2018-03-131-1/+1
|
* Fix testsErik Johnston2018-03-131-1/+1
|
* Fix unit testsErik Johnston2018-02-201-4/+6
|
* Remove context.push_actionsErik Johnston2018-02-151-1/+4
|
* Store state groups separately from events (#2784)Erik Johnston2018-02-061-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Matthew's fixes to the unit testsRichard van der Hoff2018-01-221-2/+6
| | | | Extracted from https://github.com/matrix-org/synapse/pull/2820
* Remove test of replication resourceErik Johnston2017-04-111-204/+0
|
* Change slave storage to use new replication interfaceErik Johnston2017-04-031-8/+22
| | | | | | | As the TCP replication uses a slightly different API and streams than the HTTP replication. This breaks HTTP replication.
* Fix unit testErik Johnston2017-03-201-1/+1
|
* Remote membership tests for replicationErik Johnston2017-01-311-43/+0
| | | | | This is because it now relies of the caches stream, which only works on postgres. We are trying to test with sqlite.
* Insert delta of current_state_events to be more efficientErik Johnston2017-01-201-29/+0
|
* Derive current_state_events from state groupsErik Johnston2017-01-201-28/+17
|
* Shuffle receipt handler around so that worker apps don't need to load itErik Johnston2016-11-231-1/+1
|
* Reduce DB hits for replicationErik Johnston2016-09-232-3/+3
| | | | | | | | | | | | | | Some streams will occaisonally advance their positions without actually having any new rows to send over federation. Currently this means that the token will not advance on the workers, leading to them repeatedly sending a slightly out of date token. This in turns requires the master to hit the DB to check if there are any new rows, rather than hitting the no op logic where we check if the given token matches the current token. This commit changes the API to always return an entry if the position for a stream has changed, allowing workers to advance their tokens correctly.
* Correctly handle the difference between prev and current stateErik Johnston2016-08-312-9/+5
|
* Replace context.current_state with context.current_state_idsErik Johnston2016-08-251-1/+8
|
* Add `create_requester` functionRichard van der Hoff2016-07-261-9/+11
| | | | | Wrap the `Requester` constructor with a function which provides sensible defaults, and use it throughout
* Remove room name & alias testDavid Baker2016-06-241-41/+0
| | | | as get_room_name_and_alias is now gone
* Move typing handler out of the Handlers objectMark Haines2016-05-171-1/+1
|
* Move the presence handler out of the Handlers objectMark Haines2016-05-161-1/+1
|
* Add a slaved datastore for account dataMark Haines2016-05-131-0/+56
|
* Replicate push actionsMark Haines2016-04-211-0/+43
|
* Merge pull request #738 from matrix-org/markjh/slaved_receiptsMark Haines2016-04-193-3/+43
|\ | | | | Add a slaved receipts store
| * Add a slaved receipts storeMark Haines2016-04-193-3/+43
| |
* | Replicate get_invited_rooms_for_userMark Haines2016-04-191-0/+12
|/
* Add tests for redactionsMark Haines2016-04-072-2/+51
|
* Add sensible __eq__ operators inside the tests.Mark Haines2016-04-071-1/+28
| | | | | Rather than adding them globally. This limits the changes to only affect the tests.
* Add tests for get_latest_event_ids_in_room and get_current_stateMark Haines2016-04-071-0/+62
|
* Test that room membership is replicatedMark Haines2016-04-061-8/+63
|
* Add a slaved events store classMark Haines2016-04-064-0/+199
| | | | | Add a test to check that get_room_names_and_aliases does the same thing on both the master and on the slave data store.
* Add a replication stream for state groupsMark Haines2016-03-301-3/+27
|
* Add replication stream for pushersMark Haines2016-03-151-0/+1
|
* Merge branch 'develop' into markjh/pushrule_streamMark Haines2016-03-041-7/+10
|\
| * Pass whole requester to ratelimitingDaniel Wagner-Hall2016-03-031-7/+10
| | | | | | | | This will enable more detailed decisions
* | Hook push rules up to the replication APIMark Haines2016-03-021-2/+4
|/
* Add a /replication API for extracting the updates that happened onMark Haines2016-03-012-0/+193
synapse This is necessary for replicating the data in synapse to be visible to a separate service because presence and typing notifications aren't stored in a database so won't be visible to another process. This API can be used to either get the raw data by requesting the tables themselves or to just receive notifications for updates by following the streams meta-stream. Returns updates for each table requested a JSON array of arrays with a row for each row in the table. Each table is prefixed by a header row with the: name of the table, current stream_id position for the table, number of rows, number of columns and the names of the columns. This is followed by the rows that have been added to the server since the requester last asked. The API has a timeout and is hooked up to the notifier so that a slave can long poll for updates.